SparseArray 1.0.0
SparseArray is an infrastructure package that defines the SparseArray virtual class whose purpose is to be extended by other S4 classes that wish to represent in-memory multidimensional sparse arrays.
The package currently defines two concrete SparseArray subclasses: COO_SparseArray and SVT_SparseArray. Each subclass uses its own internal representation for the nonzero multidimensional data, the “COO layout” and the “SVT layout”, respectively.
This vignette focuses on SVT_SparseArray objects.
if (!requireNamespace("BiocManager", quietly=TRUE))
install.packages("BiocManager")
BiocManager::install("SparseArray")library(SparseArray)The SVT_SparseArray class provides an efficient representation of the nonzero multidimensional data via a novel layout called the “SVT layout”.
Note that SVT_SparseArray objects mimic the behavior of ordinary matrices or arrays in R as much as possible. In particular, they suppport most of the “standard array API” defined in base R.
SVT_SparseArray objects can be constructed in many ways. A common way is to coerce an ordinary matrix or array to SVT_SparseArray:
m <- matrix(0L, nrow=6, ncol=4)
m[c(1:2, 8, 10, 15:17, 24)] <- (1:8)*10L
svt1 <- as(m, "SVT_SparseArray")
svt1## <6 x 4 SVT_SparseMatrix> of type "integer" (nzcount=8):
## [,1] [,2] [,3] [,4]
## [1,] 10 0 0 0
## [2,] 20 30 0 0
## [3,] 0 0 50 0
## [4,] 0 40 60 0
## [5,] 0 0 70 0
## [6,] 0 0 0 80a <- array(0L, 5:3)
a[c(1:2, 8, 10, 15:17, 20, 24, 40, 56:60)] <- (1:15)*10L
svt2 <- as(a, "SVT_SparseArray")
svt2## <5 x 4 x 3 SVT_SparseArray> of type "integer" (nzcount=15):
## ,,1
## [,1] [,2] [,3] [,4]
## [1,] 10 0 0 60
## [2,] 20 0 0 70
## [3,] 0 30 0 0
## [4,] 0 0 0 0
## [5,] 0 40 50 80
##
## ,,2
## [,1] [,2] [,3] [,4]
## [1,] 0 0 0 0
## [2,] 0 0 0 0
## [3,] 0 0 0 0
## [4,] 90 0 0 0
## [5,] 0 0 0 100
##
## ,,3
## [,1] [,2] [,3] [,4]
## [1,] 0 0 0 110
## [2,] 0 0 0 120
## [3,] 0 0 0 130
## [4,] 0 0 0 140
## [5,] 0 0 0 150Alternatively, the ordinary matrix or array can be passed to the
SVT_SparseArray constructor function:
svt1 <- SVT_SparseArray(m)
svt2 <- SVT_SparseArray(a)Note that coercing an ordinary matrix or array to SparseArray is also supported and will produce the same results:
svt1 <- as(m, "SparseArray")
svt2 <- as(a, "SparseArray")This is because, for most use cases, the SVT_SparseArray representation is more efficient than the COO_SparseArray representation, so the former is usually preferred over the latter.
For the same reason, the SparseArray constructor function will also
give the preference to the SVT_SparseArray representation:
svt1 <- SparseArray(m)
svt2 <- SparseArray(a)This is actually the most convenient way to turn an ordinary matrix or array into an SVT_SparseArray object.
Coercion back to ordinary matrix or array is supported:
as.array(svt1) # same as as.matrix(svt1)## [,1] [,2] [,3] [,4]
## [1,] 10 0 0 0
## [2,] 20 30 0 0
## [3,] 0 0 50 0
## [4,] 0 40 60 0
## [5,] 0 0 70 0
## [6,] 0 0 0 80as.array(svt2)## , , 1
##
## [,1] [,2] [,3] [,4]
## [1,] 10 0 0 60
## [2,] 20 0 0 70
## [3,] 0 30 0 0
## [4,] 0 0 0 0
## [5,] 0 40 50 80
##
## , , 2
##
## [,1] [,2] [,3] [,4]
## [1,] 0 0 0 0
## [2,] 0 0 0 0
## [3,] 0 0 0 0
## [4,] 90 0 0 0
## [5,] 0 0 0 100
##
## , , 3
##
## [,1] [,2] [,3] [,4]
## [1,] 0 0 0 110
## [2,] 0 0 0 120
## [3,] 0 0 0 130
## [4,] 0 0 0 140
## [5,] 0 0 0 150The standard array accessors are supported:
dim(svt2)## [1] 5 4 3length(svt2)## [1] 60dimnames(svt2) <- list(NULL, letters[1:4], LETTERS[1:3])
svt2## <5 x 4 x 3 SVT_SparseArray> of type "integer" (nzcount=15):
## ,,A
## a b c d
## [1,] 10 0 0 60
## [2,] 20 0 0 70
## [3,] 0 30 0 0
## [4,] 0 0 0 0
## [5,] 0 40 50 80
##
## ,,B
## a b c d
## [1,] 0 0 0 0
## [2,] 0 0 0 0
## [3,] 0 0 0 0
## [4,] 90 0 0 0
## [5,] 0 0 0 100
##
## ,,C
## a b c d
## [1,] 0 0 0 110
## [2,] 0 0 0 120
## [3,] 0 0 0 130
## [4,] 0 0 0 140
## [5,] 0 0 0 150Some additional accessors defined in the S4Arrays / SparseArray framework:
type(svt1)## [1] "integer"type(svt1) <- "double"
svt1## <6 x 4 SVT_SparseMatrix> of type "double" (nzcount=8):
## [,1] [,2] [,3] [,4]
## [1,] 10 0 0 0
## [2,] 20 30 0 0
## [3,] 0 0 50 0
## [4,] 0 40 60 0
## [5,] 0 0 70 0
## [6,] 0 0 0 80is_sparse(svt1)## [1] TRUEnzcount(svt1)## [1] 8sparsity(svt1)## [1] 0.6666667svt2[5:3, , "C"]## <3 x 4 SVT_SparseMatrix> of type "integer" (nzcount=3):
## a b c d
## [1,] 0 0 0 150
## [2,] 0 0 0 140
## [3,] 0 0 0 130Like with ordinary arrays in base R, assigning values of type "double" to
an SVT_SparseArray object of type "integer" will automatically change the
type of the object to "double":
type(svt2)## [1] "integer"svt2[5, 1, 3] <- NaN
type(svt2)## [1] "double"See ?SparseArray_subsetting for more information and additional examples.
The following summarization methods are provided at the moment:
anyNA(svt2)## [1] TRUErange(svt2, na.rm=TRUE)## [1] 0 150mean(svt2, na.rm=TRUE)## [1] 20.33898See ?SparseArray_summarization for more information and additional examples.
More operations will be added in the future e.g. which(), is.na(),
is.infinite(), is.nan(), etc…, as well as operations from
the Compare, Logic, Math, Math2, and Complex groups
(see ?S4groupGeneric for more information).
Two convenience functions are provided for this:
randomSparseArray(c(5, 6, 2), density=0.5)## <5 x 6 x 2 SVT_SparseArray> of type "double" (nzcount=30):
## ,,1
## [,1] [,2] [,3] [,4] [,5] [,6]
## [1,] 0.0000 0.8300 0.4200 -0.9900 0.0000 2.1000
## [2,] 0.0000 0.0000 -0.7800 0.0000 0.0000 -0.7500
## [3,] 0.0000 0.5500 0.0000 -1.1000 0.0000 -0.3800
## [4,] 0.0000 -0.5000 -0.0390 0.0000 0.0000 0.4600
## [5,] 0.0029 1.5000 -0.4100 0.0000 0.3300 1.3000
##
## ,,2
## [,1] [,2] [,3] [,4] [,5] [,6]
## [1,] 0.00 0.00 -1.90 0.49 -0.68 0.00
## [2,] 0.00 0.00 0.00 0.00 0.00 0.00
## [3,] 0.00 0.00 -0.51 0.95 0.74 -0.34
## [4,] -2.20 -1.10 0.00 0.00 0.90 -0.90
## [5,] 0.00 0.00 0.00 -0.83 0.00 0.22poissonSparseArray(c(5, 6, 2), density=0.5)## <5 x 6 x 2 SVT_SparseArray> of type "integer" (nzcount=34):
## ,,1
## [,1] [,2] [,3] [,4] [,5] [,6]
## [1,] 2 0 0 2 0 0
## [2,] 1 1 2 1 1 1
## [3,] 1 2 1 2 0 0
## [4,] 0 1 0 1 1 1
## [5,] 1 0 1 0 1 0
##
## ,,2
## [,1] [,2] [,3] [,4] [,5] [,6]
## [1,] 0 2 2 0 3 2
## [2,] 0 1 1 0 0 0
## [3,] 0 1 1 0 0 0
## [4,] 1 2 3 1 2 0
## [5,] 0 0 0 0 2 1See ?randomSparseArray for more information and additional examples.
t(svt1)## <4 x 6 SVT_SparseMatrix> of type "double" (nzcount=8):
## [,1] [,2] [,3] [,4] [,5] [,6]
## [1,] 10 20 0 0 0 0
## [2,] 0 30 0 40 0 0
## [3,] 0 0 50 60 70 0
## [4,] 0 0 0 0 0 80Like ordinary matrices in base R, SVT_SparseMatrix objects can be
combined by rows or columns, with rbind() or cbind():
svt3 <- poissonSparseMatrix(6, 2, density=0.5)
cbind(svt1, svt3)## <6 x 6 SVT_SparseMatrix> of type "double" (nzcount=13):
## [,1] [,2] [,3] [,4] [,5] [,6]
## [1,] 10 0 0 0 1 1
## [2,] 20 30 0 0 1 0
## [3,] 0 0 50 0 0 0
## [4,] 0 40 60 0 0 1
## [5,] 0 0 70 0 0 0
## [6,] 0 0 0 80 1 0See ?SparseArray_combine for more information and additional examples.
Like ordinary matrices in base R, SVT_SparseMatrix objects can be
multiplied with the %*% operator:
m4 <- matrix(0L, nrow=5, ncol=6, dimnames=list(letters[1:5], LETTERS[1:6]))
m4[c(2, 6, 12:17, 22:30)] <- 101:117
svt4 <- SparseArray(m4)
svt4 %*% svt3## [,1] [,2]
## a 215 107
## b 215 209
## c 115 0
## d 116 0
## e 117 0They also support crossprod() and tcrossprod():
crossprod(svt3)## [,1] [,2]
## [1,] 3 1
## [2,] 1 2See ?SparseMatrix_mult for more information and additional examples.
The SparseArray package provides memory-efficient row/col summarization methods for SVT_SparseMatrix objects:
rowVars(svt4, useNames=TRUE)## a b c d e
## 3468.267 1929.367 3620.167 3686.267 3752.967See ?matrixStats_methods for more information and additional examples.
rowsum() methodA rowsum() method is provided:
rowsum(svt4, group=c(1:3, 2:1))## A B C D E F
## 1 0 102 106 107 112 230
## 2 101 0 208 108 220 230
## 3 0 0 104 0 110 115See ?rowsum_methods for more information and additional examples.
Use writeSparseCSV() to write a sparse matrix to a CSV file:
csv_file <- tempfile()
writeSparseCSV(m4, csv_file)Use readSparseCSV() to read the file. This will import the data as
an SVT_SparseMatrix object:
readSparseCSV(csv_file)## <5 x 6 SVT_SparseMatrix> of type "integer" (nzcount=17):
## A B C D E F
## a 0 102 0 107 0 113
## b 101 0 103 108 109 114
## c 0 0 104 0 110 115
## d 0 0 105 0 111 116
## e 0 0 106 0 112 117See ?readSparseCSV for more information and additional examples.
The nonzero data of a SVT_SparseArray object is stored in a Sparse Vector Tree. This internal data representation is referred to as the “SVT layout”. It is similar to the “CSC layout” (compressed, sparse, column-oriented format) used by CsparseMatrix derivatives from the Matrix package, like dgCMatrix or lgCMatrix objects, but with the following improvements:
The “SVT layout” supports sparse arrays of arbitrary dimensions.
With the “SVT layout”, the sparse data can be of any type.
Whereas CsparseMatrix derivatives only support sparse data of
type "double" or "logical" at the moment.
The “SVT layout” imposes no limit on the number of nonzero elements that can be stored. With dgCMatrix/lgCMatrix objects, this number must be < 2^31.
Overall, the “SVT layout” allows more efficient operations on SVT_SparseArray objects.
See ?SVT_SparseArray for more information.
Please consult the individual man pages in the SparseArray
package to learn more about SVT_SparseArray objects and about the
package. A good starting point is the man page for SparseArray
objects: ?SparseArray
sessionInfo()## R version 4.3.0 RC (2023-04-13 r84269)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.2 LTS
##
## Matrix products: default
## BLAS: /home/biocbuild/bbs-3.17-bioc/R/lib/libRblas.so
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_GB LC_COLLATE=C
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## time zone: America/New_York
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats4 stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] SparseArray_1.0.0 S4Arrays_1.0.0 IRanges_2.34.0
## [4] S4Vectors_0.38.0 MatrixGenerics_1.12.0 matrixStats_0.63.0
## [7] BiocGenerics_0.46.0 Matrix_1.5-4 BiocStyle_2.28.0
##
## loaded via a namespace (and not attached):
## [1] crayon_1.5.2 cli_3.6.1 knitr_1.42
## [4] rlang_1.1.0 xfun_0.39 jsonlite_1.8.4
## [7] htmltools_0.5.5 sass_0.4.5 rmarkdown_2.21
## [10] grid_4.3.0 evaluate_0.20 jquerylib_0.1.4
## [13] fastmap_1.1.1 yaml_2.3.7 bookdown_0.33
## [16] BiocManager_1.30.20 compiler_4.3.0 XVector_0.40.0
## [19] lattice_0.21-8 digest_0.6.31 R6_2.5.1
## [22] bslib_0.4.2 tools_4.3.0 zlibbioc_1.46.0
## [25] cachem_1.0.7