varDiff_subset - HenrikBengtsson/matrixStats GitHub Wiki

matrixStats: Benchmark report

varDiff() benchmarks on subsetted computation

This report benchmark the performance of varDiff() on subsetted computation.

Data type "integer"

Data

> rvector <- function(n, mode = c("logical", "double", "integer"), range = c(-100, +100), na_prob = 0) {
+     mode <- match.arg(mode)
+     if (mode == "logical") {
+         x <- sample(c(FALSE, TRUE), size = n, replace = TRUE)
+     }     else {
+         x <- runif(n, min = range[1], max = range[2])
+     }
+     storage.mode(x) <- mode
+     if (na_prob > 0) 
+         x[sample(n, size = na_prob * n)] <- NA
+     x
+ }
> rvectors <- function(scale = 10, seed = 1, ...) {
+     set.seed(seed)
+     data <- list()
+     data[[1]] <- rvector(n = scale * 100, ...)
+     data[[2]] <- rvector(n = scale * 1000, ...)
+     data[[3]] <- rvector(n = scale * 10000, ...)
+     data[[4]] <- rvector(n = scale * 1e+05, ...)
+     data[[5]] <- rvector(n = scale * 1e+06, ...)
+     names(data) <- sprintf("n = %d", sapply(data, FUN = length))
+     data
+ }
> data <- rvectors(mode = mode)
> data <- data[1:4]

Results

n = 1000 vector

All elements

> x <- data[["n = 1000"]]
> idxs <- sample.int(length(x), size = length(x) * 0.7)
> x_S <- x[idxs]
> gc()
          used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 3253095 173.8    5709258 305.0  5709258 305.0
Vcells 7057946  53.9   25448368 194.2 87357391 666.5
> stats <- microbenchmark(varDiff_x_S = varDiff(x_S), `varDiff(x, idxs)` = varDiff(x, idxs = idxs), 
+     `varDiff(x[idxs])` = varDiff(x[idxs]), unit = "ms")

Table: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on integer+n = 1000 data. The top panel shows times in milliseconds and the bottom panel shows relative times.

expr min lq mean median uq max
1 varDiff_x_S 0.011496 0.011788 0.0120411 0.0119770 0.0121745 0.013815
2 varDiff(x, idxs) 0.013200 0.013456 0.0138305 0.0136700 0.0139175 0.022033
3 varDiff(x[idxs]) 0.013246 0.013518 0.0157008 0.0137625 0.0140355 0.191659
expr min lq mean median uq max
1 varDiff_x_S 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000
2 varDiff(x, idxs) 1.148225 1.141500 1.148607 1.141354 1.143168 1.594861
3 varDiff(x[idxs]) 1.152227 1.146759 1.303937 1.149077 1.152861 13.873254

Figure: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on integer+n = 1000 data. Outliers are displayed as crosses. Times are in milliseconds.

n = 10000 vector

All elements

> x <- data[["n = 10000"]]
> idxs <- sample.int(length(x), size = length(x) * 0.7)
> x_S <- x[idxs]
> gc()
          used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 3249427 173.6    5709258 305.0  5709258 305.0
Vcells 6801603  51.9   25448368 194.2 87357391 666.5
> stats <- microbenchmark(varDiff_x_S = varDiff(x_S), `varDiff(x, idxs)` = varDiff(x, idxs = idxs), 
+     `varDiff(x[idxs])` = varDiff(x[idxs]), unit = "ms")

Table: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on integer+n = 10000 data. The top panel shows times in milliseconds and the bottom panel shows relative times.

expr min lq mean median uq max
1 varDiff_x_S 0.042681 0.0430370 0.0435450 0.0433015 0.0436500 0.053150
3 varDiff(x[idxs]) 0.054690 0.0553490 0.0565993 0.0557110 0.0561540 0.112241
2 varDiff(x, idxs) 0.054479 0.0554765 0.0562948 0.0557895 0.0562245 0.083250
expr min lq mean median uq max
1 varDiff_x_S 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000
3 varDiff(x[idxs]) 1.281366 1.286079 1.299788 1.286584 1.286461 2.111778
2 varDiff(x, idxs) 1.276423 1.289042 1.292795 1.288396 1.288076 1.566322

Figure: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on integer+n = 10000 data. Outliers are displayed as crosses. Times are in milliseconds.

n = 100000 vector

All elements

> x <- data[["n = 100000"]]
> idxs <- sample.int(length(x), size = length(x) * 0.7)
> x_S <- x[idxs]
> gc()
          used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 3249499 173.6    5709258 305.0  5709258 305.0
Vcells 6865163  52.4   25448368 194.2 87357391 666.5
> stats <- microbenchmark(varDiff_x_S = varDiff(x_S), `varDiff(x, idxs)` = varDiff(x, idxs = idxs), 
+     `varDiff(x[idxs])` = varDiff(x[idxs]), unit = "ms")

Table: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on integer+n = 100000 data. The top panel shows times in milliseconds and the bottom panel shows relative times.

expr min lq mean median uq max
1 varDiff_x_S 0.325149 0.3278415 0.4898900 0.3401835 0.3847615 6.517523
3 varDiff(x[idxs]) 0.472085 0.4748955 0.5470686 0.4975670 0.5781610 0.787172
2 varDiff(x, idxs) 0.472942 0.4804530 0.5691244 0.5034945 0.6085175 0.809432
expr min lq mean median uq max
1 varDiff_x_S 1.000000 1.000000 1.000000 1.000000 1.000000 1.0000000
3 varDiff(x[idxs]) 1.451904 1.448552 1.116717 1.462643 1.502648 0.1207778
2 varDiff(x, idxs) 1.454539 1.465504 1.161739 1.480067 1.581545 0.1241932

Figure: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on integer+n = 100000 data. Outliers are displayed as crosses. Times are in milliseconds.

n = 1000000 vector

All elements

> x <- data[["n = 1000000"]]
> idxs <- sample.int(length(x), size = length(x) * 0.7)
> x_S <- x[idxs]
> gc()
          used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 3249571 173.6    5709258 305.0  5709258 305.0
Vcells 7495212  57.2   25448368 194.2 87357391 666.5
> stats <- microbenchmark(varDiff_x_S = varDiff(x_S), `varDiff(x, idxs)` = varDiff(x, idxs = idxs), 
+     `varDiff(x[idxs])` = varDiff(x[idxs]), unit = "ms")

Table: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on integer+n = 1000000 data. The top panel shows times in milliseconds and the bottom panel shows relative times.

expr min lq mean median uq max
1 varDiff_x_S 3.525973 3.895351 4.384416 4.057241 4.267187 12.95458
3 varDiff(x[idxs]) 6.494525 7.201244 8.321316 7.410835 8.592438 21.84177
2 varDiff(x, idxs) 6.785427 7.204877 11.108410 7.513769 9.155086 257.87942
expr min lq mean median uq max
1 varDiff_x_S 1.000000 1.000000 1.000000 1.00000 1.000000 1.000000
3 varDiff(x[idxs]) 1.841910 1.848677 1.897930 1.82657 2.013607 1.686027
2 varDiff(x, idxs) 1.924413 1.849609 2.533612 1.85194 2.145462 19.906425

Figure: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on integer+n = 1000000 data. Outliers are displayed as crosses. Times are in milliseconds.

Data type "double"

Data

> rvector <- function(n, mode = c("logical", "double", "integer"), range = c(-100, +100), na_prob = 0) {
+     mode <- match.arg(mode)
+     if (mode == "logical") {
+         x <- sample(c(FALSE, TRUE), size = n, replace = TRUE)
+     }     else {
+         x <- runif(n, min = range[1], max = range[2])
+     }
+     storage.mode(x) <- mode
+     if (na_prob > 0) 
+         x[sample(n, size = na_prob * n)] <- NA
+     x
+ }
> rvectors <- function(scale = 10, seed = 1, ...) {
+     set.seed(seed)
+     data <- list()
+     data[[1]] <- rvector(n = scale * 100, ...)
+     data[[2]] <- rvector(n = scale * 1000, ...)
+     data[[3]] <- rvector(n = scale * 10000, ...)
+     data[[4]] <- rvector(n = scale * 1e+05, ...)
+     data[[5]] <- rvector(n = scale * 1e+06, ...)
+     names(data) <- sprintf("n = %d", sapply(data, FUN = length))
+     data
+ }
> data <- rvectors(mode = mode)
> data <- data[1:4]

Results

n = 1000 vector

All elements

> x <- data[["n = 1000"]]
> idxs <- sample.int(length(x), size = length(x) * 0.7)
> x_S <- x[idxs]
> gc()
          used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 3249649 173.6    5709258 305.0  5709258 305.0
Vcells 7352032  56.1   25448368 194.2 87357391 666.5
> stats <- microbenchmark(varDiff_x_S = varDiff(x_S), `varDiff(x, idxs)` = varDiff(x, idxs = idxs), 
+     `varDiff(x[idxs])` = varDiff(x[idxs]), unit = "ms")

Table: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on double+n = 1000 data. The top panel shows times in milliseconds and the bottom panel shows relative times.

expr min lq mean median uq max
1 varDiff_x_S 0.010623 0.0110215 0.0113004 0.0112335 0.0114870 0.013134
2 varDiff(x, idxs) 0.012500 0.0130295 0.0133065 0.0131455 0.0133765 0.016458
3 varDiff(x[idxs]) 0.012545 0.0130485 0.0140308 0.0132810 0.0134665 0.084951
expr min lq mean median uq max
1 varDiff_x_S 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000
2 varDiff(x, idxs) 1.176692 1.182189 1.177526 1.170205 1.164490 1.253084
3 varDiff(x[idxs]) 1.180928 1.183913 1.241620 1.182267 1.172325 6.468022

Figure: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on double+n = 1000 data. Outliers are displayed as crosses. Times are in milliseconds.

n = 10000 vector

All elements

> x <- data[["n = 10000"]]
> idxs <- sample.int(length(x), size = length(x) * 0.7)
> x_S <- x[idxs]
> gc()
          used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 3249715 173.6    5709258 305.0  5709258 305.0
Vcells 7361777  56.2   25448368 194.2 87357391 666.5
> stats <- microbenchmark(varDiff_x_S = varDiff(x_S), `varDiff(x, idxs)` = varDiff(x, idxs = idxs), 
+     `varDiff(x[idxs])` = varDiff(x[idxs]), unit = "ms")

Table: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on double+n = 10000 data. The top panel shows times in milliseconds and the bottom panel shows relative times.

expr min lq mean median uq max
1 varDiff_x_S 0.036365 0.0371245 0.0377350 0.0375240 0.0379865 0.043200
3 varDiff(x[idxs]) 0.050569 0.0516530 0.0533477 0.0521385 0.0527095 0.138519
2 varDiff(x, idxs) 0.050390 0.0516445 0.0523897 0.0521690 0.0527005 0.060606
expr min lq mean median uq max
1 varDiff_x_S 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000
3 varDiff(x[idxs]) 1.390595 1.391345 1.413745 1.389471 1.387585 3.206458
2 varDiff(x, idxs) 1.385673 1.391116 1.388358 1.390284 1.387348 1.402917

Figure: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on double+n = 10000 data. Outliers are displayed as crosses. Times are in milliseconds.

n = 100000 vector

All elements

> x <- data[["n = 100000"]]
> idxs <- sample.int(length(x), size = length(x) * 0.7)
> x_S <- x[idxs]
> gc()
          used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 3249787 173.6    5709258 305.0  5709258 305.0
Vcells 7456629  56.9   25448368 194.2 87357391 666.5
> stats <- microbenchmark(varDiff_x_S = varDiff(x_S), `varDiff(x, idxs)` = varDiff(x, idxs = idxs), 
+     `varDiff(x[idxs])` = varDiff(x[idxs]), unit = "ms")

Table: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on double+n = 100000 data. The top panel shows times in milliseconds and the bottom panel shows relative times.

expr min lq mean median uq max
1 varDiff_x_S 0.267739 0.2760125 0.3771288 0.2910695 0.3280030 6.688639
3 varDiff(x[idxs]) 0.451870 0.4610180 0.5387670 0.4879550 0.5642650 1.290015
2 varDiff(x, idxs) 0.452017 0.4585500 0.5435092 0.4952310 0.6432015 0.735421
expr min lq mean median uq max
1 varDiff_x_S 1.000000 1.000000 1.000000 1.000000 1.000000 1.0000000
3 varDiff(x[idxs]) 1.687726 1.670279 1.428602 1.676421 1.720304 0.1928666
2 varDiff(x, idxs) 1.688275 1.661338 1.441176 1.701418 1.960962 0.1099508

Figure: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on double+n = 100000 data. Outliers are displayed as crosses. Times are in milliseconds.

n = 1000000 vector

All elements

> x <- data[["n = 1000000"]]
> idxs <- sample.int(length(x), size = length(x) * 0.7)
> x_S <- x[idxs]
> gc()
          used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells 3249856 173.6    5709258 305.0  5709258 305.0
Vcells 8402042  64.2   25448368 194.2 87357391 666.5
> stats <- microbenchmark(varDiff_x_S = varDiff(x_S), `varDiff(x, idxs)` = varDiff(x, idxs = idxs), 
+     `varDiff(x[idxs])` = varDiff(x[idxs]), unit = "ms")

Table: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on double+n = 1000000 data. The top panel shows times in milliseconds and the bottom panel shows relative times.

expr min lq mean median uq max
1 varDiff_x_S 3.602216 4.135621 4.994689 4.293958 4.391447 11.36425
2 varDiff(x, idxs) 9.498137 10.164840 10.696447 10.256301 10.337367 19.29023
3 varDiff(x[idxs]) 8.214287 10.213787 10.760561 10.257927 10.350938 17.58123
expr min lq mean median uq max
1 varDiff_x_S 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000
2 varDiff(x, idxs) 2.636748 2.457875 2.141564 2.388542 2.353978 1.697449
3 varDiff(x[idxs]) 2.280343 2.469711 2.154401 2.388921 2.357068 1.547064

Figure: Benchmarking of varDiff_x_S(), varDiff(x, idxs)() and varDiff(x[idxs])() on double+n = 1000000 data. Outliers are displayed as crosses. Times are in milliseconds.

Appendix

Session information

R version 3.6.1 Patched (2019-08-27 r77078)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /home/hb/software/R-devel/R-3-6-branch/lib/R/lib/libRblas.so
LAPACK: /home/hb/software/R-devel/R-3-6-branch/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] microbenchmark_1.4-6    matrixStats_0.55.0-9000 ggplot2_3.2.1          
[4] knitr_1.24              R.devices_2.16.0        R.utils_2.9.0          
[7] R.oo_1.22.0             R.methodsS3_1.7.1       history_0.0.0-9002     

loaded via a namespace (and not attached):
 [1] Biobase_2.45.0       bit64_0.9-7          splines_3.6.1       
 [4] network_1.15         assertthat_0.2.1     highr_0.8           
 [7] stats4_3.6.1         blob_1.2.0           robustbase_0.93-5   
[10] pillar_1.4.2         RSQLite_2.1.2        backports_1.1.4     
[13] lattice_0.20-38      glue_1.3.1           digest_0.6.20       
[16] colorspace_1.4-1     sandwich_2.5-1       Matrix_1.2-17       
[19] XML_3.98-1.20        lpSolve_5.6.13.3     pkgconfig_2.0.2     
[22] genefilter_1.66.0    purrr_0.3.2          ergm_3.10.4         
[25] xtable_1.8-4         mvtnorm_1.0-11       scales_1.0.0        
[28] tibble_2.1.3         annotate_1.62.0      IRanges_2.18.2      
[31] TH.data_1.0-10       withr_2.1.2          BiocGenerics_0.30.0 
[34] lazyeval_0.2.2       mime_0.7             survival_2.44-1.1   
[37] magrittr_1.5         crayon_1.3.4         statnet.common_4.3.0
[40] memoise_1.1.0        laeken_0.5.0         R.cache_0.13.0      
[43] MASS_7.3-51.4        R.rsp_0.43.1         tools_3.6.1         
[46] multcomp_1.4-10      S4Vectors_0.22.1     trust_0.1-7         
[49] munsell_0.5.0        AnnotationDbi_1.46.1 compiler_3.6.1      
[52] rlang_0.4.0          grid_3.6.1           RCurl_1.95-4.12     
[55] cwhmisc_6.6          rappdirs_0.3.1       labeling_0.3        
[58] bitops_1.0-6         base64enc_0.1-3      boot_1.3-23         
[61] gtable_0.3.0         codetools_0.2-16     DBI_1.0.0           
[64] markdown_1.1         R6_2.4.0             zoo_1.8-6           
[67] dplyr_0.8.3          bit_1.1-14           zeallot_0.1.0       
[70] parallel_3.6.1       Rcpp_1.0.2           vctrs_0.2.0         
[73] DEoptimR_1.0-8       tidyselect_0.2.5     xfun_0.9            
[76] coda_0.19-3

Total processing time was 13.25 secs.

Reproducibility

To reproduce this report, do:

html <- matrixStats:::benchmark('varDiff_subset')

Copyright Dongcan Jiang. Last updated on 2019-09-10 21:13:58 (-0700 UTC). Powered by RSP.

<script> var link = document.createElement('link'); link.rel = 'icon'; link.href = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAMAAABEpIrGAAAA21BMVEUAAAAAAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8AAP8BAf4CAv0DA/wdHeIeHuEfH+AgIN8hId4lJdomJtknJ9g+PsE/P8BAQL9yco10dIt1dYp3d4h4eIeVlWqWlmmXl2iYmGeZmWabm2Tn5xjo6Bfp6Rb39wj4+Af//wA2M9hbAAAASXRSTlMAAQIJCgsMJSYnKD4/QGRlZmhpamtsbautrrCxuru8y8zN5ebn6Pn6+///////////////////////////////////////////LsUNcQAAAS9JREFUOI29k21XgkAQhVcFytdSMqMETU26UVqGmpaiFbL//xc1cAhhwVNf6n5i5z67M2dmYOyfJZUqlVLhkKucG7cgmUZTybDz6g0iDeq51PUr37Ds2cy2/C9NeES5puDjxuUk1xnToZsg8pfA3avHQ3lLIi7iWRrkv/OYtkScxBIMgDee0ALoyxHQBJ68JLCjOtQIMIANF7QG9G9fNnHvisCHBVMKgSJgiz7nE+AoBKrAPA3MgepvgR9TSCasrCKH0eB1wBGBFdCO+nAGjMVGPcQb5bd6mQRegN6+1axOs9nGfYcCtfi4NQosdtH7dB+txFIpXQqN1p9B/asRHToyS0jRgpV7nk4nwcq1BJ+x3Gl/v7S9Wmpp/aGquum7w3ZDyrADFYrl8vHBH+ev9AUASW1dmU4h4wAAAABJRU5ErkJggg==" document.getElementsByTagName('head')[0].appendChild(link); </script>
Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️