BEDMatrix

The BEDMatrix package allows access to genotypes from a BED file without loading the entire matrix in memory. This is very convenient for large datasets, as long as only small portions of the data are pulled at once. However, there are some disadvantages:

• The BEDMatrix return object is not a regular R matrix, which confuses some users.
• The BEDMatrix package provides no way to access the full annotation tables (BIM and FAM files).
  • From BIM, only the locus ID and reference allele are returned, merged into a vector of strings and placed as column names.
  • From FAM, only the family and individual IDs are returned, merged into a vector of strings and placed as row names.
• BEDMatrix does not have any write functions.

Here is an example of that usage and how the data compares. Note in particular that the BEDMatrix representation of the genotype matrix is transposed compared to the genio matrix:

library(BEDMatrix)
# Time it too.
# Although the BIM and FAM tables are not returned,
# they are partially parsed and kept in memory,
# which can take time for extremely large files
time_read_bedmatrix_1 <- system.time(
    X_BEDMatrix <- BEDMatrix(file_plink)
)
#> Extracting number of samples and rownames from vignette-random-data778e5ef14ce6.fam...
#> Extracting number of variants and colnames from vignette-random-data778e5ef14ce6.bim...

# Inspect the first 10 loci and individuals as usual.
# Note it is transposed compared to our X.
# Also note the column and row names are different from genio's.
X_BEDMatrix[1:10, 1:10]
#>            rs1_G rs2_C rs3_G rs4_C rs5_C rs6_G rs7_G rs8_G rs9_G rs10_C
#> fam1_id1       2    NA     1     1     0     2     1     1     0      2
#> fam2_id2       1     2    NA     2     1     1     2     1     1     NA
#> fam3_id3       0     0     2     1     1    NA     2     1     0      1
#> fam4_id4       2     1     1     1    NA     0     0     1    NA      1
#> fam5_id5       2    NA     1     0     2     1     1    NA    NA      1
#> fam6_id6      NA     0     0     2     0     1     1    NA     0      1
#> fam7_id7       1     1     2     2    NA     0     0     2     2     NA
#> fam8_id8       1     1    NA     2     2     0    NA     2     1      1
#> fam9_id9      NA     1     1     0     1     0     2     0     1      2
#> fam10_id10     1     2     1    NA     0     2     2     1     1     NA

Therefore, for very large datasets, if it suffices to access the genotype data in small portions and the user is willing to deal with the limitations of the object, and no detailed annotation table information is required, then BEDMatrix is a better solution than the read_bed function provided in the genio package.

To compare the two genotype reading functions on an equal footing, let us assume that the entire genotype matrix is required in memory and stored in a regular R matrix.

# This turns it into a regular R matrix.
# Since most of the reading is actually happening now,
# we time this step now.
time_read_bedmatrix_2 <- system.time(
    X_BEDMatrix_Rmat <- as.matrix(X_BEDMatrix)
)
time_read_bedmatrix_2
#>    user  system elapsed 
#>   0.070   0.007   0.078
# Now we can test that the BEDMatrix agrees with the original matrix we simulated.
# Need to transpose first.
stopifnot( all( X == t(X_BEDMatrix_Rmat), na.rm = TRUE) )

snpStats

The snpStats package is the most fully-featured alternative to genio. One of its advantages is its memory efficiency, encoding the entire data in less memory than a native R matrix. However, this memory efficiency comes at the cost of making the data harder to access, especially for inexperienced R users. Like genio, and in contrast to the other packages, snpStats also reads the BIM and FAM tables fully, and provides a BED writer, but it is considerably harder to use.

First we illustrate data parsing. The annotation tables are similar but column names are different, and certain missing values (zero in text files) are converted to NA instead.

library(snpStats)
#> Loading required package: survival
#> Loading required package: Matrix

# Read data, time it.
time_read_snpStats_1 <- system.time(
    data_snpStats <- read.plink(file_plink)
)
time_read_snpStats_1
#>    user  system elapsed 
#>   0.030   0.003   0.032

# Inspect the data

# Genotypes.
# Note data is not visualized this way.
# This matrix is also transposed compared to the genio matrix.
data_snpStats$genotypes
#> A SnpMatrix with  1001 rows and  10001 columns
#> Row names:  fam1 ... fam1001 
#> Col names:  rs1 ... rs10001

# Locus annotations
head( data_snpStats$map )
#>     chromosome snp.name cM position allele.1 allele.2
#> rs1       chr1      rs1 NA     1000        G        T
#> rs2       chr1      rs2 NA     2000        C        A
#> rs3       chr1      rs3 NA     3000        G        A
#> rs4       chr1      rs4 NA     4000        C        T
#> rs5       chr1      rs5 NA     5000        C        T
#> rs6       chr1      rs6 NA     6000        G        T

# Individual annotations
head (data_snpStats$fam )
#>      pedigree member father mother sex    affected
#> fam1     fam1    id1     NA     NA   2 -0.89945429
#> fam2     fam2    id2     NA     NA   2  1.48550249
#> fam3     fam3    id3     NA     NA   2 -1.27863368
#> fam4     fam4    id4     NA     NA   2 -0.05182039
#> fam5     fam5    id5     NA     NA   1  0.45474930
#> fam6     fam6    id6     NA     NA   2 -0.43133813

As for BEDMatrix, assuming we ultimately desire to convert the entire data into a regular R matrix, an extra step is required:

# Transpose, then convert to a regular R matrix.
# Let's time this step too.
time_read_snpStats_2 <- system.time(
    X_snpStats <- as( t(data_snpStats$genotypes), 'numeric')
)
time_read_snpStats_2
#>    user  system elapsed 
#>   0.071   0.015   0.086

# Now we can visualize the matrix.
# First 10 loci and individuals, as before.
# Note that, compared to (genio, BEDMatrix, lfa), alleles are encoded in reverse,
# so 0s and 2s are flipped in this matrix.
X_snpStats[1:10, 1:10]
#>      fam1 fam2 fam3 fam4 fam5 fam6 fam7 fam8 fam9 fam10
#> rs1     0    1    2    0    0   NA    1    1   NA     1
#> rs2    NA    0    2    1   NA    2    1    1    1     0
#> rs3     1   NA    0    1    1    2    0   NA    1     1
#> rs4     1    0    1    1    2    0    0    0    2    NA
#> rs5     2    1    1   NA    0    2   NA    0    1     2
#> rs6     0    1   NA    2    1    1    2    2    2     0
#> rs7     1    0    0    2    1    1    2   NA    0     0
#> rs8     1    1    1    1   NA   NA    0    0    2     1
#> rs9     2    1    2   NA   NA    2    0    1    1     1
#> rs10    0   NA    1    1    1    1   NA    1    0    NA

# Again verify that the matrices are identical.
# (Here 2-X flips back 0s and 2s)
stopifnot( all( X == 2 - X_snpStats, na.rm = TRUE) )

snpStats is the only package other than genio to provide a BED writer. Here we demonstrate how hard it is to use it to write our data.

# Let's write this to another file
file_plink_copy <- tempfile('vignette-random-data-copy')

# Copy objects to not change originals
X_snpStats <- X
bim_snpStats <- as.data.frame(bim) # to use rownames
fam_snpStats <- as.data.frame(fam) # ditto

# All data requires matching row and/or column names.
# These first two were already done above.
#rownames(X_snpStats) <- bim$id
#colnames(X_snpStats) <- fam$id
# Row names here are redundant but required.
rownames(bim_snpStats) <- bim$id
rownames(fam_snpStats) <- fam$id

# We shall time several required steps in order to write genotypes in a standard R matrix,
# and the related annotation tables, to BED.
time_write_snpStats <- system.time({
    # Transpose and convert our genotypes to SnpMatrix object.
    # We flip 0s and 2s before converting
    X_snpStats <- as(2 - t(X_snpStats), 'SnpMatrix')
    
    # This complicated command is required to write the data.
    # Although X, fam, and bim are passed as for genio's write_plink,
    # here the name of every column must be specified (there are no reasonable defaults).
    # Interestingly, the parameter names of snpStats' write.plink
    # do not agree with read.plink from the same package.
    write.plink(
        file_plink_copy,
        snps = X_snpStats,
        subject.data = fam_snpStats,
        pedigree = fam,
        id = id,
        father = pat,
        mother = mat,
        sex = sex,
        phenotype = pheno,
        snp.data = bim_snpStats,
        chromosome = chr,
        genetic.distance = posg,
        position = pos,
        allele.1 = ref,
        allele.2 = alt
    )
})
#> Writing FAM file to /tmp/RtmpMsCEXK/vignette-random-data-copy778e12960a6c.fam 
#> Writing extended MAP file to /tmp/RtmpMsCEXK/vignette-random-data-copy778e12960a6c.bim 
#> Writing BED file to /tmp/RtmpMsCEXK/vignette-random-data-copy778e12960a6c.bed (SNP-major mode)

# remove the new file, no longer need it
delete_files_plink(file_plink_copy)

LFA

The lfa package also provides a minimal BED parser, ignoring the BIM and FAM files except to count loci and individuals. Its genotype matrix is exactly like genio’s: same class (R matrix) and orientation (loci along rows). The package does not provide file writers or other such utilities.

library(lfa)
# Parse the genotype matrix
time_read_lfa <- system.time(
    X_lfa <- read.bed(file_plink)
)
#> [1] "reading in 1001 individuals"
#> [1] "reading in 10001 snps"
#> [1] "snp major mode"
time_read_lfa
#>    user  system elapsed 
#>   0.206   0.059   0.267

# This genotype matrix does not have column or row names:
X_lfa[1:10, 1:10]
#>       [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
#>  [1,]    2    1    0    2    2   NA    1    1   NA     1
#>  [2,]   NA    2    0    1   NA    0    1    1    1     2
#>  [3,]    1   NA    2    1    1    0    2   NA    1     1
#>  [4,]    1    2    1    1    0    2    2    2    0    NA
#>  [5,]    0    1    1   NA    2    0   NA    2    1     0
#>  [6,]    2    1   NA    0    1    1    0    0    0     2
#>  [7,]    1    2    2    0    1    1    0   NA    2     2
#>  [8,]    1    1    1    1   NA   NA    2    2    0     1
#>  [9,]    0    1    0   NA   NA    0    2    1    1     1
#> [10,]    2   NA    1    1    1    1   NA    1    2    NA

# Again verify that the matrices are identical
stopifnot( all( X == X_lfa, na.rm = TRUE) )

Overall time comparisons

Note that reading performance varies on different machines depending on the balance between hard drive access and processor speeds (where the relative bottleneck is). That being said, the genio reader is consistently the fastest, if not a close second (see below for tests on your machine), for several reasons. Genotypes are read very efficiently using Rcpp code, and stored directly into an R matrix, which is most efficient if that is the end goal. The annotation tables are also read most efficiently, using the readr package internally.

The BEDMatrix reader is also written in Rcpp, but its main goal is to provide efficient random access to genotypes stored in a file, which makes obtaining an R matrix more awkward, though surprisingly without paying a time penalty for it. The annotation tables are read with data.table, which is actually the fastest, though the difference in performance is small compared to readr for reasonably-sized files. However, BEDMatrix does not process or return full annotation tables, which gives it an unfair advantage compared to genio, as BEDMatrix takes shortcuts to only read the columns it needs. If the annotation tables are needed, reading them will incur an additional (and in some cases considerable) time penalty.

The snpStats reader is written in C, so it is also very fast for its initial step, but converting the genotypes from the snpStats format into a native R matrix proves too expensive. The annotation tables are read with the base function read.table, which is also the slowest. In terms of completeness of output (full genotypes and annotations tables), only this package matches genio, which makes it the fairest comparison.

The lfa reader is written in pure R, which explains why it is the slowest in this comparison. The annotation tables are read with read.table, as in snpStats, but are not returned at all.

# Extract component 3 of each time object,
# which is is total time elapsed.
# Sum the two steps it takes for each of BEDMatrix and snpStats to obtain a native R matrix.
times_read <- c(
    time_read_genio[3],
    time_read_bedmatrix_1[3] + time_read_bedmatrix_2[3],
    time_read_snpStats_1[3] + time_read_snpStats_2[3],
    time_read_lfa[3]
)
names_read <- c(
    'genio',
    'BEDMatrix',
    'snpStats',
    'lfa'
)
# Create barplot
barplot(
    times_read,
    names.arg = names_read,
    main = 'BED reader runtimes',
    xlab = 'packages',
    ylab = 'runtime (s)'
)

Now we compare the writers. Only snpStats and genio have a BED writer. Not only was the genio writer much easier to use, it is also considerably faster.

times_write <- c(
    time_write_genio[3],
    time_write_snpStats[3]
)
names_write <- c(
    'genio',
    'snpStats'
)
# Create barplot
barplot(
    times_write,
    names.arg = names_write,
    main = 'BED writer runtimes',
    xlab = 'packages',
    ylab = 'runtime (s)'
)

Memory comparison

High memory consumption is the main sacrifice for the ease of use of native R matrices, and in this respect genio consumes the most memory (second only to lfa).

We use the pryr package to compare the memory usage of each native genotype object.

library(pryr)
# Store directly into a vector
sizes <- c(
    object_size( X ),
    object_size( data_genio$X ),
    object_size( X_BEDMatrix ),
    object_size( data_snpStats$genotypes ),
    object_size( X_lfa )
)
names_sizes <- c(
    'original',
    'genio',
    'BEDMatrix',
    'snpStats',
    'lfa'
)
# Create barplot
barplot(
    sizes,
    names.arg = names_sizes,
    main = 'Native genotype object sizes',
    xlab = 'packages',
    ylab = 'memory (bytes)'
)

The BEDMatrix package is ideal for very large files, since data is loaded into memory only as needed. So the main object does not actually hold any genotypes in memory. It is up to the user to decide how much data to access at once to trade-off speed and memory consumption.

snpStats manages memory better than genio, but strictly by a factor of 4. Each genotype is encoded as an integer in R (in general) and genio (in particular), which uses 4 bytes (this is actually platform dependent). On the other hand, snpStats uses strictly one byte per genotype.

lfa uses 8 bytes per genotype because its genotype matrix treats is elements as doubles rather than integers, so it uses twice as much memory as genio.

Overall, there is a narrow window in data sizes in which a genotype matrix is too large for a native R matrix but small enough for a snpStats object. That, and the much more complex interface of snpStats, makes it not very worthwhile in my opinion, unless there is need for functions provided only by that package. I prefer to use BEDMatrix for large datasets.