Compressed Files and Indexed Data

  • Bioinformatics datasets are often extremely large because they contain millions or billions of DNA sequencing reads, genomic coordinates, genetic variants, gene annotations, protein sequences, and molecular measurements. Storing and transferring these datasets in their original form can require substantial disk space, network bandwidth, and computational resources. Compressed files and indexed data structures help address these challenges by reducing storage requirements and enabling efficient retrieval of specific information without processing an entire file. They are therefore essential components of modern bioinformatics workflows, particularly in next-generation sequencing, genome analysis, transcriptomics, variant analysis, and large-scale biological data management.
  • File compression is the process of reducing the amount of storage space required to represent data. Compression algorithms identify repeated patterns, redundant information, or statistical regularities and encode them more efficiently. The original data can then be reconstructed through decompression when a lossless compression method is used. Lossless compression is particularly important in bioinformatics because even a small alteration in a DNA sequence, protein sequence, genomic coordinate, or experimental measurement may affect biological interpretation. Consequently, most standard bioinformatics data compression methods are lossless.
  • Lossy compression reduces file size by permanently discarding some information. It is commonly used in image, audio, and video applications, where small changes may be acceptable. However, it is generally unsuitable for primary biological sequence data, alignment records, and variant information because discarded information may not be recoverable. Some biological imaging applications may use lossy compression for visualization or storage when the scientific requirements permit it, but the compression method and its effects must be documented.
  • Several compression algorithms and file formats are commonly encountered in bioinformatics. Gzip uses the DEFLATE compression algorithm and is widely used to compress text files, including FASTA, FASTQ, GFF, GTF, VCF, and other tabular or sequence-based data. Files compressed using gzip commonly have the .gz extension. Bzip2 uses a different compression algorithm and often provides a different balance between compression ratio and processing speed. XZ uses the LZMA2 compression algorithm and can achieve high compression ratios, although compression and decompression may require more computational resources. ZIP is a widely used archive format that can contain one or more files and directories, often with compression. These formats serve related but distinct purposes: some primarily compress individual files, while others package multiple files into an archive.
  • Gzip is particularly common in bioinformatics because it is widely supported by command-line tools, programming libraries, and biological databases. A FASTQ file containing sequencing reads, for example, may be stored as reads.fastq.gz. This reduces the amount of storage required and makes the file easier to transfer. Many sequencing analysis programs can read gzip-compressed FASTQ files directly, eliminating the need to create a separate uncompressed copy before analysis.
  • The following commands demonstrate common compression and decompression operations in Linux.

# Compress a file using gzip

gzip reads.fastq

# Decompress a gzip file

gunzip reads.fastq.gz

# Decompress while retaining the compressed file

gzip -dk reads.fastq.gz

# Compress a file using bzip2

bzip2 genome.fasta

# Decompress a bzip2 file

bunzip2 genome.fasta.bz2

# Compress a file using xz

xz genome.fasta

# Decompress an xz file

unxz genome.fasta.xz

  • The gzip command normally replaces the original uncompressed file with its compressed version. The -k option preserves the original file when supported by the installed version. The -d option performs decompression, while -c can write the output to standard output. These options are useful when building pipelines that process compressed biological data without creating unnecessary intermediate files.

# Display the contents of a gzip-compressed text file

zcat annotations.gtf.gz | head

# Search a compressed text file

zgrep “TP53” annotations.gtf.gz

# Count lines in a compressed file

zcat annotations.gtf.gz | wc -l

# Compress standard input and write to standard output

cat annotations.gtf | gzip -c > annotations.gtf.gz

  • For large biological datasets, compression can be integrated directly into computational workflows. For example, sequencing reads can be streamed from compressed FASTQ files into quality-control or alignment tools. This reduces the need for additional disk space and can simplify data processing. However, decompression still requires computational resources, and the overall performance depends on the compression algorithm, storage device, processor, and downstream program.
  • Archiving is another important aspect of biological data management. An archive combines multiple files and directories into a single package, making it easier to organize, transfer, and preserve related data. The tar utility is commonly used on Linux systems to package files. When combined with gzip, it creates a compressed archive that commonly uses the .tar.gz or .tgz extension. Bzip2- and XZ-compressed tar archives commonly use .tar.bz2 and .tar.xz, respectively.

# Create a gzip-compressed archive

tar -czf sequencing_data.tar.gz sequencing_data/

# List the contents of an archive

tar -tzf sequencing_data.tar.gz

# Extract an archive

tar -xzf sequencing_data.tar.gz

# Create an XZ-compressed archive

tar -cJf genome_project.tar.xz genome_project/

# Extract an XZ-compressed archive

tar -xJf genome_project.tar.xz

  • ZIP archives are also used for distributing biological datasets, especially when compatibility with different operating systems is required. The zip and unzip commands can create and extract ZIP archives. Researchers should distinguish between an archive that contains multiple files and a compressed single-file format. For example, a .zip file may contain several FASTA and annotation files, while a .fasta.gz file generally contains one gzip-compressed FASTA stream.
  • Although ordinary gzip compression reduces storage requirements, it does not provide efficient random access to arbitrary regions of a large compressed text file. To retrieve a particular genomic interval from a conventional gzip-compressed GFF or VCF file, software may need to decompress and scan a substantial portion of the file. This can be inefficient when researchers repeatedly request small regions from very large datasets. Indexed compression addresses this problem by dividing data into independently accessible blocks and maintaining an index that identifies where relevant records are located.
  • BGZF, or Blocked GNU Zip Format, is an important compression format developed for genomic data processing. It is based on gzip but divides the compressed data stream into a series of small blocks that can be decompressed independently. Each block contains a limited amount of uncompressed data, allowing software to access selected portions of the file without decompressing the entire dataset. BGZF is widely used in sequencing and genome analysis, particularly with BAM files and indexed genomic text files.
  • BGZF is not simply another name for ordinary gzip. Although both use gzip-compatible compression, BGZF introduces a block structure that enables efficient random access through appropriate indexing tools. A BGZF-compressed file may use the .gz extension, so the extension alone does not reliably identify whether a file is ordinary gzip or BGZF. Specialized tools can identify and process BGZF data.
  • The bgzip utility, commonly distributed with HTSlib and SAMtools, creates BGZF-compressed files. It is frequently used to compress genomic interval files such as VCF, BED, GFF, and GTF files before indexing them with Tabix.

# Create a BGZF-compressed VCF file

bgzip variants.vcf

# Create a BGZF-compressed annotation file

bgzip annotations.gff

# Decompress a BGZF-compressed file

bgzip -d variants.vcf.gz

# Display the contents without creating an uncompressed file

bgzip -dc variants.vcf.gz | head

  • A major advantage of BGZF is its compatibility with indexing systems that use genomic coordinates. Once a suitable file has been compressed and sorted, an index can be created to locate records associated with particular chromosomes and genomic intervals. This allows researchers to retrieve specific regions rapidly without scanning the entire file.
  • An index is a separate data structure that records where information is located within a file or dataset. Instead of reading every record to find a particular gene, sequence, genomic coordinate, or variant, software consults the index to identify the relevant region. Indexing is especially useful for large datasets that are queried repeatedly. The index generally depends on the organization and content of the original file, so changes to the data may require the index to be regenerated.
  • Tabix is a widely used indexing tool for BGZF-compressed, position-sorted genomic text files. It supports formats such as VCF, BED, GFF, GTF, and other coordinate-based tabular files when their structure and sorting order meet the requirements of the selected indexing configuration. Tabix creates an index that maps genomic regions to the BGZF blocks containing the corresponding records. Researchers can then retrieve specific genomic intervals using chromosome names and coordinate ranges.
  • For example, a large VCF file may contain millions of variants distributed across all chromosomes. If a researcher wants to retrieve variants from a particular region of chromosome 17, Tabix can use the index to locate the relevant records efficiently.

# Sort a VCF file using a suitable tool before compression

bcftools sort variants.vcf -Oz -o variants.sorted.vcf.gz

# Create a Tabix index

tabix -p vcf variants.sorted.vcf.gz

# Retrieve variants from a genomic interval

tabix variants.sorted.vcf.gz chr17:7668400-7687500

# List indexed genomic sequences

tabix -l variants.sorted.vcf.gz

  • In this example, bcftools sort sorts the VCF records and writes a compressed output using BGZF-compatible compression. The -p vcf option tells Tabix to use the VCF coordinate conventions when building the index. The genomic interval uses a one-based, inclusive coordinate convention, as commonly used by Tabix region queries. Chromosome naming must also match the names used in the file. For example, chr17 and 17 are not automatically interchangeable.
  • Tabix can also index other coordinate-based formats. For BED files, the indexing configuration is different because BED uses zero-based, half-open intervals. GFF and GTF files use one-based, inclusive coordinates. The correct preset or configuration must therefore be selected for the file being indexed. Incorrect coordinate conventions or inconsistent sorting can result in missing records or incorrect retrieval.

# Compress a BED file using BGZF

bgzip regions.bed

# Create a BED index

tabix -p bed regions.bed.gz

# Retrieve records from a region

tabix regions.bed.gz chr1:100000-200000

  • The Tabix index allows genomic interval queries to be performed efficiently, but it does not replace the underlying data file. Both the compressed data and its index are required for indexed retrieval. If the data file is modified, sorted differently, or recompressed, the old index may no longer be valid and should be regenerated. Index files commonly have extensions such as .tbi or .csi, depending on the indexing method.
  • CSI, or Coordinate-Sorted Index, is an alternative indexing format used for genomic coordinate data. It supports larger coordinate ranges than the traditional Tabix TBI index and is useful for datasets whose reference sequences or coordinates exceed the limits supported by TBI. CSI is supported by several tools in the HTSlib ecosystem. The appropriate index type depends on the coordinate range, file format, and software being used.
  • Indexed data are also fundamental to sequence alignment formats. SAM is a text-based format for storing sequence alignment records, while BAM is its binary compressed counterpart. CRAM is another alignment format designed to reduce storage requirements by using reference-based compression where appropriate. BAM and CRAM files are commonly indexed so that software can retrieve alignments overlapping a selected genomic region without reading every alignment record.
  • SAMtools provides commands for sorting, indexing, and querying BAM and CRAM files. A typical workflow begins by sorting an alignment file by reference sequence and coordinate, then creating an index.

# Convert SAM to BAM and sort by genomic coordinate

samtools sort -o sample.sorted.bam sample.sam

# Create a BAM index

samtools index sample.sorted.bam

# Retrieve alignments from a genomic region

samtools view sample.sorted.bam chr1:100000-200000

# Display alignment file information

samtools flagstat sample.sorted.bam

# View the index-related files

ls sample.sorted.bam*

  • The index created by samtools index is commonly stored as a .bai file, although CSI can be used when a larger coordinate range is required. A coordinate-sorted BAM file is necessary for conventional regional indexing. Name-sorted BAM files, which group records by read name rather than genomic coordinate, are used for other operations and are not interchangeable with coordinate-sorted files for this purpose.
  • CRAM files can also be indexed and queried by genomic region. Because CRAM commonly uses a reference genome to improve compression, the correct reference sequence must be available when required by the software. Reference identification and version management are therefore important. If the wrong reference genome is supplied, decoding or interpretation may fail or produce incorrect results. CRAM workflows should preserve information about the reference sequence used during encoding.
  • Reference sequence indexing is another important application. FASTA files store nucleotide or protein sequences as text, but searching for a particular sequence region in a very large FASTA file can be inefficient. The samtools faidx utility creates an index that records sequence names, lengths, and file positions. This index enables rapid retrieval of selected subsequences without reading the entire FASTA file.
  • The following commands demonstrate FASTA indexing and sequence retrieval.

# Create an index for a FASTA file

samtools faidx reference.fa

# List indexed sequence names and lengths

cut -f1,2 reference.fa.fai

# Retrieve an entire chromosome

samtools faidx reference.fa chr1

# Retrieve a specific region

samtools faidx reference.fa chr1:100000-200000

  • The FASTA index is commonly stored in a .fai file. It contains information such as sequence names, sequence lengths, byte offsets, and the number of bases and bytes per sequence line. These details allow software to calculate where a requested region is located in the original file. The underlying FASTA file must remain unchanged after indexing. Modifying line wrapping, sequence content, or other relevant file properties can invalidate the index.
  • Another example of indexed biological data is the two-bit sequence format. The .2bit format stores DNA sequences in a compact binary representation and supports efficient access to selected regions. It is used in genome browsers and genome sequence resources, including some UCSC Genome Browser workflows. The format can represent ambiguous bases and masked regions while storing nucleotide sequences more compactly than plain-text FASTA. Specialized utilities are required to create, inspect, and retrieve sequences from two-bit files.
  • Genomic interval indexes are also used by genome browsers and analytical tools to retrieve overlapping features. BED files, for example, may contain genomic regions such as promoters, exons, regulatory elements, and sequencing peaks. Indexing enables software to identify which intervals overlap a query region without comparing the query against every record. Different tools use different index structures, including bin-based indexes, interval trees, and other coordinate-search methods.
  • Indexing is not limited to genomic coordinates. Biological databases use indexes to retrieve records by accession number, gene identifier, protein identifier, organism, keyword, or other attributes. Relational databases may use B-tree or hash indexes, while specialized sequence-search systems use data structures such as suffix arrays, k-mer indexes, and other sequence-search indexes. These approaches serve different purposes. A Tabix index locates genomic records by coordinate, a FASTA index locates sequence regions within a file, and a database index supports searches across structured records.
  • Compressed and indexed data are particularly important in next-generation sequencing workflows. FASTQ files are commonly compressed to reduce storage and transfer requirements. Alignment tools may read compressed FASTQ files directly, while resulting BAM or CRAM files are sorted and indexed for regional analysis. Variant callers generate VCF files, which can be compressed with BGZF and indexed using Tabix or CSI. Genome annotation files may also be compressed and indexed so that genes, transcripts, exons, and other features can be retrieved from selected regions. These steps support efficient analysis of large sequencing projects.
  • Compression and indexing also contribute to reproducibility and data integrity. Researchers should preserve the original data, document the compression method, record software versions, and maintain the indexes associated with the relevant files. Checksums such as MD5 or SHA-256 can be used to verify whether files have changed during transfer or storage. A checksum does not establish that a biological dataset is correct, but it can help confirm that a file has been transferred or preserved without unintended modification.
  • It is important to understand that compressed files and indexed files are not necessarily interchangeable concepts. Compression reduces storage requirements, while indexing accelerates access to selected information. A compressed file may have no index, and an indexed file may be uncompressed or use a particular compression method required by its indexing software. Some indexes depend on the exact byte positions and layout of the original file. Consequently, recompressing, sorting, editing, or rewriting a file may invalidate its index even if the biological records appear unchanged.
  • The selection of a compression and indexing strategy depends on the data type and intended analysis. Ordinary gzip compression is suitable for many text-based biological files when sequential reading is sufficient. BGZF is useful when compressed genomic text files must support random access through Tabix or related tools. BAM and CRAM are appropriate for storing sequencing alignments, with indexing enabling regional retrieval. FASTA indexing supports rapid access to reference sequence regions, while specialized binary formats such as two-bit provide compact sequence storage and efficient retrieval. Researchers should choose tools that match the file format, coordinate conventions, software ecosystem, and expected access patterns.
  • Compressed files and indexed data are fundamental to efficient bioinformatics because they make large biological datasets easier to store, transfer, search, and analyze. Understanding gzip, BGZF, ZIP and tar archives, Tabix, CSI, BAM and CRAM indexes, and FASTA indexing enables researchers to manage sequencing data and genomic resources more effectively. These techniques reduce computational overhead, support scalable analysis, and provide the foundation for many modern genome-analysis pipelines.
Author:

Leave a Reply

Your email address will not be published. Required fields are marked *