- Biological data formats and file handling are fundamental aspects of bioinformatics because biological research generates large amounts of information that must be stored, organized, transferred, processed, and analyzed using computational tools. These data may include DNA and RNA sequences, protein sequences, genome assemblies, gene annotations, genetic variants, gene expression measurements, protein structures, molecular interactions, and experimental metadata. Each type of biological information requires an appropriate format that preserves its structure and meaning while allowing compatible software systems to read and process it.
- A biological data format defines how information is organized within a file. It specifies the structure of records, the representation of biological sequences or measurements, the use of identifiers, and the arrangement of associated annotations and metadata. Some formats are designed to store simple sequence information, while others support complex genomic coordinates, alignment information, variant calls, or molecular structures. Selecting the correct format is important because different tools may support different formats, and incorrect format handling can result in data loss, processing errors, or misleading analytical results.
- Biological data formats can be broadly divided into sequence formats, sequence annotation formats, alignment formats, variant formats, expression data formats, structural biology formats, and general-purpose data formats. Common examples include FASTA, FASTQ, GenBank, EMBL, GFF3, GTF, SAM, BAM, CRAM, VCF, BED, PDB, mmCIF, CSV, TSV, JSON, and XML. Some formats are human-readable text files, while others are compressed, binary, or structured formats designed for efficient storage and computational processing.
- FASTA is one of the most widely used formats for storing nucleotide and protein sequences. A FASTA record begins with a header line marked by the greater-than symbol (>), followed by an identifier and optional descriptive information. The biological sequence is written on the subsequent lines. A FASTA file can contain a single sequence or multiple records, making it useful for storing genome sequences, individual genes, transcripts, and protein sequences.
- A simple FASTA example is shown below.
>gene_001 Example nucleotide sequence
ATGGCTAGCTAGCTAGCTAG
ATCGATCGATCGATCGATCG
>protein_001 Example protein sequence
MKTLLVLLAVAVATA
- FASTA is simple and widely supported by bioinformatics tools, including sequence alignment programs, genome assembly software, and sequence analysis libraries. However, standard FASTA does not provide a universal structure for storing detailed sample metadata, sequencing quality scores, or complex genomic annotations. Additional information may need to be stored in separate files or in more specialized formats.
- FASTQ is commonly used to store raw sequencing reads generated by high-throughput sequencing technologies. Unlike FASTA, FASTQ contains both the nucleotide sequence and a quality score for each base. Each record normally consists of four lines: a header beginning with @, the sequence, a separator line beginning with +, and a quality string. The quality string represents the confidence associated with the base calls, usually using a Phred quality score encoding.
An illustrative FASTQ record is shown below.
@read_001
ATGCGTACGTAGCTAG
+
IIIIIIIIIIIIIIII
- FASTQ files are essential for sequencing data analysis because quality scores help identify unreliable bases and guide quality-control procedures. Tools such as FastQC, fastp, and Cutadapt can examine FASTQ files, identify quality problems, and perform operations such as adapter trimming and filtering. FASTQ files are often very large, so they are commonly compressed using gzip and stored with the .fastq.gz or .fq.gz extension.
- GenBank and EMBL formats store nucleotide sequence records together with extensive annotations. These formats can contain sequence identifiers, organism information, references, gene features, coding regions, protein translations, and other biological annotations. GenBank is commonly associated with NCBI, while the European Nucleotide Archive provides records in EMBL-compatible formats. These formats are useful when researchers need both the nucleotide sequence and its associated biological information.
- A GenBank record may contain fields such as LOCUS, DEFINITION, ACCESSION, VERSION, SOURCE, FEATURES, and ORIGIN. The FEATURES section describes annotated regions, such as genes, coding sequences, regulatory regions, and other sequence features. The ORIGIN section contains the nucleotide sequence. GenBank and EMBL records are particularly useful for sequence annotation, comparative genomics, and examining the biological context of nucleotide sequences.
- Genome annotation is commonly represented using GFF3 and GTF formats. These formats describe features located on genomic sequences, including genes, transcripts, exons, coding sequences, untranslated regions, and other annotated elements. Both formats use tab-separated fields, but they differ in their conventions and attribute structures. GFF3 uses a standardized nine-column structure and supports hierarchical relationships through attributes such as ID and Parent. GTF also uses nine columns and is widely used in transcriptomics workflows, with attributes commonly including gene identifiers and transcript identifiers.
- A simplified GFF3 example is shown below.
##gff-version 3
chr1 example gene 1000 5000 . + . ID=gene001;Name=GeneA
chr1 example mRNA 1000 5000 . + . ID=transcript001;Parent=gene001
chr1 example exon 1000 1500 . + . ID=exon001;Parent=transcript001
A simplified GTF example is shown below.
chr1 example exon 1000 1500 . + . gene_id “gene001”; transcript_id “transcript001”;
- GFF3 and GTF files are widely used in genome annotation, RNA sequencing analysis, transcript reconstruction, and gene expression studies. It is important to verify the reference genome assembly and coordinate conventions associated with these files. Genomic coordinates in common annotation formats are generally one-based and inclusive, but conversion to other formats may require adjustment.
- BED is another widely used format for representing genomic regions. It is commonly used to describe intervals such as genes, promoters, enhancers, peaks from chromatin assays, and other regions of interest. The basic BED format contains at least three columns: chromosome, start position, and end position. Unlike GFF3 and GTF, standard BED coordinates are zero-based and half-open, meaning that the start position is included and the end position is excluded. This difference is important when converting genomic coordinates between formats because incorrect conversion can shift genomic regions by one base.
- A simple BED example is shown below.
chr1 999 1500 region_001
chr1 2000 2500 region_002
- BED files are supported by tools such as BEDTools and bedops and are commonly used for genomic interval operations, including overlap detection, intersection, merging, and comparison of genomic regions.
- Sequence alignment data are often stored in SAM, BAM, and CRAM formats. SAM is a text-based format that records the alignment of sequencing reads to a reference genome. It contains information such as read identifiers, mapping positions, alignment flags, mapping quality, CIGAR strings, and sequence information. BAM is the compressed binary representation of SAM and is more efficient for storage and computational processing. CRAM is a reference-based compressed format that can reduce storage requirements further by using a reference sequence during compression and decoding.
- BAM and CRAM files are commonly accompanied by index files, which allow software to retrieve alignments from selected genomic regions without reading the entire file. SAMtools and related tools are widely used to manipulate, sort, index, and inspect these files. The reference genome used for alignment and, when applicable, CRAM decoding must be correctly identified. Without the appropriate reference information, some CRAM files may not be decoded as intended.
- Variant Call Format (VCF) is used to represent genetic variants identified through sequencing or other genotyping methods. VCF records can describe single-nucleotide variants, insertions, deletions, and other types of genomic variation. The format contains a header describing metadata and sample information, followed by tab-separated variant records. Common fields include chromosome, position, reference allele, alternate allele, quality, filtering status, and additional annotations. VCF can also contain genotype information for individual samples.
- A simplified VCF example is shown below.
##fileformat=VCFv4.3
#CHROM POS ID REF ALT QUAL FILTER INFO
chr1 1000 rs123 A G 60 PASS DP=40
- VCF files are used in variant calling, population genetics, genome-wide association studies, and clinical genomics. The binary counterpart, BCF, provides a more compact representation suitable for efficient computational processing. Tools such as BCFtools and GATK support VCF and related variant-analysis operations. Researchers should record the reference genome assembly and ensure that chromosome names, coordinates, and allele representations are consistent when comparing variant records from different sources.
- Gene expression and other high-throughput measurements are often stored in tabular formats such as CSV and TSV. These formats organize data into rows and columns, with rows representing genes, transcripts, samples, or other biological entities and columns representing measurements or metadata. For example, a gene expression matrix may contain gene identifiers in the first column and expression measurements for different samples in subsequent columns. CSV uses commas as separators, while TSV uses tabs. TSV is frequently used in bioinformatics because biological annotations and descriptions may contain commas.
- An illustrative gene expression table is shown below.
gene_id sample_1 sample_2 sample_3
gene001 12.5 18.2 15.7
gene002 4.3 6.1 5.8
gene003 0.0 2.4 1.9
- Structured formats such as JSON and XML are useful for storing nested metadata, database records, and API responses. JSON represents information using objects, arrays, and key-value pairs, while XML uses nested elements and attributes. These formats are particularly useful when biological records contain complex relationships or when data are exchanged between web services. They can be processed using standard programming libraries in Python and R.
- Protein structures are commonly stored in PDB and mmCIF formats. PDB is a traditional text-based format that represents atomic coordinates and structural information for proteins, nucleic acids, and molecular complexes. The macromolecular Crystallographic Information File (mmCIF) format supports a more flexible and extensive representation of structural data and is used by the Protein Data Bank as its standard archival format. Structural files can include information about atoms, residues, chains, ligands, experimental methods, and structural quality. These formats are important in structural bioinformatics, molecular modeling, docking, and protein function analysis.
- File compression is an important part of biological data handling because sequencing and other high-throughput experiments can generate very large datasets. Common compression tools include gzip, bzip2, and xz. Compressed files reduce storage requirements and transfer times, although the compression method can affect access speed and compatibility with software. Some bioinformatics tools can read compressed files directly, while others require decompression before analysis. Specialized formats such as BAM, CRAM, and BCF already use compression internally and should be handled with compatible software rather than treated as ordinary text files.
- The Linux command line provides several useful tools for managing biological data files. Commands such as ls, cp, mv, rm, mkdir, head, tail, less, wc, and file help researchers organize and inspect files. Text-processing tools such as grep, cut, sort, awk, and sed can be used to examine or transform structured text files. However, generic text-processing commands should be applied carefully to specialized formats because they may unintentionally remove required fields, alter formatting, or break relationships between records.
- The following commands illustrate common file-handling operations in a bioinformatics environment.
# Create a project directory
mkdir -p bioinformatics_project/data
# Copy a FASTQ file into the data directory
cp sample.fastq.gz bioinformatics_project/data/
# List files and their sizes
ls -lh bioinformatics_project/data/
# Inspect the first lines of a compressed FASTQ file
zcat sample.fastq.gz | head -n 8
# Count lines in an uncompressed text file
wc -l annotation.gff3
# Check the file type
file sample.fastq.gz
- File inspection should be performed before analysis to confirm that the file is in the expected format, is not empty, and has not been corrupted. For text-based files, researchers can inspect the header, record structure, delimiters, and line endings. For binary files, specialized tools should be used to examine their contents. For example, samtools view can inspect SAM/BAM alignment records, while bcftools view can inspect VCF and BCF variant files. A file extension alone does not guarantee that a file contains valid data in the corresponding format.
- Python provides several libraries for reading and processing biological files. Biopython supports common sequence formats such as FASTA, FASTQ, GenBank, and EMBL. The pandas library is useful for tabular data, while libraries such as pysam support SAM, BAM, and VCF-related workflows. The following example demonstrates reading a FASTA file using Biopython and reporting basic information about each sequence.
from Bio import SeqIO
for record in SeqIO.parse(“sequences.fasta”, “fasta”):
print(“Identifier:”, record.id)
print(“Description:”, record.description)
print(“Sequence length:”, len(record.seq))
The following example reads a tab-separated gene expression table using pandas.
import pandas as pd
expression_data = pd.read_csv(
“gene_expression.tsv”,
sep=”\t”
)
print(expression_data.head())
print(expression_data.shape)
print(expression_data.isnull().sum())
- R also provides useful tools for biological file handling. Functions such as read.csv(), read.delim(), and write.table() can process tabular data. Bioconductor provides specialized packages for genomic ranges, sequence data, expression matrices, and other biological data structures. For example, the GenomicRanges package supports operations on genomic intervals, while packages such as Biostrings provide facilities for working with biological sequences.
# Read a tab-separated expression matrix
expression_data <- read.delim(
“gene_expression.tsv”,
header = TRUE,
stringsAsFactors = FALSE
)
# Inspect the data
head(expression_data)
dim(expression_data)
# Check missing values
colSums(is.na(expression_data))
- File naming and directory organization are also important for reliable data handling. A clear naming system helps distinguish raw data, processed data, intermediate results, reference files, and final outputs. Researchers should avoid ambiguous names such as final2, new_data, or latest_results. Instead, filenames should communicate the sample, data type, processing stage, and, where useful, the date or version. A structured project directory might contain separate folders for raw data, reference files, metadata, processed data, analysis scripts, logs, and results.
- Raw biological data should generally be preserved separately from processed data. Original sequencing reads, downloaded reference sequences, and source annotation files should not be overwritten during analysis. Processing steps should generate new output files, allowing researchers to trace how the original data were transformed. This separation is particularly important when analyses involve quality filtering, sequence trimming, alignment, normalization, variant filtering, or annotation.
- File integrity and provenance should be maintained through checksums and metadata records. A checksum is a value calculated from a file using a hashing algorithm such as SHA-256. It can be used to verify that a downloaded or transferred file has not been unintentionally modified. Researchers should record the source of each file, retrieval date, accession number, database release, file format, file size, checksum, and relevant processing information. These records support data provenance, reproducibility, and reliable collaboration.
- The following Linux commands demonstrate how to calculate and verify a SHA-256 checksum.
# Calculate a checksum
sha256sum sample.fastq.gz
# Save the checksum to a file
sha256sum sample.fastq.gz > sample.sha256
# Verify file integrity
sha256sum -c sample.sha256
- Data validation is necessary because biological files may contain missing values, duplicated records, inconsistent identifiers, invalid characters, incorrect coordinates, or malformed records. Validation should be appropriate to the format and biological data type. FASTA files should contain valid sequence characters and properly structured headers. FASTQ files should have matching sequence and quality-string lengths for every read. GFF3 and GTF files should have valid fields and consistent feature relationships. VCF files should contain correctly structured headers and variant records. Alignment files should be checked for appropriate sorting and indexing when required by downstream tools.
- Reference information must also be handled carefully. Genome assemblies, transcript annotations, protein reference sequences, and other reference resources can change over time. A file containing genomic coordinates is meaningful only when its reference assembly and coordinate conventions are known. For example, coordinates from GRCh37 and GRCh38 should not be combined without appropriate conversion and validation. Similarly, a gene annotation file should be matched to the genome assembly for which it was generated. Recording reference versions prevents incorrect comparisons and interpretation.
- Large-scale bioinformatics projects may involve millions of sequence reads, thousands of samples, or extensive collections of genomic variants. Efficient file handling therefore requires attention to storage capacity, memory use, processing time, and data transfer. Compression, indexing, streaming, and parallel processing can improve performance. Indexed formats allow researchers to retrieve selected regions without reading entire files, while streaming enables some tools to process data directly from input pipelines without creating unnecessary intermediate files. Cloud storage and high-performance computing environments may also be useful for projects involving very large datasets.
- File handling is closely connected to data provenance, reproducible retrieval, database integration, and the FAIR principles. Standardized formats improve interoperability, persistent identifiers and metadata improve findability, documented access methods support accessibility, and appropriate licensing and processing documentation improve reusability. Reproducible workflows should record the software versions, input files, reference resources, parameters, and transformations used at each stage of an analysis. Workflow systems such as Snakemake and Nextflow can help automate these steps and maintain a traceable relationship between inputs and outputs.
- Biological data formats and file handling are therefore essential foundations of bioinformatics. Understanding the structure and purpose of formats such as FASTA, FASTQ, GenBank, GFF3, GTF, BED, SAM, BAM, CRAM, VCF, CSV, JSON, PDB, and mmCIF enables researchers to select appropriate tools and manage biological information correctly. Careful file organization, validation, compression, indexing, versioning, and integrity checking help prevent errors and preserve the quality of research data. These practices support efficient analysis, reliable data exchange, reproducibility, and the integration of biological information across different databases and computational systems.