HDF5, AnnData, and Other Large-Scale Omics Data Containers

  • Large-scale omics studies generate enormous amounts of biological data, including gene expression measurements, genomic variants, single-cell transcriptomes, proteomic profiles, metabolomic measurements, and spatially resolved molecular information. Traditional text-based formats such as CSV, TSV, and plain-text matrices are useful for small datasets, but they can become inefficient when datasets contain millions of observations and thousands of measured features. Large-scale omics analysis therefore relies on specialized data containers that support efficient storage, compression, indexing, metadata organization, and selective access to data. Important examples include HDF5, AnnData, Loom, Zarr, TileDB, and related formats designed for high-dimensional biological datasets.
  • HDF5, or Hierarchical Data Format version 5, is a general-purpose file format and data model designed to store and organize large, complex datasets. It is widely used in scientific computing, bioinformatics, imaging, and other data-intensive disciplines. An HDF5 file behaves like a structured container in which datasets and groups can be organized hierarchically. Groups function like directories, while datasets contain numerical arrays, strings, or other structured data. Attributes can store additional information about datasets and groups, including descriptions, units, experimental conditions, and processing details. This hierarchical organization allows different types of biological information to be stored together in a single file while maintaining logical relationships between them.
  • A major advantage of HDF5 is its ability to handle large multidimensional arrays efficiently. It supports compression, chunking, and partial data access, allowing software to retrieve selected portions of a dataset without loading the entire file into memory. Chunking divides a large dataset into smaller blocks that can be accessed independently, while compression reduces storage requirements. These features are particularly valuable for gene expression matrices, sequencing measurements, imaging data, and large experimental datasets. HDF5 also supports multiple programming languages and libraries, including Python, R, C, and C++, making it suitable for interoperable scientific workflows.
  • In Python, the h5py library provides an interface for creating, reading, and modifying HDF5 files. The following example demonstrates how to create a simple HDF5 file containing a gene expression matrix and associated metadata.

import h5py

import numpy as np

expression = np.random.rand(1000, 500)

with h5py.File(“omics_data.h5”, “w”) as file:

    file.create_dataset(

        “expression”,

        data=expression,

        compression=”gzip”,

        chunks=True

    )

    file.attrs[“description”] = “Example gene expression dataset”

    file.attrs[“organism”] = “Homo sapiens”

with h5py.File(“omics_data.h5”, “r”) as file:

    expression_data = file[“expression”][:]

    print(expression_data.shape)

    print(file.attrs[“organism”])

  • This example stores a numerical matrix and basic descriptive metadata in an HDF5 file. Real omics datasets generally require more extensive metadata, including sample identifiers, feature identifiers, experimental conditions, measurement units, and information about data processing.
  • AnnData, short for Annotated Data, is a data structure and file format developed primarily for high-dimensional biological data, particularly single-cell and spatial transcriptomics. It is closely associated with the Scanpy ecosystem in Python and provides a standardized way to organize measurements together with information about observations and variables. In a typical single-cell RNA sequencing dataset, observations represent individual cells, while variables represent genes. The main expression matrix is stored in the X component, while cell-level information is stored in obs and gene-level information is stored in var.
  • AnnData also supports additional matrices and annotations. The layers component can store alternative versions of the main expression matrix, such as raw counts, normalized counts, or transformed values. The obsm component stores multidimensional observation-level information, including principal component coordinates, spatial coordinates, and other cell embeddings. The varm component stores multidimensional variable-level information. The obsp and varp components can store pairwise relationships among observations and variables, respectively, such as cell-neighbor graphs or gene-correlation matrices. The uns component stores unstructured information, including analysis parameters, quality-control summaries, clustering information, and other metadata.
  • AnnData files are commonly saved using the .h5ad extension. This file format uses HDF5 as its underlying storage technology, combining the flexibility of hierarchical storage with a data model designed for annotated biological matrices. AnnData is particularly useful because it keeps expression measurements, sample annotations, gene information, and analysis results together in a structured object. It also supports efficient access to large datasets, including backed reading modes that allow data to be accessed from disk without loading every matrix into memory.
  • The following Python example demonstrates how to create and save a basic AnnData object.

import numpy as np

import pandas as pd

import anndata as ad

expression = np.random.poisson(

    lam=2,

    size=(100, 500)

)

cell_names = [f”Cell_{i}” for i in range(100)]

gene_names = [f”Gene_{i}” for i in range(500)]

adata = ad.AnnData(

    X=expression,

    obs=pd.DataFrame(

        {“cell_type”: [“Unknown”] * 100},

        index=cell_names

    ),

    var=pd.DataFrame(

        {“gene_symbol”: gene_names},

        index=gene_names

    )

)

adata.layers[“raw_counts”] = expression.copy()

adata.write_h5ad(“single_cell_data.h5ad”)

print(adata)

An AnnData object can be loaded and explored using the following commands.

import anndata as ad

adata = ad.read_h5ad(“single_cell_data.h5ad”)

print(adata.shape)

print(adata.obs.head())

print(adata.var.head())

  • AnnData is widely used in single-cell RNA sequencing workflows involving quality control, normalization, dimensionality reduction, clustering, cell-type annotation, and differential expression analysis. It is also increasingly used in spatial transcriptomics, where spatial coordinates and tissue-related annotations can be stored alongside molecular measurements. Its flexible structure makes it suitable for integrating experimental metadata, computational results, and multiple representations of the same biological data.
  • Loom is another HDF5-based format developed for storing large-scale single-cell gene expression datasets. A Loom file generally organizes data into a main matrix, row attributes, column attributes, and additional layers. The main matrix commonly contains genes as rows and cells as columns. Row attributes describe genes, while column attributes describe cells. Layers can contain alternative expression measurements or other matrices aligned with the same genes and cells. Loom was designed to support efficient access to large single-cell datasets and has been used in tools such as velocyto for RNA velocity analysis.
  • Although Loom and AnnData both use HDF5 technology, their data models and conventions differ. AnnData has become particularly prominent in the Scanpy ecosystem, while Loom remains relevant in workflows and datasets that use its matrix-oriented structure. Conversion between these formats is possible, but users should check whether all metadata, matrix layers, identifiers, and analysis results are preserved. A successful conversion of the main expression matrix does not necessarily mean that every annotation or auxiliary data component has been transferred correctly.
  • Zarr is a format and software ecosystem for storing chunked, compressed, multidimensional arrays. It is particularly useful for datasets that are too large to handle efficiently as single monolithic files or that need to be accessed through cloud and distributed storage systems. Like HDF5, Zarr supports chunking and compression, but it is designed around independently addressable chunks and can work with different storage backends, including local directories and object storage. This makes it useful for large imaging datasets, spatial omics, high-resolution microscopy, and other applications involving multidimensional biological measurements.
  • Zarr is especially relevant when data must be accessed over networks or processed in distributed environments. Individual chunks can be retrieved without downloading an entire dataset, provided that the storage system and application support the required access pattern. Zarr can also be used with tools such as Dask to process large arrays in parallel or in manageable blocks. However, performance depends on chunk dimensions, compression choices, storage layout, network conditions, and the pattern of data access. Poorly selected chunk sizes can increase the number of storage operations and reduce performance.
  • The following example illustrates the creation of a Zarr array using Python. The exact API can vary between Zarr versions, so the installed version and its documentation should be checked when adapting code for a production workflow.

import numpy as np

import zarr

data = np.random.rand(1000, 500)

array = zarr.create_array(

    store=”omics_data.zarr”,

    shape=data.shape,

    chunks=(100, 100),

    dtype=”float64″,

    overwrite=True

)

array[:] = data

print(array.shape)

print(array[0:10, 0:10])

  • TileDB is another storage system designed for multidimensional arrays and large-scale scientific data. It supports sparse and dense arrays, compression, indexing, and efficient access to selected regions of data. These capabilities are useful in genomics, single-cell analysis, spatial biology, and other applications involving large and sparse datasets. TileDB can represent data using structured arrays and supports different access patterns, including querying selected coordinates and subsets of dimensions. It can also support cloud-based storage and distributed data access. Its flexible model makes it suitable for datasets that combine measurements, metadata, and multidimensional biological information.
  • MuData is a data structure designed to represent multimodal biological datasets. Modern single-cell experiments may measure several molecular modalities from the same cells, including gene expression, chromatin accessibility, surface proteins, and other molecular features. Storing all these measurements in a single expression matrix would make it difficult to preserve their distinct structures and annotations. MuData addresses this challenge by organizing multiple annotated data modalities within a common container. It is associated with the Muon ecosystem and can be used with AnnData objects for individual modalities. This approach supports integrated analysis while preserving modality-specific measurements and metadata.
  • For example, a multimodal experiment may contain an RNA expression matrix, an ATAC-seq accessibility matrix, and a protein abundance matrix. Each modality can have its own features, annotations, and measurement characteristics, while shared cell identifiers help connect the measurements across modalities. This structure is valuable for multi-omics integration, cell-state characterization, regulatory analysis, and the study of relationships between gene expression and chromatin accessibility.
  • SpatialData is a data framework designed for spatial omics and imaging-based biological datasets. Spatial experiments often combine gene expression measurements with tissue images, segmentation masks, spatial coordinates, and geometrical information. These data types have different structures and spatial resolutions, making them difficult to manage in a conventional expression matrix. SpatialData provides a common framework for representing spatially organized elements, including images, labels, points, shapes, and annotated tables. It is designed to support spatial relationships and transformations between coordinate systems, helping researchers connect molecular measurements with tissue organization.
  • OME-Zarr is an important related format for storing multidimensional biological imaging data using Zarr technology and conventions developed by the Open Microscopy Environment. It supports images with multiple dimensions, such as spatial coordinates, channels, time points, and image planes, together with metadata and multiscale representations. OME-Zarr is useful in microscopy, digital pathology, spatial biology, and imaging-based omics because it supports efficient access to large images at different resolutions. It should not be confused with AnnData or a general single-cell expression matrix: its primary purpose is the standardized representation of multidimensional image data.
  • Other specialized containers and file formats are also used in omics research. Seurat, a widely used R package for single-cell analysis, provides its own object model for storing expression assays, cell metadata, dimensionality-reduction results, graphs, and analysis information. Seurat objects can be saved in R-specific formats, while the H5Seurat format provides an HDF5-based representation intended for storing Seurat data. Conversion between Seurat and AnnData is possible through tools such as SeuratDisk and other interoperability workflows, although differences between their data models mean that some information may require explicit mapping.
  • The Matrix Market format is commonly used to store sparse numerical matrices, especially in single-cell sequencing data. Sparse matrices are important because many gene expression datasets contain a large proportion of zero values. Instead of storing every matrix element, sparse formats store only nonzero values and their positions, reducing memory and storage requirements. The Matrix Market format is frequently encountered in single-cell data distributions that provide a matrix file together with separate feature and barcode files. Although it is useful for exchanging sparse matrices, it does not provide the comprehensive metadata organization offered by AnnData or MuData.
  • Apache Parquet is a column-oriented storage format used in data engineering and large-scale analytics. It supports compression, efficient column selection, and integration with distributed computing frameworks. It can be useful for storing large tabular omics datasets, including sample metadata, variant annotations, gene-level summaries, and processed results. However, Parquet is not a direct replacement for a multidimensional annotated matrix container. Its usefulness depends on whether the data are naturally represented as tables and on the analysis tools used to access them.
  • The choice of a data container depends on the biological data type, dataset size, analytical requirements, and software ecosystem. HDF5 is a flexible general-purpose hierarchical storage format, while AnnData provides a specialized structure for annotated high-dimensional biological matrices. Loom is relevant to certain single-cell expression workflows, Zarr is useful for chunked arrays and cloud-oriented access, and TileDB supports multidimensional and sparse data management. MuData is designed for multimodal experiments, while SpatialData and OME-Zarr address spatial and imaging-related data. Parquet is useful for large tabular datasets, and Matrix Market remains important for exchanging sparse matrices.
  • The distinction between a storage format and a data model is important. HDF5 and Zarr provide mechanisms for storing datasets and metadata, but they do not automatically define the biological meaning of every stored field. AnnData, MuData, and SpatialData provide more specific organizational conventions that help software interpret biological observations, variables, modalities, and spatial relationships. A file may be technically readable while still being difficult to interpret if its metadata, identifiers, units, or conventions are missing.
  • Interoperability is a major consideration when working with large-scale omics containers. Researchers frequently move data between tools written in Python, R, and other programming languages. For example, single-cell data may be generated in one software environment, processed in another, and integrated with spatial or proteomic measurements using a third. Conversion can introduce problems if gene identifiers are changed, cell barcodes are reordered, sparse matrices are converted to dense matrices, or metadata fields are omitted. Researchers should verify the dimensions, observation identifiers, feature identifiers, data types, and relevant annotations after every conversion. They should also check whether transformations such as normalization or logarithmic scaling have been applied, because two matrices with identical dimensions may represent different types of measurements.
  • Storage performance depends not only on the selected format but also on how the data are organized. Compression can reduce file size, but it may increase processing time during reading and writing. Chunking can improve partial access, but the optimal chunk dimensions depend on the most common query patterns. For example, a workflow that repeatedly retrieves individual cells may benefit from a different layout than one that retrieves groups of genes across all cells. Sparse storage is particularly valuable when most matrix entries are zero, whereas dense storage may be more efficient for relatively complete matrices. Benchmarking with representative datasets and real analysis operations is therefore important before selecting a storage strategy for large projects.
  • Memory management is another essential issue. Loading an entire single-cell dataset into memory may be impractical when it contains millions of cells or a very large number of features. Backed access, lazy loading, chunk-based processing, sparse matrices, and distributed computing can reduce memory requirements. Python tools such as Dask can support operations on arrays that are larger than available memory, while Zarr and suitable HDF5 access patterns can allow selected portions of data to be retrieved from storage. However, not every analysis operation can be performed efficiently in an out-of-memory manner, and some software functions may create temporary copies of large matrices. Researchers should monitor memory usage and understand whether their chosen analysis tools operate on data in memory or access data incrementally.
  • Data integrity and reproducibility are equally important. Large omics containers should include clear descriptions of their contents, organism, genome or reference version where applicable, measurement units, sample identifiers, feature identifiers, processing history, and software versions. For single-cell data, metadata should distinguish raw counts from normalized or transformed expression values. For spatial data, coordinate systems, image dimensions, and transformations should be documented. For genomic data, reference genome builds and annotation versions are essential for interpreting coordinates and variant positions. Maintaining checksums, file versions, and provenance records helps researchers detect accidental changes and reproduce analyses.
  • Large-scale omics containers also require appropriate backup and data-sharing strategies. HDF5 files can be convenient for distributing a complete dataset as a single file, while Zarr can be advantageous when data are stored in cloud object storage and accessed selectively. The choice should consider file size, expected access patterns, long-term preservation, access permissions, and the requirements of public repositories. Data should be accompanied by documentation explaining the structure of the container and the meaning of its fields. Where possible, researchers should use established community conventions rather than creating undocumented custom layouts.
  • HDF5, AnnData, and other large-scale omics containers have become essential components of modern bioinformatics because they provide efficient ways to manage complex biological datasets. Their capabilities support single-cell transcriptomics, spatial biology, genomics, proteomics, metabolomics, and high-dimensional imaging. Understanding the differences between general storage technologies, specialized biological data models, and interoperable exchange formats helps researchers select suitable tools, preserve biological information, and build scalable and reproducible computational workflows.
Author:

Leave a Reply

Your email address will not be published. Required fields are marked *