Biological Data Retrieval, APIs, and Interoperability

  • Biological data retrieval, application programming interfaces (APIs), and interoperability are essential components of bioinformatics because biological research depends on accessing, combining, and analyzing information stored across many independent databases and computational resources. Genomic sequences, protein records, molecular structures, gene annotations, genetic variants, gene expression profiles, biological pathways, phenotypes, and scientific literature are distributed across numerous repositories. Researchers need reliable methods to retrieve these data, process them in computational workflows, and integrate information from different sources. Biological data retrieval provides access to relevant records, APIs enable automated communication between databases and software applications, and interoperability allows information from different resources to be exchanged and interpreted consistently.
  • Biological data retrieval refers to the process of searching for, accessing, downloading, and extracting biological information from databases, web services, repositories, and other data platforms. Retrieval may involve a single record, a collection of records, an entire dataset, or repeated queries performed automatically through a computational pipeline. For example, a researcher may retrieve a nucleotide sequence from GenBank, obtain protein functional annotations from UniProt, identify associated domains using InterPro, retrieve a three-dimensional structure from the Protein Data Bank, and collect pathway information from Reactome. These resources provide complementary information, and combining their records can produce a more complete understanding of a biological system.
  • Biological databases provide several methods for retrieving information. Web-based interfaces allow users to search records using keywords, accession numbers, gene symbols, organism names, genomic coordinates, and other identifiers. Advanced search interfaces may support filters based on taxonomy, sequence length, publication date, experimental method, evidence type, or database category. Bulk download systems provide complete datasets or selected collections of records for large-scale analysis. APIs provide programmatic access, allowing software applications and scripts to submit queries, retrieve results, and integrate data automatically. File transfer services and cloud-based repositories provide additional options for accessing large biological datasets, particularly sequencing data and reference genomes.
  • Accession numbers and persistent identifiers are central to biological data retrieval. An accession number is a stable or versioned identifier assigned to a database record, allowing researchers to retrieve a particular sequence, protein, structure, or other biological entity. Examples include GenBank accession numbers, UniProt accession identifiers, Protein Data Bank identifiers, and RefSeq accessions. Gene identifiers may be provided by resources such as NCBI Gene, Ensembl, and organism-specific databases. Persistent identifiers help distinguish records that may have similar names and allow information to be linked across databases. However, identifiers are not always interchangeable, and the same biological entity may have different identifiers in different resources. Researchers should therefore use cross-reference services and identifier-mapping tools when connecting records.
  • An API is a defined interface that allows one software system to communicate with another. In bioinformatics, APIs enable scripts, applications, and analytical pipelines to retrieve biological records without requiring users to navigate database websites manually. A typical API request contains a resource address, query parameters, and sometimes authentication information. The database processes the request and returns data in a specified format. For example, a Python script can submit a request for a gene identifier, retrieve its sequence or annotation, and pass the result to another analytical program. This makes APIs particularly useful for high-throughput research, automated database searches, reproducible workflows, and integration of multiple biological resources.
  • Representational State Transfer (REST) is one of the most widely used architectural styles for web APIs. REST-based services generally expose resources through URLs and use standard HTTP methods such as GET, POST, PUT, PATCH, and DELETE, depending on the service. In biological data retrieval, GET requests are commonly used to retrieve records, while POST requests may be used to submit complex queries or data. A request may specify an accession number, gene identifier, organism, genomic region, or other parameter. The server returns the requested information, often in JSON, XML, or another machine-readable format. REST APIs are used by resources such as Ensembl, UniProt, Europe PMC, and several NCBI services.
  • NCBI provides several important programmatic retrieval services. The Entrez Programming Utilities (E-utilities) allow researchers to search and retrieve information from NCBI databases, including PubMed, Gene, Nucleotide, Protein, and other resources. ESearch can identify records matching a query, EFetch can retrieve record data, ESummary provides document summaries, and ELink supports connections between related records. These services are useful for retrieving sequences, identifying publications, collecting gene information, and exploring relationships among database entries. NCBI also provides datasets and other programmatic services for accessing genome assemblies, genes, and biological datasets. Researchers should follow NCBI usage policies, including applicable request-rate limits and identification requirements, when developing automated retrieval systems. <Link url=”https://www.ncbi.nlm.nih.gov/books/NBK25501/” title=”NCBI E-utilities Documentation”/> <Link url=”https://www.ncbi.nlm.nih.gov/datasets/” title=”NCBI Datasets”/>.
  • The Ensembl REST API provides programmatic access to genome annotations, genes, transcripts, proteins, variation, comparative genomics, and other genomic information. Researchers can retrieve gene records, obtain sequences, examine transcript structures, identify genomic features, and access variant consequences. Ensembl also provides BioMart, which supports the retrieval of selected biological attributes and filters from integrated datasets. These services are particularly useful for large-scale genomic annotation and comparative genomics. Ensembl APIs and BioMart queries can be incorporated into Python, R, and other computational environments to automate the retrieval of gene and genome information. <Link url=”https://rest.ensembl.org/” title=”Ensembl REST API”/> <Link url=”https://www.ensembl.org/biomart/martview/” title=”Ensembl BioMart”/>.
  • UniProt provides programmatic access to protein sequences, functional annotations, organism information, protein names, subcellular locations, cross-references, and other protein-related data. Its REST API supports searching, retrieving individual records, and downloading sets of protein entries. Researchers can use UniProt services to identify proteins, retrieve amino acid sequences, examine functional annotations, and connect protein records with other biological databases. UniProt records may include information about protein domains, catalytic activity, post-translational modifications, disease associations, and experimental evidence. These annotations are particularly useful for functional genomics, proteomics, comparative protein analysis, and biological network studies. <Link url=”https://www.uniprot.org/help/api” title=”UniProt API Documentation”/>.
  • The European Bioinformatics Institute (EMBL-EBI) provides a range of APIs and web services for accessing biological information. These include services associated with European Nucleotide Archive (ENA), InterPro, Europe PMC, Expression Atlas, BioSamples, BioStudies, and other resources. Europe PMC provides programmatic access to scientific literature, abstracts, publication metadata, and related information. InterPro services allow users to retrieve protein family, domain, and functional site annotations. ENA services provide access to nucleotide sequence data and associated metadata. These resources support integrated biological research by allowing data retrieval across multiple domains of molecular biology. <Link url=”https://www.ebi.ac.uk/services” title=”EMBL-EBI Services”/> <Link url=”https://europepmc.org/RestfulWebService” title=”Europe PMC RESTful Web Service”/> <Link url=”https://www.ebi.ac.uk/ena/browser/api/” title=”ENA Browser API”/>.
  • The Protein Data Bank (PDB) provides APIs and downloadable resources for retrieving three-dimensional macromolecular structures and their associated metadata. The RCSB PDB Search API supports searches based on structural properties, molecular components, organisms, experimental methods, and other criteria. The Data API provides detailed information about structures, entities, chains, and related records. These services are important for structural bioinformatics, molecular modelling, protein–ligand interaction analysis, and comparative structural studies. PDB records can also be connected with sequence and protein information from resources such as UniProt and other structural databases. <Link url=”https://www.rcsb.org/docs/programmatic-access/web-apis-overview” title=”RCSB PDB Web APIs”/>.
  • Some biological databases provide APIs based on query languages or specialized service architectures rather than conventional REST endpoints. SPARQL is a query language used to retrieve information from knowledge graphs and Resource Description Framework (RDF) datasets. It allows researchers to query relationships among biological entities, such as genes, proteins, diseases, pathways, and publications. SPARQL endpoints can support complex queries across interconnected biological knowledge. GraphQL is another API approach in which clients specify the fields and relationships they want returned. It can be useful when retrieving related information from complex data models, although its availability and implementation differ among biological resources. SOAP-based web services have also been used in bioinformatics, particularly in older database integration systems. Researchers should choose the interface supported by the relevant database rather than assuming that all resources provide the same API architecture.
  • Bulk data retrieval is important when individual API requests are insufficient for large-scale analyses. Biological repositories commonly provide downloadable files through HTTP, FTP, cloud storage, or dedicated download services. Researchers may download reference genomes, genome annotations, nucleotide sequences, protein databases, variation datasets, gene expression matrices, and structural information. Large sequencing datasets may be distributed through specialized archives such as the Sequence Read Archive (SRA), European Nucleotide Archive (ENA), and DNA Data Bank of Japan (DDBJ). Bulk retrieval is useful for building local databases, conducting population-scale analyses, and processing datasets repeatedly without submitting thousands of individual queries. However, large downloads require appropriate storage, network capacity, file-integrity checks, and documentation of the dataset version.
  • Biological data are distributed in many file formats, each designed to represent particular types of information. FASTA is widely used for nucleotide and protein sequences, while FASTQ stores sequencing reads together with their quality scores. GenBank and EMBL flat-file formats contain sequence records with annotations and feature information. GFF and GTF formats describe genomic features such as genes, transcripts, exons, and coding sequences. VCF is used to represent genetic variants and genotypes, while BED is commonly used for genomic intervals. SAM and BAM formats store sequence alignment information, and CRAM provides a compressed representation of alignment data. PDB and mmCIF formats describe macromolecular structures. JSON and XML are widely used for structured API responses, while CSV and TSV are common for tabular biological data. Understanding these formats is essential for correctly interpreting retrieved information and transferring it between analytical tools.
  • JSON (JavaScript Object Notation) is widely used in modern biological APIs because it represents structured information in a readable and machine-processable form. JSON objects contain key–value pairs, while arrays store collections of values. A protein record returned by an API may include its accession, protein name, organism, sequence length, functional annotations, and cross-references. XML (Extensible Markup Language) is another structured format used by biological services and databases. It represents information using nested elements and attributes and is often used in established bioinformatics standards and data exchange systems. Both formats can represent complex biological records, but their structures must be understood before information is extracted programmatically.
  • Interoperability refers to the ability of different databases, software applications, and information systems to exchange data and use the exchanged information meaningfully. In bioinformatics, interoperability is particularly challenging because biological databases may use different identifiers, naming conventions, data models, vocabularies, file formats, and annotation standards. For example, one resource may identify a gene using an NCBI Gene ID, another may use an Ensembl Gene ID, and a third may use a species-specific identifier. These identifiers may refer to the same biological entity, but they cannot always be substituted directly. Identifier mapping and cross-referencing are therefore important steps in integrating biological data.
  • Cross-references connect records from different databases and help establish relationships among biological entities. UniProt records, for example, may contain links to Ensembl, RefSeq, PDB, InterPro, Gene Ontology, and pathway resources. NCBI records also contain links to related sequence, gene, protein, and literature entries. These cross-references allow researchers to move from a nucleotide sequence to its translated protein, from a protein to its structural record, or from a gene to its associated pathways and publications. However, cross-references may be incomplete, outdated, or based on different biological interpretations. A link between two records does not necessarily mean that they contain identical information or represent exactly the same level of biological evidence.
  • Identifier mapping is the process of converting or connecting identifiers used by different databases. Tools such as UniProt ID Mapping, Ensembl BioMart, NCBI’s gene and sequence resources, and other mapping services support this process. A common challenge is that one gene may have several transcript identifiers, one transcript may encode different protein products, and one protein may have multiple database accessions. Some identifiers may be retired, merged, or replaced when records are updated. Mapping may therefore produce one-to-many, many-to-one, or unmatched relationships. Researchers should preserve the original identifiers and record the mapping source, date, and method so that the results can be checked and reproduced.
  • Controlled vocabularies and biological ontologies improve interoperability by providing standardized terms for describing biological concepts. The Gene Ontology (GO) offers structured terminology for molecular functions, biological processes, and cellular components. The Human Phenotype Ontology (HPO) provides standardized terms for clinical phenotypes, while disease ontologies such as Mondo support the consistent representation of diseases. Sequence Ontology provides terminology for genomic sequence features, and the Experimental Factor Ontology (EFO) supports the representation of experimental variables and biomedical concepts. Ontologies help reduce ambiguity caused by different names for the same concept and allow databases to connect records through standardized relationships. They are especially important for data integration, semantic searches, functional enrichment, and automated reasoning.
  • The FAIR principles provide a widely recognized framework for improving biological data management and interoperability. FAIR stands for Findable, Accessible, Interoperable, and Reusable. Findable data should have persistent identifiers and descriptive metadata that allow researchers and computational systems to locate them. Accessible data should be retrievable through clearly defined and preferably standardized protocols, subject to appropriate access controls. Interoperable data should use shared formats, vocabularies, and standards so that they can be combined with other datasets. Reusable data should include sufficient documentation, provenance, licensing information, and methodological details to support future research. FAIR principles do not necessarily require that every dataset be openly accessible; controlled access may be necessary for sensitive or restricted biological information.
  • Data provenance is another important aspect of biological data retrieval and integration. Provenance describes where data originated, how they were generated, which transformations were applied, and which resources or software versions were used. For example, a gene annotation may have been retrieved from a particular Ensembl release, mapped to a different genome assembly, and combined with protein information from a specified UniProt release. If these steps are not documented, it may become difficult to reproduce the analysis or explain differences between results. Provenance information should include database names, accession numbers, release dates, retrieval dates, software versions, query parameters, and relevant processing steps. Persistent identifiers and versioned records help maintain traceability when databases are updated.
  • API-based retrieval introduces several technical considerations. Rate limits restrict how frequently a client can send requests to a service, helping databases maintain performance and availability. Excessive requests may lead to temporary blocking or errors. Some services require authentication through API keys or other access mechanisms, particularly for high-volume requests or controlled datasets. Pagination is commonly used when a query returns a large number of records; the client must retrieve results across multiple pages or batches. Asynchronous jobs may be used for queries that require substantial computation. Researchers should also handle network interruptions, server errors, timeouts, and incomplete responses. Well-designed retrieval scripts include error handling, retry mechanisms with appropriate delays, logging, and validation of returned data.
  • Efficient API use requires attention to query design. A narrowly defined query using stable identifiers is generally more reliable than a broad keyword search. Researchers should use filters such as organism, database, genomic region, record type, and evidence category when available. When retrieving large collections, it is often more efficient to submit one carefully constructed query or use a bulk download rather than issue thousands of individual requests. Duplicate records should be identified and handled consistently, especially when information is collected from multiple databases. Researchers should also check whether an API returns all matching records or only a limited result set, because incomplete retrieval can affect downstream statistical and biological conclusions.
  • Programming languages such as Python and R are widely used for biological data retrieval. Python libraries such as requests, Biopython, and pandas support HTTP requests, sequence retrieval, parsing, and tabular data processing. Biopython includes modules for working with sequence formats and selected biological services. R packages such as rentrez, httr2, and Bioconductor packages support access to external resources and integration with genomic data analysis workflows. These tools allow researchers to automate searches, download records, parse responses, and combine data with local analyses. The choice of language and library depends on the database interface, data volume, analytical requirements, and the reproducibility needs of the project.
  • Biological data interoperability is particularly important in multi-omics research. Genomics, transcriptomics, proteomics, metabolomics, epigenomics, and other fields generate data at different biological levels. Integrating these datasets requires connecting genes, transcripts, proteins, metabolites, pathways, and phenotypes through consistent identifiers and biological relationships. For example, a researcher studying a disease may combine genomic variants with gene expression measurements, protein abundance, pathway annotations, and clinical phenotypes. This integration can reveal relationships that are not apparent from one data type alone. However, differences in sample identifiers, experimental conditions, measurement scales, genome assemblies, and data quality can introduce errors if they are not addressed systematically.
  • Interoperability also supports reproducible bioinformatics workflows. Workflow systems such as Nextflow, Snakemake, and Galaxy can incorporate data retrieval steps alongside sequence processing, annotation, statistical analysis, and visualization. Automated workflows can retrieve reference data, record database releases, validate downloaded files, and pass standardized outputs to downstream tools. Containers and reproducible software environments help ensure that the same retrieval and analysis procedures can be executed consistently across different computational systems. A robust workflow should distinguish between data retrieved dynamically from an external service and data stored locally at a fixed version. Dynamic retrieval may provide the latest annotations, but it can also produce different results when the source database changes.
  • Despite these capabilities, biological data retrieval and interoperability have several limitations. Databases may contain incomplete records, inconsistent annotations, outdated identifiers, conflicting evidence, or different interpretations of the same biological entity. Some resources update frequently, while others release major updates less often. API endpoints and response formats may also change, potentially breaking previously developed scripts. A database may provide access to a record without guaranteeing that its annotations are experimentally validated. Differences in genome assemblies, transcript versions, taxonomic classifications, and controlled vocabularies can create incompatibilities when records are combined. Researchers must therefore evaluate the quality and scope of each resource rather than assuming that retrieved data are automatically accurate or comparable.
  • Access restrictions and licensing are additional considerations. Some biological datasets are openly available, while others require registration, authentication, controlled access, or compliance with specific data-use agreements. Human genomic data may contain sensitive information and require safeguards to protect participant privacy. Certain databases also impose restrictions on redistribution or commercial use. Researchers should review the relevant access policies, licences, and ethical requirements before downloading, storing, sharing, or incorporating data into public repositories. Interoperability should not be achieved by bypassing the access conditions under which the original data were provided.
  • Biological data retrieval, APIs, and interoperability collectively provide the infrastructure for modern computational biology. APIs from resources such as NCBI, Ensembl, UniProt, EMBL-EBI, and RCSB PDB enable automated access to diverse biological records, while standardized identifiers, file formats, ontologies, and metadata support their integration. The FAIR principles, provenance tracking, version control, and reproducible workflows help ensure that retrieved information can be understood, validated, and reused. By applying these methods carefully, bioinformatics researchers can combine information from multiple biological databases, support large-scale computational analyses, improve reproducibility, and develop more comprehensive interpretations of complex biological systems.
Author:

Leave a Reply

Your email address will not be published. Required fields are marked *