- Data provenance and reproducible retrieval are important principles in bioinformatics that ensure biological data can be traced to their original sources, retrieved consistently, and reused reliably in scientific research. Data provenance refers to the history and origin of data, including where they were obtained, who generated them, how they were collected, and which processing or transformation steps were performed. Reproducible retrieval refers to the ability to obtain the same data, or a clearly documented equivalent dataset, again at a later time using recorded identifiers, database versions, query parameters, access methods, and retrieval procedures. Together, these practices improve transparency, reliability, traceability, and reproducibility in computational biology.
- Bioinformatics research frequently depends on data retrieved from multiple public and specialized resources. Researchers may obtain nucleotide sequences from GenBank or the European Nucleotide Archive (ENA), protein records from UniProt, genome annotations from Ensembl, genetic variants from dbSNP or ClinVar, protein structures from the Protein Data Bank (PDB), and functional information from resources such as Gene Ontology and Reactome. These databases are continuously updated as new sequences, annotations, experimental results, and scientific evidence become available. Records may be corrected, replaced, expanded, or withdrawn. Consequently, retrieving data using only a search term or a database webpage may not be sufficient to reproduce an earlier analysis. Recording the source and retrieval conditions is essential.
- Data provenance describes the complete history of a dataset from its origin to its use in an analysis. It includes information about the original data producer, source repository, accession number, collection date, sample characteristics, experimental methods, processing procedures, and subsequent modifications. For example, the provenance of a genome sequence may include the organism, biological sample, sequencing technology, sequencing laboratory, assembly method, genome assembly version, annotation pipeline, and repository accession. For a gene expression dataset, provenance may include the tissue or cell type, experimental treatment, sequencing platform, library preparation method, quality-control procedures, reference genome, alignment software, and normalization method.
- Provenance can be considered at several levels. Source provenance identifies where the original data came from, such as a public database, research laboratory, publication, or sequencing experiment. Experimental provenance describes how the biological material was collected and how the experiment was conducted. Computational provenance records the software, algorithms, parameters, reference files, and processing steps used to transform the data. Administrative provenance may include the responsible organization, data ownership, licenses, access permissions, and dates of submission or modification. These different forms of provenance provide complementary information that helps researchers evaluate the origin, reliability, and appropriate use of biological data.
- Persistent identifiers are fundamental to data provenance. An accession number or persistent identifier provides a stable reference to a particular record or dataset. Examples include GenBank accession numbers for nucleotide sequences, UniProt accessions for proteins, Ensembl identifiers for genes and transcripts, BioProject and BioSample accessions for research projects and biological samples, and DOIs for datasets and publications. A stable identifier makes it possible to refer to a specific biological record without relying on an ambiguous description or a temporary webpage address. Researchers should record the complete identifier, including its version suffix when one is provided. Versioned accessions can distinguish successive versions of a sequence or record and help identify exactly which biological data were used.
- Database versions and release information are also essential. Biological databases frequently update their records, annotation pipelines, reference assemblies, and underlying evidence. For example, a gene annotation retrieved from one Ensembl release may differ from the corresponding annotation in a later release because of revised transcript models, updated genome assemblies, or improved evidence. Similarly, a protein record may receive new functional annotations as experimental knowledge develops. Recording the database name, release number, record accession, version, and retrieval date helps distinguish the information used in an earlier analysis from the current version. Where available, researchers should preserve the original downloaded files or use archived database releases rather than relying exclusively on the latest records.
- Reproducible retrieval requires more than recording the name of a database. Researchers should document the exact query or search strategy used to obtain the data. This includes search terms, accession lists, filters, organism restrictions, taxonomic identifiers, date ranges, sequence types, evidence criteria, and other query parameters. For API-based retrieval, the endpoint, request method, query parameters, relevant headers, and response format should also be recorded. If data were retrieved through a web interface, the search criteria and any selected filters should be documented. These details make it easier for another researcher to repeat the retrieval process and understand why particular records were included or excluded.
- For example, searching a nucleotide database for all sequences associated with a particular organism may produce different results at different times because new records are added and existing records are updated. A reproducible workflow should record the organism name or taxonomic identifier, the database searched, the exact query, the retrieval date, and the resulting accession list. Saving the retrieved accession list is particularly important because repeating the same query in the future may return a different set of records. When the original data are available, the downloaded files should also be preserved with their associated metadata.
- Reproducible retrieval is especially important when working with application programming interfaces (APIs). APIs allow researchers to retrieve biological information programmatically from resources such as NCBI, UniProt, and Ensembl. However, API responses may change as databases are updated, endpoints are modified, or annotation systems evolve. A reliable retrieval workflow should record the API endpoint, request parameters, retrieval date, response format, and relevant software dependencies. It should also preserve the returned data when appropriate. Where possible, researchers should use versioned API endpoints, archived releases, or stable accession identifiers to reduce the effects of changes in database content and service behavior.
- For example, the following Python code retrieves a nucleotide record from NCBI using its accession number and saves both the retrieved sequence and basic retrieval information. The example uses Biopython and requires an internet connection.
from Bio import Entrez, SeqIO
from datetime import datetime, timezone
import json
Entrez.email = “researcher@example.com”
accession = “NC_000913.3”
with Entrez.efetch(
db=”nuccore”,
id=accession,
rettype=”gb”,
retmode=”text”
) as handle:
record = SeqIO.read(handle, “genbank”)
# Save the retrieved sequence record
SeqIO.write(record, “retrieved_sequence.gb”, “genbank”)
# Save retrieval provenance
metadata = {
“database”: “NCBI Nucleotide”,
“accession”: accession,
“retrieval_date”: datetime.now(timezone.utc).isoformat(),
“retrieval_method”: “NCBI Entrez efetch”,
“record_id”: record.id,
“description”: record.description,
“sequence_length”: len(record.seq)
}
with open(“retrieval_metadata.json”, “w”) as file:
json.dump(metadata, file, indent=4)
print(“Sequence and retrieval metadata saved.”)
- This example demonstrates how an accession number, retrieval method, date, and basic record information can be preserved alongside the downloaded data. In a research workflow, additional information should be recorded where relevant, including the full query, database release, API endpoint, software version, request parameters, and any transformations applied after retrieval. The example accession is illustrative; researchers should verify that the record is appropriate for their organism and analysis.
- Data provenance is also important when retrieving information from multiple databases. A gene may have different identifiers in NCBI Gene, Ensembl, and other annotation resources, while its corresponding protein may have a separate UniProt accession. Cross-references help connect these records, but identifiers are not always interchangeable. Some identifiers refer to genes, others to transcripts, proteins, genomic locations, or specific sequence versions. Researchers should therefore record the identifier type and source database, and document any mapping procedure used to connect records. This prevents confusion when different resources represent biological entities at different levels.
- The following Python example illustrates how to preserve a simple list of database identifiers and their sources in a structured CSV file.
import csv
records = [
{
“database”: “NCBI Gene”,
“identifier”: “example_gene_id”,
“record_type”: “Gene”
},
{
“database”: “Ensembl”,
“identifier”: “example_ensembl_id”,
“record_type”: “Gene”
},
{
“database”: “UniProt”,
“identifier”: “example_protein_id”,
“record_type”: “Protein”
}
]
with open(“database_identifiers.csv”, “w”, newline=””) as file:
writer = csv.DictWriter(
file,
fieldnames=[“database”, “identifier”, “record_type”]
)
writer.writeheader()
writer.writerows(records)
- In actual research, illustrative identifiers should be replaced with verified identifiers obtained from the relevant databases. The source database and record type should always be retained so that the meaning of each identifier remains clear.
- Computational provenance records how retrieved data are processed and transformed. Bioinformatics workflows often involve multiple steps, such as downloading raw sequencing reads, performing quality control, trimming adapters, aligning reads to a reference genome, identifying variants, annotating those variants, and conducting statistical analysis. Each step may affect the final results. A reproducible workflow should record the software name and version, command-line options, parameters, reference files, input and output files, and execution environment. This information allows researchers to trace the origin of an analytical result and identify which processing step produced a particular output.
- Workflow management systems such as Snakemake, Nextflow, and Common Workflow Language (CWL) can help preserve computational provenance. These systems describe the sequence of analytical steps, dependencies, inputs, outputs, and execution requirements. Workflow definitions can be stored in version-control systems such as Git, allowing researchers to track changes to analysis procedures. Containers created with Docker or Apptainer can preserve software environments and dependencies, reducing differences between computational systems. However, containers alone do not guarantee complete reproducibility; input data, reference resources, configuration files, software versions, and workflow parameters must also be documented.
- Checksums provide another important mechanism for preserving data integrity. A checksum is a value calculated from a file using a hashing algorithm such as MD5 or SHA-256. If a file changes, its checksum will generally change as well. Recording checksums allows researchers to verify that downloaded files have not been corrupted or unintentionally modified. For example, a downloaded FASTQ file can be checked against a checksum supplied by the repository or calculated when the file is first retrieved. Checksums do not establish the scientific correctness of a dataset, but they help verify file integrity and support reliable data transfer and storage.
- The following command-line examples demonstrate how to calculate and verify checksums on a Linux system.
# Calculate a SHA-256 checksum
sha256sum sample.fastq.gz
# Save the checksum
sha256sum sample.fastq.gz > sample.sha256
# Verify the file against the saved checksum
sha256sum -c sample.sha256
- Data retrieval logs provide a practical record of how data were obtained. A retrieval log may include the date and time, database, accession number, query, API endpoint, file name, file size, checksum, and retrieval status. Logs are particularly useful when retrieving large datasets or running automated pipelines. They can help identify incomplete downloads, failed requests, unexpected changes in file size, and differences between retrieval attempts. For large projects, retrieval logs should be stored with the analysis files and maintained under version control where appropriate.
- The following example shows a simple tab-separated retrieval log.
retrieval_date database accession filename status
2026-10-24 NCBI NC_000913.3 retrieved_sequence.gb success
2026-10-24 UniProt P12345 protein_record.json success
- The dates and identifiers in this example are illustrative. A complete retrieval log should also include relevant query parameters, database release information, checksums, and software details.
- Version control is useful for managing changes to retrieval scripts, query definitions, metadata files, and workflow configurations. Git allows researchers to record the history of code and configuration changes, compare different versions, and identify when a retrieval procedure was modified. However, version control systems are not generally suitable for storing very large sequencing datasets or other massive biological files. Such data should be preserved in appropriate repositories or data-storage systems, while their identifiers, checksums, metadata, and retrieval procedures are maintained alongside the analysis code.
- Data provenance and reproducible retrieval are closely connected to the FAIR principles: Findability, Accessibility, Interoperability, and Reusability. Persistent identifiers and detailed metadata improve findability. Documented access procedures and stable retrieval mechanisms support accessibility. Standardized identifiers and formats improve interoperability. Provenance, licensing, version information, and processing documentation support reusability. FAIR principles provide the broader framework, while provenance and reproducible retrieval provide practical methods for maintaining the history and traceability of biological data.
- Several standards and frameworks support the recording and exchange of provenance information. The W3C PROV model provides a general framework for describing entities, activities, and agents involved in data production and transformation. The Research Object Crate (RO-Crate) specification supports the description and packaging of research data, software, workflows, and related metadata. Workflow systems can also record execution details and relationships between inputs and outputs. In bioinformatics, these approaches can be combined with community-specific metadata standards, repository accession systems, and structured file formats to improve the documentation of research datasets.
- Reproducible retrieval can be challenging because biological databases and online services are not static. Records may be updated, removed, merged, or replaced; APIs may change; servers may become temporarily unavailable; and access conditions may be revised. Some repositories provide archived releases or versioned records, while others provide only the current version. Researchers should therefore preserve important input files when permitted, record retrieval dates and versions, save accession lists and query definitions, and document any unavailable or restricted resources. When exact reproduction is not possible, the recorded provenance should make it clear which parts of the retrieval process have changed.
- Ethical and legal considerations must also be included in provenance records. Human genomic and clinical datasets may contain sensitive personal information and may be subject to consent restrictions, institutional approvals, data-use agreements, or privacy regulations. Researchers should document the applicable access conditions without exposing confidential information. Data provenance should support accountability and traceability while respecting privacy, intellectual property, and licensing requirements.
- In practical bioinformatics research, data provenance and reproducible retrieval should be planned from the beginning of a project rather than added after the analysis has been completed. Researchers should record database names, persistent identifiers, record versions, retrieval dates, search queries, API parameters, file checksums, software versions, and processing steps. Original data should be preserved where permitted, and retrieval scripts, metadata, logs, and workflow definitions should be organized in a consistent project structure. These practices make it easier to repeat analyses, investigate unexpected results, compare datasets, and share research outputs with other scientists.
- Data provenance and reproducible retrieval are essential for reliable and transparent bioinformatics. They establish where biological data originated, how they were obtained, which versions were used, and how they were processed. By combining persistent identifiers, structured metadata, database versioning, retrieval logs, checksums, version control, and reproducible computational workflows, researchers can create analyses that are easier to verify, repeat, and reuse. As biological databases and computational resources continue to expand, these practices will remain central to data integrity, scientific reproducibility, and collaborative research.