- The National Center for Biotechnology Information (NCBI) is one of the most important providers of biological and biomedical data resources. It maintains databases covering nucleotide sequences, protein sequences, genes, genomes, genetic variation, molecular structures, scientific literature, taxonomy, gene expression, and other areas of biological research. These resources support a wide range of activities, including sequence analysis, genome annotation, comparative genomics, functional characterization, molecular biology, and biomedical research. NCBI Entrez and E-utilities provide an integrated system for searching, retrieving, and connecting information across many of these databases.
- Entrez is NCBI’s integrated search and retrieval system. It allows users to search multiple biological databases through a common interface and explore relationships between different types of records. For example, a researcher investigating a human gene can retrieve its gene record, associated nucleotide sequences, transcripts, protein products, genetic variants, related publications, and other linked information. Similarly, a protein record may be connected to its nucleotide sequence, conserved domains, molecular structures, and functional annotations. Entrez helps organize these relationships and provides a unified way to navigate the extensive biological information maintained by NCBI.
- The Entrez system includes several databases with different biological and scientific purposes. PubMed provides access to biomedical and life-science literature, while PubMed Central provides access to full-text biomedical and life-science articles. Nucleotide contains nucleotide sequence records, including submitted sequences and records from other sources. Protein contains protein sequence records, while Gene provides information about genes, their genomic locations, transcripts, products, and associated references. Genome and Assembly provide genome-related information, including genome assemblies and their associated metadata. Other important resources include SRA for sequence read archives, BioProject and BioSample for research project and sample metadata, Taxonomy for organism classification, Structure for macromolecular structures, and dbSNP for short genetic variation records. The availability and organization of records differ among these resources, and users should select the database that best matches their research question.
- E-utilities, or Entrez Programming Utilities, are a collection of web-based application programming interfaces (APIs) that allow users to access Entrez databases programmatically. They provide a way to perform database searches, retrieve record summaries, download complete records, identify relationships between records, and submit lists of identifiers for subsequent processing. Instead of manually entering queries into the NCBI website, researchers can use E-utilities through web requests, command-line tools, or programming languages such as Python, R, and Perl. This makes E-utilities particularly useful for large-scale biological data retrieval, automated annotation, and reproducible bioinformatics workflows.
- E-utilities use a common web-service structure. Requests are sent to an NCBI E-utilities base address, followed by the name of the required utility and its parameters. The general base URL is https://eutils.ncbi.nlm.nih.gov/entrez/eutils/. Each utility has a specific function, and parameters determine which database is accessed, what records are selected, and how the results are returned. Common parameters include db, which specifies the Entrez database; term, which provides the search expression; id, which specifies one or more record identifiers; retmode, which controls the output format; and retmax, which specifies the maximum number of records to return in certain operations. Additional parameters allow users to control date ranges, sorting, filtering, and retrieval behavior.
- The E-utilities collection contains nine principal utilities: EInfo, ESearch, EPost, ESummary, EFetch, ELink, EGQuery, ESpell, and ECitMatch. Each performs a different task, but they can be combined into workflows that search databases, retrieve records, and connect related information. Understanding the purpose of each utility is essential for developing efficient data-retrieval procedures.
- EInfo provides information about Entrez databases and their available search fields. It can return a list of databases supported by Entrez or provide information about a particular database, including its indexed fields, available links to other databases, record statistics, and update information. This is useful when designing programmatic searches because different databases support different search fields and query options. For example, a researcher can use EInfo to examine the available fields in PubMed or the Nucleotide database before constructing a complex query. EInfo can also help determine which database links are available for connecting records.
- ESearch performs text-based searches within a specified Entrez database. It accepts a search expression and returns the identifiers, or UIDs, of records matching the query. A UID is an internal Entrez identifier used to refer to a record within a particular database. ESearch can search using gene names, organism names, accession numbers, publication terms, and other indexed information. It also supports Boolean operators, field restrictions, and date-based search conditions. For example, a researcher interested in publications about CRISPR technology can search PubMed using a suitable query, while a researcher looking for nucleotide sequences from a particular organism can search the Nucleotide database with organism and sequence-related terms.
- A basic ESearch request for PubMed can be constructed as follows:
- https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=CRISPR
- The request specifies the PubMed database and searches for records matching the term CRISPR. The response normally contains a count of matching records and a list of their UIDs, subject to the requested retrieval settings. These identifiers can then be passed to other E-utilities to retrieve summaries or complete records. ESearch therefore commonly acts as the starting point of an automated retrieval workflow.
- ESummary retrieves concise summaries of records identified by their UIDs. Instead of downloading complete database records, users can obtain selected descriptive information that helps them examine and filter search results. The content of a summary depends on the database. A PubMed summary may include bibliographic information such as the article title, authors, journal, and publication date, whereas a gene or protein summary may contain biological descriptions and record-specific metadata. ESummary is useful when researchers need to review many search results without downloading all the underlying records.
- EFetch retrieves complete database records or selected data representations for specified identifiers. It is commonly used when users need the actual biological information rather than a short summary. Depending on the database and supported options, EFetch can return nucleotide sequences, protein sequences, publication records, genome information, and other record contents. Output formats may include XML, text, FASTA, or other database-specific representations. For example, researchers can use EFetch to download nucleotide sequences from NCBI’s Nucleotide database or protein sequences from the Protein database. The selected output format and parameters should be appropriate for the database being accessed.
- ELink identifies relationships between records within the same database or across different Entrez databases. It is particularly valuable for biological data integration because it allows users to move from one type of information to another. For example, a PubMed record may be linked to related records in other resources, while a gene record may be linked to nucleotide sequences, protein records, or other associated information. ELink can retrieve linked UIDs and support the construction of multi-step retrieval workflows. It is useful when researchers want to explore relationships between genes, proteins, sequences, publications, and other biological records.
- EPost allows users to submit a list of UIDs to the Entrez History server. The submitted identifiers are stored as a reusable set, which can then be accessed by other E-utilities. This is particularly useful when researchers already have a large list of identifiers from an earlier analysis or another database. Instead of including the entire list in every subsequent request, users can submit it once and refer to the stored set through the History server. EPost is commonly combined with ESummary, EFetch, ELink, and ESearch to retrieve or refine large groups of records.
- EGQuery performs a global search across Entrez databases and reports the number of records matching a query in the different databases. It is useful for identifying where information about a particular topic may be available. For example, a search for a gene name may return counts from Gene, Nucleotide, Protein, and other relevant databases. EGQuery is primarily a discovery tool for examining the distribution of matching records across Entrez resources; it does not replace the database-specific retrieval functions provided by ESearch, ESummary, and EFetch.
- ESpell provides spelling suggestions for search terms. It can help identify possible spelling errors or alternative forms of a query that may improve search results. This is useful when biological names, gene symbols, organism names, or technical terms are entered incorrectly. However, spelling suggestions should be reviewed carefully because biological terminology can contain similar-looking names that refer to different genes, organisms, or molecular entities.
- ECitMatch is designed to identify PubMed records from bibliographic citation information. It accepts citation details such as journal, publication year, volume, first page, and author information, and attempts to identify the corresponding PubMed identifier (PMID). It is useful when researchers have reference lists containing citations but need to obtain standardized PubMed identifiers for automated literature retrieval or reference management.
- A common E-utilities workflow combines several utilities to perform a complete search and retrieval task. For example, a researcher may begin with ESearch to identify PubMed records matching a scientific query, use ESummary to retrieve brief descriptions of the results, and then use EFetch to download selected full records. Another workflow may use ESearch to identify gene records, ELink to retrieve related protein records, and EFetch to download the associated protein sequences. These combinations make it possible to retrieve biological information systematically without manually searching each database.
- A typical retrieval pipeline can be represented as ESearch → ESummary or EFetch. ESearch identifies records, ESummary retrieves their summaries, and EFetch downloads their complete contents. When a researcher already has a list of identifiers, EPost can be used to submit the list, followed by ESummary or EFetch. When related information is needed from another database, ELink can be introduced into the workflow. These pipelines can be extended by combining several utilities to retrieve records from one database that are connected to records in another.
- The Entrez History server is an important feature for managing large retrieval tasks. When ESearch is used with the appropriate history option, the matching set of records can be stored on the History server. The response provides a query key and a web environment identifier that can be used in subsequent requests. These values allow later utilities to operate on the stored result set without repeatedly transmitting the complete list of UIDs. This approach reduces the number and size of requests and is particularly useful for large-scale searches and batch downloads.
- For example, a researcher may search for nucleotide records from a particular organism, store the matching records in the Entrez History server, and retrieve the sequences in manageable batches. This is more efficient than sending a separate request for every sequence. Similarly, a list of thousands of PubMed identifiers can be submitted through EPost and processed in batches using ESummary or EFetch. Batch retrieval should be designed carefully to avoid excessive server requests and to ensure that all records are collected successfully.
- E-utilities can be accessed directly through web browsers because their requests use standard web addresses and parameters. However, programmatic access is more useful for repeated or large-scale retrieval. Python provides libraries such as Biopython, which includes the Bio.Entrez module for working with NCBI E-utilities. Researchers can use this module to search databases, retrieve records, and parse returned information. R packages such as rentrez provide functions for interacting with Entrez databases, while command-line approaches can be incorporated into shell scripts and automated pipelines.
- Biopython’s Entrez module can be used to perform an automated search. A simple example is:
- from Bio import Entrez
- Entrez.email = “researcher@example.org”
- handle = Entrez.esearch(
- db=”pubmed”,
- term=”CRISPR”,
- retmax=10
- )
- results = Entrez.read(handle)
- handle.close()
- print(results[“IdList”])
- This example searches PubMed for records matching the term CRISPR and retrieves up to ten matching identifiers. The returned identifiers can subsequently be used with ESummary or EFetch. The email address should be replaced with a valid contact address, and software should follow NCBI’s usage guidelines. For larger projects, requests should be handled in batches, with suitable delays, error handling, and logging.
- EFetch can be used with Biopython to retrieve biological sequences. For example, a nucleotide record can be downloaded in FASTA format using an accession or UID:
- from Bio import Entrez
- Entrez.email = “researcher@example.org”
- handle = Entrez.efetch(
- db=”nuccore”,
- id=”NC_045512.2″,
- rettype=”fasta”,
- retmode=”text”
- )
- sequence = handle.read()
- handle.close()
- print(sequence)
- This example illustrates how a specific nucleotide record can be retrieved in FASTA format. The accession shown is an illustrative example of an accession-version identifier. Researchers should verify that the selected record, sequence version, organism, and database are appropriate for their analysis. The same general approach can be adapted for protein records and other supported Entrez databases by changing the database and retrieval parameters.
- NCBI provides access to E-utilities through a public service, but automated users must follow its request-rate and usage policies. The standard limit is generally up to three requests per second without an API key. An NCBI API key permits a default rate of up to ten requests per second, while higher rates may require special approval. Users should include appropriate tool and email information in automated requests, use batch retrieval where possible, avoid unnecessary repeated queries, and respect server responses. Excessive requests may lead to throttling or temporary blocking. Current limits and requirements should be checked in the official NCBI documentation because policies can be updated.
- Error handling is an important part of reliable E-utilities programming. Requests may fail because of invalid database names, incorrect parameters, malformed search expressions, unavailable records, network interruptions, or rate-limit restrictions. Some requests may return valid responses containing no matching records, which should be distinguished from technical errors. Automated workflows should check response status, validate returned content, handle timeouts, retry temporary failures appropriately, and record unsuccessful requests. It is also useful to preserve the original query and returned identifiers so that retrieval can be repeated or investigated later.
- The output of E-utilities may be returned in XML, JSON, text, or other formats depending on the utility and database. XML has traditionally been important in Entrez because it preserves structured information about biological records and their relationships. JSON is convenient for many modern programming applications, while FASTA is widely used for sequence retrieval. Researchers should select an output format that matches their downstream analysis and use appropriate parsers rather than treating structured responses as ordinary text. Biopython and other programming libraries provide tools for parsing common biological data formats.
- E-utilities also support integration with other bioinformatics resources. NCBI identifiers can be combined with identifiers from UniProt, Ensembl, the Protein Data Bank, and other databases through cross-references and mapping procedures. A researcher may retrieve a gene record from NCBI, identify its associated transcripts and protein products, obtain protein sequences, and then use the corresponding protein identifiers to investigate functional domains or molecular structures in external resources. Such integration is useful for building comprehensive biological annotation workflows and connecting genomic, transcriptomic, proteomic, structural, and literature information.
- It is important to distinguish E-utilities from other NCBI access systems. E-utilities provide programmatic access to Entrez databases, whereas NCBI Datasets offers dedicated services and command-line tools for retrieving selected genome, gene, taxonomy, and sequence data. Specialized NCBI resources may also provide their own APIs or download systems. The most appropriate method depends on the type of information required, the scale of retrieval, and the structure of the intended workflow. E-utilities remain particularly useful for searches and record retrieval across the Entrez database network.
- The main advantages of Entrez and E-utilities include centralized searching, access to diverse biological databases, automated record retrieval, cross-database linking, batch processing, and compatibility with common programming languages. They reduce the need for manual downloads and allow researchers to develop repeatable workflows for large-scale biological data analysis. Their limitations include database-specific search syntax, varying record formats, incomplete or changing cross-references, request-rate restrictions, and differences in the scope and quality of annotations across databases. Users must also consider record versioning and database updates when interpreting retrieved information.
- For reproducible bioinformatics research, the complete retrieval process should be documented. Important details include the database name, search expression, retrieval date, selected filters, returned identifiers, accession versions, output format, and relevant E-utilities parameters. For programmatic workflows, researchers should also record the software and library versions, API request settings, batching strategy, and any error-handling procedures. Large datasets should be stored with their source information and retrieval metadata so that later analyses can be traced to the original records.
- NCBI Entrez and E-utilities form an important foundation for automated biological data retrieval. Entrez provides an integrated environment for searching and exploring biological information, while E-utilities make these resources accessible through programmable web services. By combining ESearch, ESummary, EFetch, ELink, EPost, EInfo, EGQuery, ESpell, and ECitMatch, researchers can construct workflows for searching, retrieving, linking, and processing biological records. Their integration with Python, R, command-line tools, and bioinformatics pipelines makes them valuable for sequence analysis, genome annotation, literature mining, comparative genomics, and large-scale biological data management.