- Biological databases provide application programming interfaces (APIs) that allow researchers to retrieve biological information automatically. These services are commonly used to access nucleotide sequences, protein records, gene annotations, genetic variants, publications, and other scientific data. When working with these APIs, researchers must understand pagination, rate limits, authentication, and API reliability. These concepts are important because database queries may return thousands or millions of records, access may be restricted or limited, and network or server problems can interrupt data retrieval.
- Pagination is the process of dividing a large collection of search results into smaller groups called pages. Instead of returning every matching record in one response, an API returns a limited number of records at a time. The program then requests the next page until all required records have been retrieved. Pagination reduces the amount of data transferred in a single request, limits memory requirements, and makes large database searches easier to manage.
- Different APIs implement pagination in different ways. Some use page numbers and page sizes, while others use record offsets and maximum result counts. Some services provide a continuation token or a URL in the response header that points to the next page. Researchers must follow the pagination method documented by the particular database rather than assuming that every API uses the same parameters.
- For example, an API may use page to identify the requested page and size to specify the number of records per page. The following is an illustrative URL:
https://example.org/api/genes?page=1&size=100
- This request asks for the first 100 records. The next request would retrieve the following page:
https://example.org/api/genes?page=2&size=100
- Another common method uses offset and limit. The limit parameter specifies the number of records to return, while offset specifies the starting position in the result set.
https://example.org/api/genes?limit=100&offset=0
- The next group can be retrieved by increasing the offset:
https://example.org/api/genes?limit=100&offset=100
- NCBI E-utilities provides pagination through parameters such as retstart and retmax. The retmax parameter controls the number of records returned, while retstart identifies the starting position. For example, a search can retrieve the first 100 records and then retrieve the next group by changing the starting position. NCBI also provides the Entrez History server, which is useful when searches and downloads involve large numbers of records. Instead of repeatedly submitting the same search, researchers can store the search results on the History server and retrieve records in batches.
- The following example illustrates how Python can retrieve multiple pages from an API that uses page and size parameters:
import requests
base_url = “https://example.org/api/genes”
all_records = []
page = 1
page_size = 100
while True:
params = {
“page”: page,
“size”: page_size
}
response = requests.get(
base_url,
params=params,
timeout=30
)
response.raise_for_status()
data = response.json()
records = data[“results”]
if not records:
break
all_records.extend(records)
if len(records) < page_size:
break
page += 1
print(“Total records retrieved:”, len(all_records))
- This is a general example. The endpoint, response field, and pagination parameters must be adapted to the API being used. Some APIs provide a next URL or continuation token instead of requiring the program to calculate the next page.
- The following R example demonstrates the same general approach using an API with page-based pagination:
library(httr2)
library(jsonlite)
base_url <- “https://example.org/api/genes”
all_records <- list()
page <- 1
page_size <- 100
repeat {
response <- request(base_url) |>
req_url_query(
page = page,
size = page_size
) |>
req_perform()
data <- resp_body_json(response)
records <- data$results
if (length(records) == 0) {
break
}
all_records <- c(all_records, records)
if (length(records) < page_size) {
break
}
page <- page + 1
}
cat(“Total records retrieved:”, length(all_records), “\n”)
- UniProt REST API uses pagination for search results and provides a continuation link in the HTTP response headers. Researchers should follow the returned Link information to retrieve subsequent pages instead of assuming that every service uses page numbers. Its search service supports up to 500 results per page, and large searches can require multiple requests. This is particularly important when retrieving protein records, functional annotations, or other information for large groups of accessions.
- Rate limits are restrictions placed on the number of requests that an application or user can send to an API within a specified period. They help databases manage server resources and provide reliable access to all users. A rate limit may be defined as a maximum number of requests per second, minute, hour, or day. The exact limit depends on the database, API endpoint, authentication status, and type of request.
- For example, NCBI E-utilities generally supports up to three requests per second per IP address without an API key and up to ten requests per second by default when an API key is supplied. Higher rates may be available by request. These limits apply to E-utilities and should not automatically be assumed to apply to every NCBI service. Researchers should check the current usage policy for the specific service they intend to use.
- When a program sends requests too frequently, the server may temporarily restrict access or return an error. The HTTP status code 429 Too Many Requests commonly indicates that the rate limit has been exceeded. Some servers provide a Retry-After response header indicating how long the client should wait before sending another request. A responsible program should detect this condition and pause rather than repeatedly sending requests.
- Rate limits can be managed by introducing delays between requests, retrieving records in batches, and avoiding unnecessary repeated queries. Instead of requesting information separately for every gene or protein, researchers should use batch endpoints when available. Caching previously retrieved results can also reduce the number of requests. For very large datasets, bulk downloads or database-specific command-line tools may be more suitable than repeated API calls.
- The following Python example introduces a short delay between requests:
import requests
import time
url = “https://example.org/api/genes”
for gene_id in [“GENE1”, “GENE2”, “GENE3”]:
response = requests.get(
url,
params={“id”: gene_id},
timeout=30
)
response.raise_for_status()
data = response.json()
print(data)
time.sleep(1)
- The time.sleep(1) command pauses the program for one second between requests. This is only an illustrative delay. The appropriate request frequency must follow the rules of the database being accessed.
- Authentication is the process of verifying the identity of a user or application before allowing access to a service. Some biological APIs provide public access without authentication, while others require an API key, username and password, access token, or institutional authorization. Authentication may be required to increase permitted request rates, access private datasets, or use specialized services.
- An API key is a unique value assigned to a user or application. It is included with an API request so that the database can identify the client and apply the appropriate access policy. NCBI, for example, allows users to create an API key through their NCBI account. The key can be supplied to E-utilities through the api_key parameter. Researchers should follow the database’s instructions for using and protecting their keys.
- A basic example of an API request containing an API key is: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=123456&api_key=YOUR_API_KEY
- The example uses a placeholder. Researchers must replace YOUR_API_KEY with their own valid key and should never publish a real key in an article, public repository, or shared script.
- Some APIs use token-based authentication. OAuth 2.0 is a widely used authorization framework in which an application obtains an access token and includes it in subsequent requests. A common method is to send the token in the HTTP authorization header. Tokens may expire and need to be renewed according to the service’s authentication procedure. OAuth and other token-based systems are more common in services that provide controlled access or allow applications to act on behalf of users.
- A typical authenticated HTTP request may use the following header: Authorization: Bearer YOUR_ACCESS_TOKEN
- For APIs that use an API key in a request header, the required header name depends on the service documentation. Researchers should not assume that every provider uses the same authentication format.
- Python can retrieve credentials from environment variables rather than placing them directly in source code. This reduces the risk of accidentally sharing API keys when scripts are published or transferred to other users.
import os
import requests
api_key = os.environ.get(“NCBI_API_KEY”)
if not api_key:
raise RuntimeError(“NCBI API key is not configured.”)
url = “https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi”
params = {
“db”: “pubmed”,
“id”: “123456”,
“api_key”: api_key,
“retmode”: “json”
}
response = requests.get(
url,
params=params,
timeout=30
)
response.raise_for_status()
data = response.json()
print(data)
- Before running this example, the API key must be configured as an environment variable named NCBI_API_KEY. On Linux or macOS, this can be done in the terminal:
- export NCBI_API_KEY=”your_actual_api_key”
- The key should be kept private. For long-term projects, researchers may use secure credential managers or protected environment configuration rather than storing secrets in scripts.
- R can also use environment variables to access API credentials. For example:
library(httr2)
api_key <- Sys.getenv(“NCBI_API_KEY”)
if (api_key == “”) {
stop(“NCBI API key is not configured.”)
}
response <- request(
“https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi”
) |>
req_url_query(
db = “pubmed”,
id = “123456”,
api_key = api_key,
retmode = “json”
) |>
req_perform()
data <- resp_body_json(response)
print(data)
- API reliability refers to the ability of a service and the software using it to retrieve information consistently and correctly. Even well-maintained biological databases may occasionally experience network interruptions, maintenance periods, server overload, temporary service failures, or changes in their API structure. A retrieval program should therefore be designed to handle errors rather than assuming that every request will succeed.
- HTTP status codes help identify the result of a request. A status code in the 200 range generally indicates success. A 400 response commonly indicates an invalid request, while 401 indicates that authentication is missing or invalid. A 403 response indicates that access is forbidden. A 404 response means that the requested resource was not found. A 429 response indicates that the client has sent too many requests. Status codes in the 500 range generally indicate a server-side problem.
- Programs should distinguish between errors that require a correction and errors that may be temporary. For example, an invalid accession number or incorrect query parameter may require the researcher to change the request. A temporary server error or network interruption may be resolved by trying again after a delay. Repeating an invalid request without correcting it will not solve the problem.
- A common reliability technique is retry logic. If a request fails because of a temporary network problem or server error, the program waits and attempts the request again. Exponential backoff increases the waiting time after each unsuccessful attempt. For example, the program may wait one second before the first retry, two seconds before the second, four seconds before the third, and so on. A maximum number of attempts should be defined to prevent endless retries.
- The following Python example demonstrates a basic retry mechanism:
import requests
import time
url = “https://example.org/api/genes”
max_retries = 5
for attempt in range(max_retries):
try:
response = requests.get(
url,
timeout=30
)
if response.status_code == 429:
retry_after = response.headers.get(“Retry-After”)
if retry_after:
wait_time = float(retry_after)
else:
wait_time = 2 ** attempt
time.sleep(wait_time)
continue
if response.status_code in [500, 502, 503, 504]:
time.sleep(2 ** attempt)
continue
response.raise_for_status()
data = response.json()
print(data)
break
except requests.exceptions.RequestException as error:
if attempt == max_retries – 1:
raise
time.sleep(2 ** attempt)
else:
raise RuntimeError(“Request failed after maximum retries.”)
- This example is a general starting point. A production workflow should also validate response content, distinguish permanent client errors from temporary failures, respect Retry-After when provided, and use a maximum waiting time. Authentication errors such as 401 and invalid requests such as 400 generally require correction rather than repeated retries.
- R programs can also implement retry logic. The following example uses the httr2 package, which provides facilities for HTTP requests and retry handling:
library(httr2)
url <- “https://example.org/api/genes”
response <- request(url) |>
req_timeout(30) |>
req_retry(
max_tries = 5,
backoff = ~ min(2 ^ (.x – 1), 60)
) |>
req_perform()
data <- resp_body_json(response)
print(data)
- This example uses a general endpoint. Retry behavior should be adapted to the API’s policies and the types of errors that can occur. Researchers should avoid retrying requests that repeatedly fail because of invalid parameters or missing permissions.
- Another important reliability technique is caching. Caching means storing previously retrieved results locally so that the same information does not need to be downloaded repeatedly. For example, if a program retrieves the annotation of a particular gene, it can save the response and reuse it in later analyses. This reduces network traffic, saves time, and helps maintain consistency within a project. Cached information should include its retrieval date and source version because biological records may be updated.
- Logging is also important in reliable API workflows. A log file can record the time of each request, the endpoint used, query parameters, response status, number of records retrieved, and any errors encountered. These records help researchers identify failed downloads and understand how data were collected. Logging is particularly valuable when a program runs for several hours or processes thousands of records.
- Large retrieval jobs should also use checkpoints. A checkpoint records the progress of a program so that it can resume from the last successfully completed stage after an interruption. For example, a program retrieving 10,000 protein records could save each completed batch to a local file. If the program stops after retrieving 6,000 records, it can continue with the remaining records rather than starting again. Checkpointing is useful for long-running genome and transcriptome projects.
- API responses should always be validated before the retrieved information is accepted for analysis. A successful request does not guarantee that the response contains the expected biological records. Programs should check that required fields are present, that sequences are not empty, that accession numbers are valid, and that the returned organism and record type match the research requirements. Researchers should also record database releases, genome assemblies, retrieval dates, and relevant accession versions.
- The choice of pagination method, request frequency, authentication mechanism, and reliability strategy depends on the database. NCBI E-utilities, UniProt REST API, Ensembl REST API, and other biological services have their own documentation and usage policies. For example, NCBI provides API keys and batch-retrieval mechanisms, while UniProt provides continuation links for paginated search results. Researchers should consult the official documentation before developing automated retrieval software.
- Pagination, rate limits, authentication, and API reliability are essential considerations in programmatic bioinformatics. Pagination allows large datasets to be retrieved in manageable batches, rate-limit management prevents excessive requests, authentication provides controlled access, and reliability techniques help programs recover from interruptions and errors. By combining these methods with caching, logging, validation, and checkpointing, researchers can develop efficient and reproducible workflows for retrieving biological information from online databases.