- Bulk downloading is an important part of bioinformatics because researchers often need to obtain large amounts of biological data from online databases. Instead of downloading individual records one by one, they can download complete datasets or selected collections of files directly to their computers or servers. These datasets may include DNA sequences, RNA sequences, protein sequences, genome assemblies, sequencing reads, gene annotations, genetic variants, and protein structures.
- Many biological databases provide bulk-download services to support large-scale research. Important examples include NCBI, the European Nucleotide Archive (ENA), Ensembl, UniProt, and the Protein Data Bank (PDB). Researchers use these resources to collect biological information for genome analysis, transcriptomics, proteomics, comparative genomics, metagenomics, and other bioinformatics applications.
- FTP and HTTPS are two common methods of downloading biological data. FTP stands for File Transfer Protocol and has traditionally been used to transfer large files between servers and computers. HTTPS stands for Hypertext Transfer Protocol Secure and provides encrypted communication between a user’s computer and a remote server. Many biological databases now provide HTTPS download links, while some also maintain FTP-compatible services or alternative transfer methods.
- Bulk downloading usually begins by selecting the appropriate biological database and identifying the required files. Researchers should decide which organism, genome assembly, accession number, sequencing project, or biological dataset they need. They can then search the database, locate the relevant download directory, and select the files required for their research. Downloading only the necessary files saves time, storage space, and computational resources.
- Several major biological databases provide bulk-download facilities. NCBI provides genome assemblies, nucleotide sequences, protein sequences, annotations, and other biological resources. The European Nucleotide Archive provides raw sequencing reads and assembled sequence data. Ensembl provides genome sequences, gene annotations, transcript sequences, protein sequences, and variation data. UniProt provides protein sequences and functional annotations, while the Protein Data Bank provides three-dimensional biological structure files.
- The command-line tools wget and curl are widely used for downloading biological data from FTP and HTTPS servers. These tools allow researchers to download files directly from a terminal without opening a web browser. They are particularly useful when working with remote Linux servers or high-performance computing systems.
- The following command downloads a file using wget: wget https://example.org/data/sequences.fasta.gz
- The following command downloads a file using curl: curl -L -o sequences.fasta.gz https://example.org/data/sequences.fasta.gz
- In these examples, the address is illustrative. Researchers must replace it with the actual download address provided by the biological database. The -L option allows curl to follow redirects, while -o specifies the name of the downloaded file.
- Biological databases often distribute large files in compressed formats such as .gz and .bz2. Compression reduces file size and makes downloading and storing large datasets more efficient. Common examples include .fasta.gz, .fastq.gz, .gff.gz, and .gtf.gz. Researchers can decompress these files using Linux commands when necessary.
- For example, the following command decompresses a gzip file: gunzip sequences.fasta.gz
- To extract a compressed file while retaining the original compressed version, researchers can use: gzip -dk sequences.fasta.gz
- Some bioinformatics programs can read compressed files directly, so decompression is not always required.
- The rsync utility is another useful tool for downloading and synchronizing large biological datasets. It can transfer multiple files and avoid downloading files that are already present and unchanged. This makes it useful when researchers need to maintain local copies of genome databases or regularly updated biological resources.
- A basic example of the rsync command is: rsync -av –progress source_directory/ destination_directory/
- Here, -a enables archive mode, -v displays detailed information, and –progress shows transfer progress. For downloading from a biological database, researchers must use the remote address and directory structure specified by that database. The command should not be used with an invented or unsupported server address.
- Python can also automate the downloading of biological datasets. The requests library allows Python programs to access HTTPS services and retrieve files directly from remote servers. For large files, it is better to download data in smaller chunks rather than loading the complete file into computer memory.
- The following Python example downloads a compressed FASTA file:
import requests
url = “https://example.org/data/sequences.fasta.gz”
output_file = “sequences.fasta.gz”
response = requests.get(url, stream=True, timeout=60)
response.raise_for_status()
with open(output_file, “wb”) as file:
for chunk in response.iter_content(chunk_size=1024 * 1024):
if chunk:
file.write(chunk)
print(“Download completed successfully.”)
- The stream=True option allows the file to be downloaded in chunks. The timeout option prevents the program from waiting indefinitely, and raise_for_status() reports unsuccessful HTTP responses. Researchers should add error handling, retry mechanisms, and file-integrity checks when developing larger automated download programs.
- R can also be used to download biological datasets. The built-in download.file() function supports file retrieval from HTTPS addresses. This is useful when researchers want to combine data downloading with statistical analysis and other bioinformatics operations in R.
- The following R example downloads a compressed FASTA file:
url <- “https://example.org/data/sequences.fasta.gz”
download.file(
url = url,
destfile = “sequences.fasta.gz”,
mode = “wb”,
method = “libcurl”
)
print(“Download completed successfully.”)
- The destfile argument specifies where the file will be saved. The mode = “wb” option ensures that the file is written in binary mode, which is important for compressed files. R packages such as curl, httr2, and Bioconductor packages can provide additional facilities for retrieving and processing biological data.
- Bulk downloading can also be automated using file lists and metadata tables. Many biological databases provide accession lists, manifests, and file reports containing information about available datasets. These files may include accession numbers, sample identifiers, file names, download addresses, and data types. A Python or R program can read such a list and automatically download all required files. This approach is especially useful when working with hundreds or thousands of sequencing samples.
- Different types of biological data are distributed in different file formats. DNA and protein sequences are commonly stored in FASTA format, while raw sequencing reads are often provided in FASTQ format. Genome annotations are commonly available in GFF3 or GTF format. Genetic variants are frequently distributed in VCF format, and aligned sequencing reads are commonly stored in BAM or CRAM format. Protein structures are often provided in PDB or mmCIF format. Researchers should understand the purpose of each format before downloading and processing the data.
- Metadata should always be collected together with the downloaded biological files. Metadata provide information about the origin and characteristics of a dataset. For sequencing projects, they may include organism names, sample identifiers, tissue types, experimental conditions, sequencing platforms, and accession numbers. For genome assemblies, they may include assembly accessions, version numbers, genome build information, and annotation releases. Without reliable metadata, it can be difficult to interpret biological results or reproduce an analysis.
- Data integrity is another important consideration when downloading large datasets. Network interruptions, incomplete transfers, and storage problems can result in damaged or incomplete files. Researchers should check downloaded file sizes and use checksums provided by the database whenever available. Checksums are calculated values that help determine whether a downloaded file matches the original file.
- For example, the following Linux command calculates an MD5 checksum: md5sum sequences.fasta.gz
- The resulting checksum can be compared with the value supplied by the database. SHA-256 checksums can also be calculated using: sha256sum sequences.fasta.gz
- These checks help identify accidental file corruption or changes during transfer.
- Large biological datasets can require considerable storage space. Raw sequencing data, genome assemblies, and structural datasets may contain many large files. Researchers should check available disk space before starting a download and organize files into separate directories. A useful structure may include folders for raw data, processed data, reference genomes, annotations, metadata, and analysis results. Clear file names should preserve important accession numbers and version information.
- Biological databases are regularly updated. New sequences may be added, existing records may be corrected, and gene annotations may change between releases. Consequently, downloading the same dataset at different times may produce different results. Researchers should record the database name, download address, retrieval date, accession number, genome assembly, database release, and file checksum. Keeping original downloaded files is also useful when an analysis needs to be reproduced later.
- For very large datasets, specialized transfer services may be more efficient than ordinary downloads. Globus and Aspera are examples of technologies used to transfer large research datasets. They can support high-volume transfers and help manage interruptions when moving data between research institutions, storage systems, and computational facilities. Their availability depends on the database and the storage systems involved.
- Researchers must also follow database access policies. Some biological datasets are freely available, while others require registration, authentication, or special permission. Human genomic and clinical datasets may contain sensitive information and can be subject to ethical approvals, privacy requirements, and data-use agreements. Automated downloading must respect these restrictions and should never be used to bypass access controls.
- Bulk downloads and FTP/HTTPS access are essential for large-scale bioinformatics research. They allow researchers to obtain complete biological datasets, build local reference collections, process sequencing projects, and develop automated analysis pipelines. By using tools such as wget, curl, rsync, Python, and R, researchers can download biological information efficiently. Careful management of metadata, file formats, storage, checksums, and database versions helps ensure that downloaded data remain reliable and suitable for reproducible scientific analysis.