Remote Servers, SSH, and Secure File Transfer

  • Remote servers, Secure Shell (SSH), and secure file transfer are essential components of modern bioinformatics because biological datasets and computational analyses often require more storage, memory, processing power, or specialized software than a personal computer can provide. Researchers routinely access remote Linux servers, high-performance computing (HPC) clusters, cloud computing platforms, and institutional computing facilities to analyze genomic, transcriptomic, proteomic, and other biological data. Understanding remote access and secure data transfer enables researchers to manage files, run computational tools, automate analyses, and collaborate efficiently while protecting sensitive research information.
  • A remote server is a computer that provides computational resources, storage, applications, or other services to users connecting from another computer. In bioinformatics, remote servers may host reference genomes, sequencing datasets, biological databases, analysis software, and workflow environments. They may also provide access to large amounts of memory, multiple processors, GPUs, and distributed storage systems. A local computer can connect to a remote server over a network, allowing the researcher to execute commands and manage data without physically accessing the machine.
  • Remote computing is particularly useful when analyzing whole-genome sequencing data, assembling large genomes, processing RNA sequencing experiments, performing metagenomic classification, or running computationally intensive molecular simulations. Instead of downloading every dataset and installing all software locally, researchers can transfer data to a remote system, run analyses where the resources are available, and retrieve the results when needed. This approach reduces the limitations imposed by local hardware and can improve the organization of large-scale research projects.
  • Secure Shell, commonly known as SSH, is a network protocol that provides encrypted communication between computers. It allows users to log in to remote systems, execute commands, transfer files through compatible tools, and establish secure tunnels. SSH protects data transmitted over the connection from many forms of network interception and supports authentication mechanisms that help verify the user’s identity. It is widely used on Linux and Unix-like systems and is also available on modern versions of Windows.
  • A typical SSH connection is established using the ssh command:

ssh username@server.example.org

  • Here, username is the account name on the remote server, and server.example.org is the server’s hostname or address. Both values are illustrative and must be replaced with the actual connection details provided by the institution or service administrator. After the connection is established and authentication succeeds, the user can execute commands within the remote computing environment. The remote shell generally provides access to the user’s permitted files, installed software, computational resources, and system utilities.
  • When connecting to a server for the first time, SSH may display the server’s host key fingerprint and ask whether the host should be trusted. This fingerprint helps establish the server’s identity and should be verified through an authoritative source, such as the system administrator or official institutional documentation. Blindly accepting an unexpected host key, particularly after a previously trusted key changes, can expose a connection to a potential impersonation attack. SSH stores trusted host keys in a file commonly located at ~/.ssh/known_hosts, allowing subsequent connections to detect certain identity changes.
  • SSH normally supports password-based and public-key authentication, depending on server configuration. Public-key authentication uses a key pair consisting of a private key and a public key. The private key remains under the user’s control, while the public key is installed or registered on the remote server. During authentication, the client proves possession of the private key without sending the private key itself to the server. This method is often convenient for researchers who connect frequently to computing clusters or cloud servers.
  • An SSH key pair can be generated with the following command:

ssh-keygen -t ed25519 -C “research-access”

  • The command creates an Ed25519 key pair when supported by the installed SSH implementation and server configuration. Users should normally protect the private key with a passphrase and store it in a secure location. The default private and public key filenames are often id_ed25519 and id_ed25519.pub, respectively. The public key can then be registered with the remote service according to its instructions. Some environments require keys to be added through an administrative portal rather than copied directly to the server.
  • On systems where it is supported and permitted, the ssh-copy-id utility can install a public key on a remote account:

ssh-copy-id username@server.example.org

  • This command is a convenience rather than a requirement. If the utility is unavailable, the public key may need to be added manually to the remote account’s ~/.ssh/authorized_keys file. The private key must never be copied to the remote server or shared with other users. Access permissions are important: SSH may reject private keys or authorization files that are accessible to unauthorized users. A private key can often be secured with chmod 600 ~/.ssh/id_ed25519, while the .ssh directory commonly uses chmod 700 ~/.ssh.
  • An SSH agent can store access to an unlocked private key during a working session, reducing the need to enter its passphrase repeatedly. Tools such as ssh-add can add a key to an agent, provided an agent is available and correctly configured. Users should consider the security of the computer and session where the agent operates, especially on shared systems. Institutional security policies may also require hardware-backed keys, multifactor authentication, temporary credentials, or other authentication mechanisms.
  • For frequent connections, SSH configuration can simplify commands and centralize connection settings. The configuration file is commonly located at ~/.ssh/config. An example entry is:
  • Host bioinfo-server

    HostName server.example.org

    User username

    Port 22

    IdentityFile ~/.ssh/id_ed25519

  • After configuring the entry with the appropriate values, the user can connect by running ssh bioinfo-server. Port 22 is the default SSH port, but some servers use a different port. The IdentityFile setting identifies the private key to use for authentication. Users should not place private-key contents or passwords in the configuration file. Configuration options should match the server’s requirements, and unnecessary options that weaken host verification should be avoided.
  • Once connected to a remote server, researchers can use familiar Linux commands to navigate directories, inspect files, run software, and organize data. Commands such as pwd, ls, cd, mkdir, cp, mv, find, and du work much as they do in a local Linux environment, subject to the permissions and tools available on the server. For example, pwd displays the current working directory, while du -sh data/ estimates the total disk space occupied by a directory. Understanding remote directory structures is important because paths on the server are separate from paths on the local computer.
  • Remote login is not the same as transferring files. SSH provides the secure communication foundation for several tools that copy or synchronize data between computers. The most common command-line options include Secure Copy (SCP), SSH File Transfer Protocol (SFTP), and rsync over SSH. Each serves a slightly different purpose. SCP is useful for straightforward copying, SFTP provides interactive or scripted file-transfer operations, and rsync is particularly valuable for synchronizing large directories and transferring only changed data when its comparison rules permit.
  • SCP copies files between local and remote systems using an SSH-based connection. To transfer a local file to a remote server, a researcher can use:

scp sample.fastq.gz username@server.example.org:~/data/

  • This command copies sample.fastq.gz into the remote user’s data directory. The directory must exist or be created beforehand, and the user must have permission to write to it. To copy a remote file to the local working directory, the direction is reversed:

scp username@server.example.org:~/results/summary.tsv .

  • The period at the end represents the current local directory. A directory can be transferred recursively using the appropriate recursive option, such as scp -r, although synchronization tools may be more efficient for repeated transfers. SCP implementations and option behavior can differ, so the local command’s documentation should be consulted when handling unusual paths, ports, or large transfers.
  • SFTP provides a secure file-transfer interface over SSH. A connection can be opened with:

sftp username@server.example.org

  • Once connected, users can use commands such as pwd and ls to inspect the remote working directory, lpwd and lls to inspect the local directory, cd and lcd to change remote and local directories, and get and put to download and upload files. For example, get results.tsv downloads a remote file, while put metadata.tsv uploads a local file. The get -r and put -r forms support recursive directory transfers in commonly used OpenSSH SFTP clients. SFTP is useful for interactive work and can also be used in scripted workflows where supported.
  • The rsync utility is especially useful for bioinformatics because sequencing files and intermediate analysis outputs can be very large. It can compare source and destination files and transfer data efficiently, often avoiding the need to retransmit unchanged content. A typical upload command is:

rsync -avh –progress sample_data/ \

    username@server.example.org:~/project/sample_data/

  • The options shown enable archive-style copying, human-readable output, and progress reporting. The trailing slash after sample_data/ means that its contents are copied into the destination directory rather than creating another nested sample_data directory inside it. Without the trailing slash, the directory itself is normally included in the destination path. This distinction matters when organizing large sequencing projects.
  • A reverse transfer can retrieve remote results:

rsync -avh –progress \

    username@server.example.org:~/project/results/ ./results/

  • For repeated transfers, rsync can reduce network traffic by detecting differences between files. Its behavior depends on options, file metadata, and the selected comparison method. Interrupted transfers can often be resumed efficiently by running the same command again. For very large files or unreliable connections, options such as –partial may help retain incomplete transferred files, although the appropriate settings depend on the workflow and the risks of leaving partial data in place.
  • Researchers should understand that the –delete option in rsync removes destination files that do not exist in the source, subject to the command’s path and filtering rules. Although this option can help create an exact mirror, it can also delete important data if the source or destination is specified incorrectly. Before using destructive options, researchers should perform a dry run with –dry-run and carefully review the proposed changes. Maintaining a separate backup is advisable for important research datasets.
  • Secure file transfer is especially important for genomic and clinical data because files may contain personal, confidential, or otherwise restricted information. Encrypted transfer protects data while it travels between systems, but it does not automatically secure files stored on either computer. Access controls, disk encryption, appropriate account permissions, secure backups, and institutional data-handling policies remain necessary. Researchers working with human genomic data should follow applicable ethics approvals, consent requirements, data-sharing agreements, and privacy regulations. Data should be transferred only to authorized systems and only when the relevant permissions allow it.
  • File integrity should be checked after transferring important datasets. A successful transfer message is useful, but it does not replace verification when data integrity is critical. Cryptographic checksums, such as SHA-256, can be computed on the source and destination files and compared to confirm that their contents match. For example:

sha256sum sample.fastq.gz

  • The command prints a checksum for the file. Running it on both systems should produce the same checksum if the files are identical. Checksums are especially useful for FASTQ files, reference genomes, alignment files, variant call files, and archived analysis outputs. Some data providers also publish checksums that can be used to verify downloaded files against the original distribution.
  • Large bioinformatics transfers can be affected by network interruptions, bandwidth limits, storage quotas, and connection timeouts. Researchers should check the available disk space before transferring large datasets and avoid sending unnecessary intermediate files across the network. Compression may reduce transfer time for some data, but many biological file formats, including .gz-compressed FASTQ files and compressed BAM files, are already compressed. Recompressing them may provide little benefit while increasing CPU use. Where available, high-performance data-transfer services, institutional data gateways, or parallel transfer tools may be more appropriate for extremely large projects.
  • Remote analysis often takes longer than a single interactive session. An SSH connection may close because of network problems, laptop sleep, or session timeouts. Terminal multiplexers such as tmux and screen allow users to keep terminal sessions running on the remote server and reconnect later. For example, a researcher can start tmux, launch a permitted interactive task, detach from the session, and reconnect through SSH at a later time. However, terminal multiplexers do not replace a job scheduler on an HPC cluster. Long-running or resource-intensive jobs should generally be submitted through the cluster’s supported scheduler, such as Slurm or PBS, according to local policies.
  • SSH tunneling can provide secure access to services that are available only within a remote network. For example, a local port can be forwarded to a service running on a remote machine, enabling a browser or other client to reach it through an SSH connection. This may be useful for accessing a permitted visualization service, Jupyter interface, or database that is not exposed directly to the public internet. Tunnels should be configured according to institutional guidance, and users should ensure that the service is appropriately authenticated and does not expose confidential data to unintended users. SSH tunneling should not be used to bypass access restrictions or security controls.
  • Remote bioinformatics work may also involve graphical applications. X11 forwarding can transmit graphical application interfaces over SSH when the client, server, and required display infrastructure support it. However, graphical forwarding can be slower and more complicated than command-line work, especially for large visualization tasks. Remote desktop systems, browser-based interfaces, and interactive computing platforms may be more appropriate in some environments. Regardless of the interface, the same principles of secure authentication, authorized access, and appropriate data handling apply.
  • Automation can simplify routine transfers and reduce repetitive manual work. Researchers may write shell scripts that use scp, sftp, or rsync to upload input data, retrieve results, and record transfer logs. Scripts should check exit statuses, verify that expected files exist, and avoid assuming that every transfer succeeds. For automated connections, public-key authentication or another approved noninteractive authentication method is usually preferable to embedding passwords in scripts. Private keys and authentication tokens must not be committed to public version-control repositories or included in shared workflow files.
  • For example, a simple transfer script can use rsync and stop if the command fails:

#!/usr/bin/env bash

set -euo pipefail

LOCAL_DIR=”./results/”

REMOTE=”username@server.example.org:~/project/results/”

rsync -avh –progress “LOCALDIR””REMOTE”

echo “Transfer completed successfully.”

  • This is a basic template, not a complete production transfer system. It assumes that SSH access is already configured and that the local and remote paths are correct. The success message is printed only if rsync returns a successful exit status. For research pipelines, additional checks may be needed to verify file counts, checksums, permissions, logs, and whether the destination contains complete and expected results.
  • Troubleshooting remote access begins with checking the hostname, username, network connection, port, and authentication method. An error such as Permission denied (publickey) often indicates that the server did not accept the available key or that the account is configured for a different authentication method. A Connection timed out error may indicate network restrictions, an unreachable server, or an incorrect port. A Host key verification failed message may indicate a changed or mismatched host key; users should verify the server’s identity before taking corrective action. Verbose SSH output, enabled with ssh -v username@server.example.org, can provide additional diagnostic information without requiring changes to server settings.
  • File-transfer failures may result from insufficient disk space, missing directories, incorrect permissions, expired credentials, interrupted connections, or filenames containing spaces and special characters. Quoting local paths and using clear directory structures helps prevent command-line parsing errors. When moving many files, researchers should avoid relying on broad wildcards without inspecting which files they match. Transfer logs and checksums can help distinguish a completed transfer from one that ended prematurely.
  • Good remote-computing practice includes using unique accounts, protecting private keys, verifying server host keys, keeping software updated, and using only authorized access methods. Users should avoid sharing accounts, leaving credentials in shell history, or copying confidential datasets to personal devices without permission. Remote access policies may restrict login methods, data storage locations, permitted software, or the use of external cloud systems. Researchers should also close sessions when appropriate and protect unattended devices that retain authenticated SSH access.
  • In reproducible bioinformatics workflows, remote access and file transfer should be documented alongside the analytical commands. Useful records include server or environment identifiers, software versions, input filenames, transfer checksums, job identifiers, output locations, and relevant workflow logs. Passwords, private keys, and other secrets should never be recorded in research notebooks or shared documentation. Data-transfer scripts can be version-controlled when they contain no credentials and use configurable paths. Recording where data originated and how it was transferred helps researchers trace results and identify missing or corrupted inputs.
  • Remote servers, SSH, and secure file-transfer tools form a practical foundation for large-scale biological data analysis. SSH enables encrypted remote access, SCP and SFTP support secure file copying, and rsync provides efficient synchronization for large datasets and repeated transfers. When these tools are combined with checksum verification, secure authentication, careful storage management, job scheduling, and reproducible documentation, bioinformatics researchers can work reliably across personal computers, institutional servers, HPC clusters, and cloud environments. These skills are essential for handling modern sequencing datasets and building secure, efficient, and reproducible computational biology workflows.
Author:

Leave a Reply

Your email address will not be published. Required fields are marked *