Data Analysis with pandas in Python for Bioinformatics

  • Data analysis is a fundamental component of bioinformatics because biological research generates large quantities of structured and semi-structured data. Gene-expression measurements, genomic variants, protein properties, clinical observations, sequencing quality metrics, and experimental results often need to be organized, cleaned, transformed, summarized, and statistically analyzed before meaningful conclusions can be drawn. Python’s pandas library provides powerful tools for handling tabular data and is widely used in bioinformatics workflows involving genomics, transcriptomics, proteomics, metabolomics, and systems biology.
  • pandas is an open-source Python library designed for data manipulation and analysis. It provides data structures that support labeled rows and columns, flexible indexing, missing-value handling, grouping, aggregation, merging, reshaping, and time-series operations. Its two principal data structures are the Series and the DataFrame. A Series represents a one-dimensional labeled collection of values, while a DataFrame represents a two-dimensional table with labeled rows and columns. These structures make it easier to manage biological measurements alongside gene identifiers, sample names, experimental conditions, genomic coordinates, and other metadata.
  • pandas can be installed using Python’s package manager:
  • python -m pip install pandas numpy
  • The library is conventionally imported using the alias pd. NumPy is often used alongside pandas because it provides efficient numerical operations that complement pandas’ labeled data structures.

import pandas as pd

import numpy as np

print(“pandas version:”, pd.__version__)

print(“NumPy version:”, np.__version__)

  • A pandas Series can store measurements associated with biological samples, genes, or proteins. Each value has an associated index, which can be used to identify the observation. For example, a Series might contain the expression measurements of one gene across several samples. Unlike an ordinary Python list, a Series supports label-based selection and a wide range of analytical operations.

import pandas as pd

expression = pd.Series(

    [12.5, 18.2, 9.7, 24.1],

    index=[“Sample_A”, “Sample_B”, “Sample_C”, “Sample_D”],

    name=”Gene_1″

)

print(expression)

print(“Mean expression:”, expression.mean())

print(“Highest expression:”, expression.max())

print(“Sample with highest expression:”, expression.idxmax())

  • In this example, each expression value is associated with a sample identifier. The mean() method calculates the average, max() identifies the largest measurement, and idxmax() returns the index label corresponding to that maximum. This label-based structure helps prevent confusion when measurements from different samples or experimental conditions are analyzed.
  • A DataFrame is particularly useful when several biological variables must be analyzed together. For example, a gene-expression dataset may contain gene identifiers in one column and measurements from multiple samples in other columns. Each row represents a gene, and each sample column contains the corresponding expression measurements.

import pandas as pd

data = {

    “Gene”: [“Gene_A”, “Gene_B”, “Gene_C”, “Gene_D”],

    “Sample_1”: [12, 25, 8, 17],

    “Sample_2”: [15, 22, 10, 19],

    “Sample_3”: [18, 28, 9, 21]

}

df = pd.DataFrame(data)

print(df)

print(“Dimensions:”, df.shape)

print(“Column names:”, df.columns.tolist())

print(“Data types:”)

print(df.dtypes)

  • The shape attribute reports the number of rows and columns, while columns and dtypes provide information about the variables and their data types. Checking these properties is an important first step when exploring an unfamiliar dataset. Incorrect data types, duplicated identifiers, or unexpected column names can cause errors later in the analysis.
  • Biological datasets are frequently stored in CSV or TSV files. pandas provides read_csv() for importing these formats. Although the function is named read_csv(), it can also read tab-separated files by setting sep=”\t”. The head() method displays the first few rows, tail() displays the last few rows, and info() summarizes the DataFrame’s structure and nonmissing values.

import pandas as pd

df = pd.read_csv(“gene_expression.csv”)

print(df.head())

print(df.info())

print(df.describe())

For a tab-separated file, the data can be imported as follows:

df = pd.read_csv(“gene_expression.tsv”, sep=”\t”)

  • The describe() method produces summary statistics for numerical columns, including counts, means, standard deviations, minimums, and quartiles. For biological data, these summaries can reveal unusual values, unexpected ranges, and potential data-quality issues. However, summary statistics alone do not establish whether the measurements are biologically meaningful or whether the data meet the assumptions of a particular statistical analysis.
  • pandas also supports importing Excel files, JSON documents, and data from several database systems. Functions such as read_excel(), read_json(), and read_sql() allow data from different sources to be integrated into one analytical workflow. The required dependencies must be installed for certain formats, such as Excel files. When working with genomic or sequencing formats such as FASTA, FASTQ, BAM, VCF, or GenBank, specialized libraries such as Biopython, pysam, or cyvcf2 are generally more appropriate for parsing the biological records. The resulting sequence information, annotations, or variant measurements can then be organized in pandas DataFrames for further analysis.
  • Data selection is one of the most common pandas operations. A column can be selected by its name, while multiple columns can be selected by passing a list of names. The .loc[] indexer selects rows and columns by labels, whereas .iloc[] selects them by integer position. Boolean indexing allows researchers to filter records according to scientific criteria, such as expression thresholds, quality scores, variant frequencies, or annotation categories.

import pandas as pd

df = pd.DataFrame({

    “Gene”: [“Gene_A”, “Gene_B”, “Gene_C”, “Gene_D”],

    “Expression”: [12, 25, 8, 17],

    “Condition”: [“Control”, “Treatment”, “Control”, “Treatment”]

})

high_expression = df[df[“Expression”] > 15]

treatment_genes = df[df[“Condition”] == “Treatment”]

print(high_expression)

print(treatment_genes)

  • In this example, the first filter selects genes with expression measurements above 15, while the second selects genes assigned to the treatment condition. The numerical threshold is illustrative and should not be interpreted as a universal biological cutoff. Real analyses should define filtering rules according to the measurement scale, experimental design, and scientific question.
  • Data cleaning is essential because biological datasets may contain missing values, duplicate records, inconsistent labels, invalid measurements, and formatting problems. pandas provides methods such as isna(), notna(), dropna(), and fillna() for identifying and handling missing values. The duplicated() and drop_duplicates() methods can identify and remove duplicate rows. String methods can standardize labels and identifiers, while conversion functions can help correct inappropriate data types.

import pandas as pd

df = pd.DataFrame({

    “Gene”: [“Gene_A”, “Gene_B”, “Gene_C”, “Gene_D”],

    “Expression”: [12.5, None, 18.3, 15.1],

    “Condition”: [“Control”, “control”, “Treatment”, “Treatment”]

})

print(“Missing values:”)

print(df.isna().sum())

df[“Condition”] = df[“Condition”].str.strip().str.title()

print(df)

  • This example identifies missing values and standardizes the capitalization of condition labels. Such standardization helps avoid treating Control and control as different experimental groups. Missing values should not automatically be replaced with zero or a group mean. In biological research, missingness can reflect measurement failures, detection limits, experimental design, or underlying biology. The method used to handle missing data should be justified and documented.
  • Numerical data can be converted to appropriate types using methods such as astype() and pd.to_numeric(). For example, an imported expression column may contain numbers stored as text because of inconsistent formatting. Conversion with pd.to_numeric() can identify invalid values, while its errors parameter controls how conversion failures are handled. Using errors=”coerce” converts invalid entries to missing values, which should then be inspected rather than silently ignored.

df[“Expression”] = pd.to_numeric(

    df[“Expression”],

    errors=”coerce”

)

  • Data transformation involves creating new columns, changing representations, or deriving measurements from existing data. For example, researchers may calculate expression ratios, transform numerical measurements, or derive quality-control indicators. pandas supports these operations through arithmetic, string processing, conditional selection, and methods such as assign().

import pandas as pd

df = pd.DataFrame({

    “Gene”: [“Gene_A”, “Gene_B”, “Gene_C”],

    “Control”: [10, 20, 15],

    “Treatment”: [15, 30, 12]

})

df[“Difference”] = df[“Treatment”] – df[“Control”]

print(df)

  • The new column records the difference between treatment and control measurements for each gene. Such differences can be useful for exploratory analysis, but they are not necessarily equivalent to statistically estimated treatment effects. Appropriate statistical models should account for the experimental design, biological replicates, dependence among observations, and other relevant sources of variation.
  • For positive values, researchers sometimes calculate fold changes by dividing the treatment measurement by the control measurement. Logarithmic fold change is often more convenient because increases and decreases become more symmetric around zero. However, ordinary ratios are undefined when the denominator is zero, and log transformations require careful treatment of zeros and negative values. Pseudocounts may be used in some contexts, but their choice affects the results and should be justified.

import numpy as np

import pandas as pd

df = pd.DataFrame({

    “Gene”: [“Gene_A”, “Gene_B”, “Gene_C”],

    “Control”: [10, 20, 15],

    “Treatment”: [20, 10, 30]

})

df[“Fold_change”] = df[“Treatment”] / df[“Control”]

df[“Log2_fold_change”] = np.log2(df[“Fold_change”])

print(df)

  • Here, the fold change compares treatment with control, and the log2 fold change expresses that ratio on a logarithmic scale. A value of 1 corresponds to a twofold increase, while −1 corresponds to a twofold decrease. These interpretations assume positive, nonzero measurements and the stated ratio direction. For RNA-sequencing count data, differential-expression tools such as DESeq2 or edgeR are generally preferable to calculating fold changes alone because they account for library size, variability, and the statistical structure of the data.
  • Grouping and aggregation allow researchers to summarize data according to categories. The groupby() method divides observations into groups, after which functions such as mean(), median(), sum(), count(), and agg() calculate group-level summaries. This is useful for comparing expression measurements across experimental conditions, summarizing variant counts by chromosome, calculating average protein properties by class, or examining the number of genes in functional categories.

import pandas as pd

df = pd.DataFrame({

    “Gene”: [“Gene_A”, “Gene_B”, “Gene_C”, “Gene_D”],

    “Expression”: [12, 25, 18, 30],

    “Condition”: [“Control”, “Control”, “Treatment”, “Treatment”]

})

summary = df.groupby(“Condition”)[“Expression”].agg(

    [“count”, “mean”, “median”, “std”]

)

print(summary)

  • The output summarizes the number of observations, mean, median, and standard deviation for each condition. These descriptive statistics can help researchers explore the data before performing inferential analyses. When replicates are limited, variability estimates may be unstable, and differences between group means should not automatically be interpreted as statistically significant effects.
  • Merging and joining datasets are particularly important in bioinformatics because information about the same gene, protein, variant, or sample may be stored in different files. One file may contain gene-expression measurements, another may contain gene annotations, and a third may contain pathway assignments. pandas provides merge() and join() to combine data using shared identifiers.

import pandas as pd

expression = pd.DataFrame({

    “Gene_ID”: [“G001”, “G002”, “G003”],

    “Expression”: [15.2, 27.1, 8.4]

})

annotations = pd.DataFrame({

    “Gene_ID”: [“G001”, “G002”, “G004”],

    “Function”: [

        “Metabolism”,

        “Signal transduction”,

        “DNA repair”

    ]

})

combined = expression.merge(

    annotations,

    on=”Gene_ID”,

    how=”left”,

    validate=”one_to_one”

)

print(combined)

  • The left join retains every gene from the expression table and adds annotation information when a matching identifier exists. Genes without matching annotations receive missing values in the annotation column. The validate parameter checks the expected relationship between the keys; in this example, it requires each key to be unique in both tables. This is useful because duplicate identifiers can unexpectedly multiply rows during a merge and distort subsequent analyses.
  • Data reshaping changes the arrangement of rows and columns without necessarily changing the underlying observations. pandas provides melt() to convert wide tables into long tables and pivot() or pivot_table() to create wider representations. Long-format data are often convenient for plotting, statistical modeling, and analysis of repeated measurements, while wide-format data can be useful for certain matrix-based operations.
  • For example, a gene-expression table may contain one column per sample. Converting it to long format produces columns for the gene, sample, and expression value. This structure can simplify comparisons across sample groups and integration with sample metadata. When using pivot(), the combination of identifiers must uniquely identify each value; pivot_table() can aggregate duplicate combinations when an appropriate aggregation function is specified.
  • Correlation analysis is another common task. pandas provides the corr() method to calculate correlations among numerical columns. Correlation matrices can help explore relationships between gene-expression measurements, protein abundance, phenotypic traits, or environmental variables. The default Pearson correlation measures linear association, while Spearman correlation can be requested for rank-based analysis.

import pandas as pd

df = pd.DataFrame({

    “Gene_A”: [10, 12, 15, 18, 20],

    “Gene_B”: [5, 7, 9, 11, 13],

    “Gene_C”: [20, 18, 15, 12, 10]

})

pearson = df.corr(method=”pearson”)

spearman = df.corr(method=”spearman”)

print(“Pearson correlation:”)

print(pearson)

print(“Spearman correlation:”)

print(spearman)

  • The resulting matrices show pairwise correlations among the genes. Strong correlation may indicate shared regulation, common biological processes, or technical effects, but it does not prove a direct biological interaction. Correlations may also be influenced by batch effects, outliers, confounding variables, and the selection of samples. More advanced analyses may require dedicated statistical models or network-analysis methods.
  • pandas can also support preliminary data exploration for genomic variants. A tabular dataset might contain chromosome, position, reference allele, alternate allele, quality score, and allele frequency. Researchers can filter variants by quality, summarize variants by chromosome, or merge them with gene annotations. However, variant file formats such as VCF have specialized conventions and metadata. Dedicated libraries should generally parse these files before pandas is used to analyze the resulting structured records.
  • Similarly, in single-cell transcriptomics, pandas can organize sample metadata, cell annotations, gene-level summaries, and quality-control tables. The expression matrices themselves may be large and sparse, making specialized tools such as AnnData and Scanpy more appropriate for the full workflow. pandas remains valuable for joining annotations, examining cell categories, and summarizing quality-control measurements.
  • Exporting data is the final stage of many pandas workflows. The to_csv() method writes a DataFrame to a CSV or TSV file, while to_excel() exports data to an Excel workbook. Other methods support formats such as JSON and Parquet, depending on the installed dependencies. Researchers should choose an output format that preserves the required data types and metadata. Identifiers such as gene IDs and chromosome labels should not be unintentionally converted into numerical values, and missing-value representations should be handled consistently.

df.to_csv(“processed_results.csv”, index=False)

df.to_csv(“processed_results.tsv”, sep=”\t”, index=False)

  • Setting index=False prevents pandas from writing the DataFrame’s row index as an additional column. If the index contains meaningful biological identifiers, however, it should be retained or explicitly included as a named column. Data provenance, preprocessing choices, software versions, and filtering criteria should be documented to support reproducibility.
  • Although pandas is powerful, it has limitations. Large datasets may require substantial memory, particularly when columns contain general-purpose Python objects or many repeated strings. Loading an entire dataset into memory may be inefficient when only a subset of the records is needed. Researchers can improve performance by selecting only required columns, specifying appropriate data types, using categorical data for repeated labels, processing files in chunks, or using columnar formats such as Parquet. For numerical calculations over large arrays, NumPy may be more appropriate, while distributed or out-of-core tools may be needed for datasets that exceed available memory.
  • Reliable pandas analysis also requires careful handling of missing values, duplicate identifiers, inconsistent sample labels, and unexpected data types. Researchers should verify row counts before and after merges, check the uniqueness of biological identifiers where required, validate numerical ranges, and compare summaries with known expectations. Automated tests can be used to confirm that data-cleaning and transformation functions behave correctly when new datasets are processed.
  • In conclusion, pandas is a fundamental Python library for organizing, cleaning, transforming, summarizing, and analyzing tabular biological data. Its Series and DataFrame structures simplify data selection, filtering, aggregation, merging, reshaping, and export. When combined with NumPy, SciPy, Biopython, and specialized bioinformatics frameworks, pandas supports reproducible workflows across genomics, transcriptomics, proteomics, metabolomics, and systems biology. Effective use of pandas requires not only familiarity with its methods but also an understanding of biological data quality, experimental design, statistical interpretation, and the limitations of the chosen analytical approach.
Author:

Leave a Reply

Your email address will not be published. Required fields are marked *