Variant Filtering and Quality Scores

  • Variant filtering and quality assessment are essential stages in bioinformatics because raw variant-calling results often contain both genuine genetic variants and false-positive calls caused by sequencing errors, alignment problems, contamination, and technical artifacts. Variant callers identify candidate single-nucleotide variants (SNVs), insertions and deletions (indels), and structural variants by analysing sequencing reads aligned to a reference genome. However, not every candidate variant is biologically real or sufficiently supported for downstream analysis. Variant filtering evaluates the reliability of these calls using quality scores, read evidence, genotype information, statistical measures, and biological context. It helps researchers produce more accurate variant datasets for clinical genetics, population genomics, cancer research, evolutionary studies, and agricultural breeding.
  • Variant quality scores summarize the strength of evidence supporting a variant call. They are usually calculated using statistical or probabilistic models that evaluate sequencing reads, base quality, mapping quality, allele observations, and other characteristics. A high-quality score generally indicates stronger evidence for a variant, while a low score suggests that the call may be unreliable. However, quality scores are not identical across variant callers. Different programs use different algorithms, assumptions, and scoring systems, so a score generated by one caller should not automatically be interpreted in the same way as a score generated by another.
  • The Variant Call Format (VCF) is commonly used to store variant calls and their quality information. Each VCF record contains fields such as chromosome, position, reference allele, alternate allele, quality score, filter status, and additional annotations. The QUAL field represents the variant-level quality score assigned by the caller, while the FILTER field indicates whether the record passed the applied filtering criteria. The INFO and FORMAT fields can contain additional site-level and sample-level metrics. These may include sequencing depth, genotype quality, allele depth, mapping quality, strand-bias statistics, and other caller-specific measurements.
  • A simplified VCF record may look like this:

#CHROM  POS     ID  REF ALT QUAL FILTER INFO FORMAT SAMPLE

chr1    10500   .   A   G   60   PASS   .    GT:DP:GQ  0/1:35:45

  • In this example, the reference allele is A and the alternate allele is G. The QUAL value is 60, and the FILTER value is PASS, indicating that the record passed the filters applied in this workflow. The genotype is 0/1, representing a heterozygous genotype in a typical diploid setting. The DP value indicates a depth of 35, while GQ represents the genotype quality score. These values describe different aspects of confidence: QUAL concerns the variant call, whereas GQ concerns the genotype assigned to the individual sample.
  • The Phred quality scale is widely used in sequencing and variant analysis. It expresses the probability of an error on a logarithmic scale. The general relationship is:

[
Q = -10\log_{10}(P_{\text{error}})
]

  • where (Q) is the Phred quality score and (P_{\text{error}}) is the estimated probability of error. A higher Phred score corresponds to a lower estimated error probability. For example, a score of 20 corresponds to an estimated error probability of 1%, while a score of 30 corresponds to 0.1%. A score of 40 corresponds to 0.01%. These interpretations apply to the probability represented by the particular quality score; a variant-level score, base-quality score, mapping-quality score, and genotype-quality score refer to different error models and should not be treated as interchangeable.
  • Base quality (BQ) measures the confidence that an individual nucleotide has been read correctly during sequencing. Sequencing instruments assign quality scores to individual bases, and these values are stored in FASTQ records. Low base quality can result from signal deterioration, sequencing chemistry limitations, or technical problems. Variant callers use base-quality information when evaluating whether observed alternate alleles are supported by reliable nucleotide observations. Reads containing many low-quality bases may provide weak evidence for a variant, even when the overall read depth is high.
  • Mapping quality (MQ) measures the confidence that a sequencing read has been aligned to the correct genomic location. A read may have high base quality but still map ambiguously if it originates from a repetitive sequence or duplicated genomic region. Mapping quality is therefore particularly important in regions containing segmental duplications, pseudogenes, repetitive elements, and paralogous sequences. Low mapping quality can produce false-positive variant calls because reads from another genomic location may be incorrectly aligned to the region being analysed. Mapping quality is commonly represented in alignment files such as SAM and BAM and is used by many variant callers during filtering.
  • Read depth, commonly represented by DP, describes the number of sequencing reads or observations supporting a genomic position or genotype. Higher depth generally provides more evidence for genotype estimation, but high depth does not automatically guarantee a reliable call. If the reads are biased, duplicated, contaminated, or incorrectly mapped, additional sequencing may reinforce an erroneous result. Low-depth positions can produce uncertain genotypes, missed variants, or inaccurate allele-frequency estimates. Depth should therefore be considered together with mapping quality, base quality, allele balance, and other metrics.
  • Allele depth (AD) records the number of reads supporting each allele at a variant site. For a biallelic variant, the reference and alternate allele depths may be reported separately. The variant allele fraction (VAF), sometimes called the variant allele frequency in somatic analysis, is calculated as the proportion of allele observations supporting the alternate allele. For example, if 18 of 36 informative reads support the alternate allele, the observed VAF is 0.5. In a typical diploid germline sample, a heterozygous variant may have a VAF near 0.5, but deviations can occur because of sampling variation, mapping bias, copy-number changes, contamination, or mosaicism. In tumour samples, VAF is additionally influenced by tumour purity, subclonal composition, and local copy number.
  • Genotype quality (GQ) measures confidence in the genotype assigned to a sample at a particular variant site. It is commonly represented as a Phred-scaled measure of the probability that the assigned genotype is incorrect. A high GQ indicates strong evidence for the selected genotype, while a low GQ indicates uncertainty. A variant may have a high site-level quality score but a low genotype quality in one sample, particularly when the sample has insufficient coverage or conflicting read evidence. Genotype quality is therefore important when filtering individual genotypes rather than entire variant sites.
  • Allele balance is another useful measure, especially for heterozygous variants. It compares the number of reads supporting the alternate allele with the total number of reads supporting the reference and alternate alleles. In an ordinary diploid germline sample, a heterozygous variant is often expected to have an allele balance near 0.5. However, the expected balance may differ in regions affected by copy-number changes, allelic imbalance, mapping bias, or unusual ploidy. In cancer genomics, allele balance can provide information about tumour purity, loss of heterozygosity, and subclonal mutations. Filtering based on allele balance must therefore account for the biological context rather than applying one universal threshold.
  • Strand bias occurs when alternate alleles are supported disproportionately by reads from one sequencing orientation. A genuine variant is generally expected to have support from both forward and reverse read orientations, although the balance may vary with library preparation, local sequence context, and sequencing technology. Strong strand bias can indicate a technical artifact. Several variant callers calculate strand-bias statistics, including Fisher’s exact test-based measures. In GATK, annotations such as FisherStrand (FS) and StrandOddsRatio (SOR) can be used to evaluate this type of bias.
  • Read-position bias occurs when alternate alleles are concentrated near the ends of sequencing reads rather than distributed across read positions. Sequencing errors and alignment artifacts are sometimes more common near read ends. The ReadPosRankSum annotation in GATK compares the read positions of reference and alternate observations using a statistical test. A strong difference may indicate that the variant is supported mainly by observations in less reliable read positions. This metric is one component of quality assessment and should be interpreted with other evidence.
  • Other useful quality annotations include mapping-quality rank-sum statistics, base-quality rank-sum statistics, read-depth ratios, clipping information, and local sequence complexity. MappingQualityRankSumTest evaluates differences in mapping quality between reads supporting reference and alternate alleles. BaseQualityRankSumTest compares base qualities between these groups. Excessive soft clipping may indicate that reads do not align cleanly to the reference, which can be relevant near insertions, deletions, and structural rearrangements. No single annotation can reliably identify every technical artifact, so multiple sources of evidence are generally considered together.
  • Variant filtering can be performed at the site level or genotype level. Site-level filtering removes or flags an entire variant record when the evidence across the dataset is inadequate. Genotype-level filtering marks individual sample genotypes as low quality or missing while retaining the variant site for other samples. This distinction is important in cohort studies. A variant may be genuine in some individuals but poorly supported in others. Removing the entire site because of low-quality genotypes in a subset of samples can discard useful information, whereas retaining uncertain genotypes without marking them can distort allele-frequency estimates and association analyses.
  • Hard filtering is a commonly used approach in which variants are retained or removed according to predefined thresholds. For example, a workflow may exclude variants with low quality scores, insufficient read depth, poor mapping quality, or strong strand bias. GATK provides VariantFiltration for applying filtering expressions to VCF files. BCFtools also supports filtering through expressions based on VCF fields. Hard filtering is relatively straightforward and can be useful when appropriate thresholds are known, but thresholds should be selected according to the sequencing platform, variant caller, organism, sample type, and study objective.
  • An illustrative BCFtools filtering command is:

# Retain variants with QUAL of at least 30

bcftools filter \

    -i ‘QUAL>=30’ \

    cohort.vcf.gz \

    -Oz \

    -o cohort.qual_filtered.vcf.gz

# Index the filtered VCF

bcftools index cohort.qual_filtered.vcf.gz

  • This command applies only a variant-level quality threshold. It does not evaluate all relevant sources of error and should not be treated as a complete filtering workflow. A more extensive filtering strategy may include depth, genotype quality, mapping quality, allele balance, missingness, and caller-specific annotations. The thresholds should be based on empirical assessment and validated for the particular dataset.
  • GATK VariantFiltration can apply several site-level filters using annotations present in a VCF. An illustrative example is:

# Apply example site-level filters

gatk VariantFiltration \

    -R reference.fa \

    -V cohort.vcf.gz \

    –filter-name “LowQual” \

    –filter-expression “QD < 2.0” \

    –filter-name “HighStrandBias” \

    –filter-expression “FS > 60.0” \

    –filter-name “ReadPositionBias” \

    –filter-expression “ReadPosRankSum < -8.0” \

    -O cohort.filtered.vcf.gz

  • The thresholds shown are illustrative and are not universal recommendations. Their suitability depends on the caller, data type, organism, and variant-calling workflow. In particular, GATK annotations may not be available in VCF files produced by other callers, and filtering expressions must match the annotations actually present in the input file.
  • Variant Quality Score Recalibration (VQSR) is a statistical filtering approach available in GATK. Rather than applying only fixed thresholds to individual annotations, VQSR models patterns across multiple quality annotations and uses known high-confidence variant resources to help distinguish reliable calls from likely artifacts. It commonly uses a machine-learning model based on Gaussian mixture modelling. The model evaluates combinations of annotations and assigns a variant quality score recalibration score (VQSLOD), which represents evidence that a variant belongs to the true-variant model rather than the error model. Variants are then filtered according to selected sensitivity levels and available training resources. VQSR generally requires sufficiently large datasets and appropriate truth, training, and known-sites resources. It may not be suitable for small cohorts, non-human organisms without suitable resources, or datasets with limited high-quality variant calls.
  • An illustrative VQSR workflow for SNPs is:

# Build the SNP recalibration model

gatk VariantRecalibrator \

    -R reference.fa \

    -V cohort.vcf.gz \

    –resource:hapmap,known=false,training=true,truth=true,prior=15.0 hapmap.vcf.gz \

    –resource:omni,known=false,training=true,truth=false,prior=12.0 omni.vcf.gz \

    –resource:1000G,known=false,training=true,truth=false,prior=10.0 thousand_genomes.vcf.gz \

    –resource:dbsnp,known=true,training=false,truth=false,prior=2.0 dbsnp.vcf.gz \

    -an QD \

    -an MQ \

    -an FS \

    -an SOR \

    -an ReadPosRankSum \

    -mode SNP \

    -O snps.recal \

    –tranches-file snps.tranches

# Apply the recalibration model

gatk ApplyVQSR \

    -R reference.fa \

    -V cohort.vcf.gz \

    –recal-file snps.recal \

    –tranches-file snps.tranches \

    –truth-sensitivity-filter-level 99.0 \

    -mode SNP \

    -O cohort.snps.recalibrated.vcf.gz

  • This example demonstrates the general structure of VQSR rather than a universally executable configuration. The selected resources, annotations, priors, model settings, and sensitivity thresholds must be appropriate for the reference assembly and dataset. SNPs and indels are commonly modelled separately. If the cohort is too small or lacks suitable training resources, carefully designed hard filtering may be more appropriate.
  • Variant filtering in somatic analysis requires additional considerations because mutations may be present in only a small fraction of tumour cells. Somatic callers such as GATK Mutect2, Strelka2, and VarDict use evidence from tumour samples and, when available, matched normal samples. Their filtering procedures consider sequencing artifacts, contamination, germline variation, allele fraction, and other factors. In Mutect2 workflows, FilterMutectCalls evaluates candidate variants using caller-specific annotations and statistical models. Tumour purity, local copy number, sequencing depth, and subclonal composition must be considered when interpreting low-VAF variants. A low VAF does not automatically mean that a call is false, and a high VAF does not prove that a variant is biologically important.
  • An illustrative somatic filtering command using GATK is:

# Filter Mutect2 candidate variants

gatk FilterMutectCalls \

    -R reference.fa \

    -V somatic_candidates.vcf.gz \

    -O somatic_filtered.vcf.gz

  • A complete somatic workflow may also include contamination estimation, orientation-bias modelling, and artifact-specific filtering. These steps depend on the tissue type, library preparation, sequencing technology, and analysis design. Paired tumour-normal data are particularly valuable because they help distinguish acquired mutations from inherited variants, although clonal haematopoiesis, mosaicism, and contamination can complicate the interpretation.
  • Long-read variant filtering also requires technology-specific quality assessment. PacBio HiFi reads generally have high consensus accuracy, while Oxford Nanopore read accuracy depends on the chemistry, basecalling model, and sequencing workflow. Long-read callers may use read support, alignment quality, strand or orientation evidence, breakpoint support, and other platform-specific metrics. For structural variants, filtering may consider the number of supporting reads, breakpoint consistency, read length, mapping quality, and support across independent molecules. Tools such as Sniffles2 and cuteSV provide filtering options for structural variant calls, but their parameters should be selected according to the sequencing technology and study objective.
  • Structural variant filtering differs from small-variant filtering because large genomic changes may be supported by complex alignment patterns. Deletions, insertions, inversions, duplications, and translocations can produce split reads, discordant read pairs, abnormal coverage, or long reads spanning variant breakpoints. Short-read structural variant callers such as Manta, DELLY, and GRIDSS use combinations of these signals, while long-read callers such as Sniffles2 and cuteSV use long-alignment evidence. Important quality considerations include the number of supporting reads, breakpoint confidence, event size, local repeat content, mapping ambiguity, and consistency between independent evidence types. A structural variant with a small number of supporting reads may still be genuine in a low-coverage dataset, so thresholds should be evaluated in the context of read length and sequencing depth.
  • Cohort-level filtering is necessary when many samples are analysed together. Important measures include variant missingness, genotype missingness, allele frequency, heterozygosity, Hardy–Weinberg equilibrium, and sample-level sequencing quality. Variants with high missing genotype rates may be excluded because their allele-frequency estimates and association results can be unreliable. However, missingness may also reflect real biological differences or population-specific variation, so its distribution should be examined before filtering. In case-control studies, differential missingness between groups can create false associations and should be evaluated explicitly.
  • Population allele frequency is often used to distinguish common and rare variants. The minor allele frequency (MAF) represents the frequency of the less common allele at a biallelic site in a specified population or dataset. Rare-variant studies may focus on variants below selected frequency thresholds, whereas common-variant studies often analyse more frequent alleles. Frequency thresholds should not be treated as universal indicators of pathogenicity. A rare variant is not necessarily harmful, and a common variant may still have biological relevance. Population ancestry and the representativeness of reference databases are important because allele frequencies differ among populations.
  • Hardy–Weinberg equilibrium testing is another quality-control method used in many population and association studies. Under the assumptions of Hardy–Weinberg equilibrium, expected genotype frequencies for a biallelic locus with allele frequencies (p) and (q) are (p^2), (2pq), and (q^2). Significant deviations may indicate genotyping errors, population structure, inbreeding, selection, non-random mating, or other biological factors. The test is commonly applied to control samples in case-control studies, but its interpretation requires care. Deviations in cases may be biologically meaningful, and population stratification can produce departures from equilibrium even when genotyping is accurate.
  • Tools such as PLINK and PLINK 2 are widely used for cohort-level genotype quality control. They support filtering by sample and variant missingness, allele frequency, heterozygosity, and other measures. An illustrative PLINK 2 workflow is:

# Import a VCF file and create PLINK 2 files

plink2 \

    –vcf cohort.filtered.vcf.gz \

    –make-pgen \

    –out cohort

# Generate sample and variant missingness reports

plink2 \

    –pfile cohort \

    –missing \

    –out cohort_missingness

# Calculate allele frequencies

plink2 \

    –pfile cohort \

    –freq \

    –out cohort_frequency

# Calculate heterozygosity statistics

plink2 \

    –pfile cohort \

    –het \

    –out cohort_heterozygosity

  • These commands generate reports for quality assessment; they do not automatically determine which samples or variants should be excluded. Researchers should examine the distributions, identify outliers, consider population composition, and choose filtering thresholds appropriate to the study.
  • Variant normalization is also important before comparing or filtering variant datasets. Equivalent indels can be represented differently in VCF, particularly in repetitive sequences. Left alignment and normalization against the same reference genome help ensure consistent representation. BCFtools norm can perform common normalization operations:

# Normalize variants against the reference genome

bcftools norm \

    -f reference.fa \

    -m -any \

    cohort.filtered.vcf.gz \

    -Oz \

    -o cohort.normalized.vcf.gz

# Index the normalized VCF

bcftools index cohort.normalized.vcf.gz

  • Normalization improves consistency but does not guarantee that all callers or databases will represent every complex variant identically. Reference assembly, chromosome naming, multiallelic representation, and variant decomposition should be considered when comparing datasets.
  • The effectiveness of variant filtering should be evaluated using suitable benchmarking resources. Precision measures the proportion of called variants that are true positives, while recall measures the proportion of true variants that are detected. The F1 score combines precision and recall. Reference datasets from the Genome in a Bottle (GIAB) project are widely used for benchmarking human germline SNV and small-indel calling in well-characterised genomic regions. Tools such as hap.py can compare call sets with truth data. For structural variants, benchmarking tools such as Truvari can compare variant calls while accounting for differences in representation and breakpoint positions. Somatic variant benchmarking is more challenging because tumour samples contain subclonal mutations and variable tumour purity. Reference materials, synthetic mixtures, and orthogonal validation can help assess performance at different allele fractions.
  • Orthogonal validation may be necessary for variants with important clinical, biological, or research implications. Sanger sequencing is commonly used to confirm selected SNVs and small indels, although it may be less sensitive for low-frequency mosaic or somatic variants. Targeted deep sequencing, digital PCR, quantitative PCR, long-read sequencing, and other specialised assays may be more suitable depending on the variant class and allele fraction. Structural variants may require breakpoint-specific PCR, fluorescence in situ hybridization (FISH), or other methods. Validation should be selected according to the expected size, genomic context, and abundance of the variant.
  • Variant filtering has several limitations. Strict thresholds can remove genuine variants, especially rare variants, low-frequency somatic mutations, and variants in difficult genomic regions. Permissive thresholds can retain false positives caused by sequencing artifacts or alignment errors. Different sequencing technologies and variant callers produce different quality metrics, so filtering criteria cannot always be transferred directly between workflows. A high-quality score does not guarantee biological significance, and a low-quality score does not always mean that a variant is false. Quality scores describe evidence and uncertainty under particular computational assumptions; biological interpretation requires additional evidence.
  • Reproducibility is essential in variant quality assessment. Researchers should document the reference genome assembly, sequencing platform, sample type, read-processing procedures, aligner, variant caller, software versions, filtering annotations, thresholds, and normalization steps. The original unfiltered VCF should be retained so that filtering decisions can be reviewed or revised. Quality-control reports should record the number of variants before and after each filtering stage, the reasons for excluding calls, and any limitations in the dataset. Workflow systems such as Nextflow, Snakemake, and Cromwell can support consistent execution, while Conda environments and containers help preserve software dependencies. Human genomic data should also be managed according to appropriate privacy, consent, and access requirements.
  • Variant filtering and quality scores are fundamental to producing reliable genomic datasets. Quality measures such as Phred scores, read depth, mapping quality, genotype quality, allele balance, and strand-bias statistics provide complementary information about the evidence supporting each call. Hard filtering, statistical recalibration, caller-specific models, and cohort-level quality control help remove unreliable results while preserving genuine genetic variation. The most appropriate strategy depends on the sequencing technology, variant class, sample type, population, and research objective. Careful filtering, benchmarking, validation, and documentation are therefore essential for accurate and reproducible variant analysis.
Author:

Leave a Reply

Your email address will not be published. Required fields are marked *