Genetic Differentiation and FST in Population Genetics

  • Genetic differentiation is the process by which populations become genetically distinct through differences in allele frequencies, genotype distributions, and patterns of genetic variation. It is a central concept in population genetics, evolutionary biology, conservation genetics, and genomic research. Populations may become genetically differentiated because of restricted gene flow, genetic drift, mutation, natural selection, population isolation, founder effects, and differences in demographic history. Measuring genetic differentiation helps researchers understand how populations evolve, how they are related, and how genetic variation is distributed within and among populations.
  • Populations of the same species often share many genetic variants, but the frequencies of these variants may differ. For example, an allele may occur at a high frequency in one population and at a lower frequency in another. Such differences can arise when populations become geographically separated or experience different environmental conditions. Over generations, genetic drift and natural selection may cause their allele frequencies to diverge. Migration and interbreeding can reduce these differences by introducing genetic material between populations.
  • Genetic differentiation can occur at different levels of biological organization. It may involve differences between individuals within a population, between subpopulations within a region, or between geographically separated populations. At the genomic level, differentiation may affect individual genes, specific chromosomal regions, or the genome as a whole. Some genomic regions may show strong differentiation because of local adaptation or selection, while other regions remain relatively similar due to ongoing gene flow or shared ancestry.
  • One of the most widely used statistical measures of genetic differentiation is the fixation index, commonly represented by FST. It was developed within the framework of population genetics to describe the proportion of genetic variation associated with differences among populations relative to the total genetic variation. FST is part of the broader family of F-statistics, which includes measures of inbreeding and population structure. In its common interpretation, FST measures the extent to which genetic variation is partitioned among populations rather than within populations.
  • The basic interpretation of FST is that values near zero indicate little genetic differentiation, while higher values indicate greater differences in allele frequencies among populations. An FST value of zero is consistent with no differentiation at the measured loci under the chosen model, whereas a value approaching one indicates very strong differentiation. However, FST should not be interpreted as a direct measure of evolutionary distance, the percentage of different genes between populations, or the proportion of individuals that are migrants. Its meaning depends on the genetic data, population sampling, allele frequencies, and statistical method used.
  • A commonly used expression for FST is based on the relationship between total genetic diversity and average genetic diversity within populations: FST = (HT − HS) / HT
  • Here, HT represents the expected genetic diversity in the total population when all sampled populations are considered together, while HS represents the average expected genetic diversity within the individual populations. For a biallelic locus, expected heterozygosity in a population with allele frequency p and alternative allele frequency q is: H = 2pq
  • When allele frequencies differ among populations, the total expected heterozygosity can be greater than the average within-population heterozygosity. The difference between these quantities provides information about population differentiation. In practice, FST is estimated using statistical methods appropriate to the sampling design and genetic markers.
  • For example, consider two populations with different frequencies of a biallelic genetic variant. In Population 1, the frequency of allele A is 0.8, while in Population 2 it is 0.2. The expected heterozygosity within each population is:

Population 1:

H1 = 2 × 0.8 × 0.2 = 0.32

Population 2:

H2 = 2 × 0.2 × 0.8 = 0.32

The average within-population heterozygosity is therefore 0.32. If the two populations are equally represented, the pooled allele frequency of A is 0.5, giving total expected heterozygosity of 0.50. The corresponding FST is:

FST = (0.50 − 0.32) / 0.50

FST = 0.36

  • This simplified example illustrates how substantial differences in allele frequencies can produce genetic differentiation even when the two populations have the same heterozygosity within each population. Real datasets contain many loci and may require estimators that account for sample size, missing data, unequal population sizes, and marker characteristics.
  • FST is closely related to the broader framework of Wright’s F-statistics, which describes genetic variation at different levels of population organization. The three commonly discussed statistics are FIS, FST, and FIT. FIS measures the departure from expected heterozygosity within subpopulations, often interpreted in relation to inbreeding or non-random mating. FST measures differentiation among subpopulations relative to the total population. FIT measures the overall departure from expected heterozygosity for individuals relative to the total population.
  • Their relationship is commonly expressed as: (1 − FIT) = (1 − FIS)(1 − FST)
  • These statistics help distinguish genetic patterns within individuals, within populations, and among populations. For example, a positive FIS may indicate a heterozygote deficit within subpopulations, potentially associated with inbreeding or non-random mating. A positive FST indicates differentiation among populations. However, these statistics do not independently identify the biological cause of the observed pattern.
  • Several evolutionary and demographic processes influence genetic differentiation. Genetic drift is one of the most important. It refers to random changes in allele frequencies from one generation to another. Drift is especially influential in populations with small effective population sizes, where random sampling of alleles can produce substantial changes over time. When two populations become isolated, their allele frequencies may change independently, leading to increasing genetic differentiation even in the absence of natural selection.
  • Gene flow, or migration followed by reproduction, generally reduces genetic differentiation by transferring alleles between populations. When individuals migrate and reproduce in another population, they introduce genetic variants that may alter local allele frequencies. The strength of this effect depends on migration rates, reproductive success, population size, and the genetic structure of the populations. Limited gene flow can allow populations to diverge, while extensive gene flow may maintain genetic similarity.
  • Natural selection can also contribute to genetic differentiation. Different environmental conditions may favor different alleles in different populations. For example, populations living at different temperatures, altitudes, disease pressures, or food-resource conditions may experience different selective pressures. As beneficial alleles increase in frequency, the affected genomic regions may become more differentiated. This process is known as local adaptation. Genome-wide differentiation may reflect demographic history, while unusually high differentiation at particular loci may suggest possible selection. Such signals require further investigation because demographic processes and technical factors can also produce unusual patterns.
  • Mutation introduces new genetic variants and contributes to long-term genetic divergence. Population bottlenecks and founder effects can also produce substantial differentiation. A bottleneck occurs when population size decreases sharply, reducing genetic diversity and changing allele frequencies through drift. A founder effect occurs when a new population is established by a small number of individuals carrying only a subset of the genetic variation present in the source population. Over generations, these processes can produce allele-frequency differences between the new population and its ancestors.
  • Genetic differentiation is also influenced by geographic distance and barriers to migration. Physical barriers such as oceans, mountains, deserts, and fragmented habitats may restrict gene flow. In some species, geographic distance alone can reduce the probability of mating between individuals from distant locations. This may produce isolation by distance, in which genetic differentiation tends to increase with geographic separation. However, geographic patterns are influenced by species biology, dispersal behavior, historical events, and environmental conditions.
  • FST can be estimated from different types of genetic markers, including single-nucleotide polymorphisms, microsatellites, insertion–deletion variants, and other polymorphic loci. SNPs are especially widely used in modern population genomics because they can be measured across large numbers of genomic positions. Microsatellites have also been important in population and conservation genetics because of their high polymorphism, although their mutation patterns and allele scoring can complicate comparisons. The marker type, mutation rate, allele-frequency distribution, and genomic coverage can all influence estimates of differentiation.
  • Several statistical estimators of FST have been developed. The estimator introduced by Weir and Cockerham is widely used because it accounts for important aspects of sampling and hierarchical population structure. Other approaches include estimators based on heterozygosity, allele frequencies, and molecular distances. Hudson’s estimator is also commonly used in population-genomic analyses. Different estimators may produce different values, especially when sample sizes are small, populations are unequal, or genetic markers have unusual allele-frequency distributions. Researchers should therefore report the estimator and software used rather than treating all FST values as directly interchangeable.
  • FST values are sometimes described using broad interpretive categories. For example, values below approximately 0.05 are often described as indicating weak differentiation, values around 0.05–0.15 as moderate differentiation, values around 0.15–0.25 as substantial differentiation, and values above 0.25 as strong differentiation. These categories are only rough conventions, not universal biological thresholds. The significance and biological importance of a particular value depend on the species, marker system, population history, sampling strategy, and research question. A small FST value may still be meaningful when estimated across many loci or when populations are separated by strong ecological barriers.
  • A key distinction is that FST is not the same as individual genetic distance. FST describes population-level differentiation based on the distribution of genetic variation. It does not directly measure how genetically similar two particular individuals are. Individual relatedness, ancestry, and genetic distance require other methods, such as genomic relationship matrices, identity-by-descent analysis, principal component analysis, or model-based ancestry estimation.
  • Genetic differentiation is widely studied using principal component analysis (PCA), clustering methods, phylogenetic approaches, and population-genetic statistics. PCA summarizes genetic variation across individuals and can reveal major axes of population structure. Model-based methods such as ADMIXTURE and STRUCTURE estimate ancestry proportions under specified assumptions. Genetic distance matrices and phylogenetic trees can summarize relationships among populations, although the resulting relationships should not automatically be interpreted as exact population histories. FST provides a complementary quantitative measure of differentiation based on allele-frequency variation.
  • Pairwise FST analysis is commonly used to compare every pair of populations in a study. The resulting matrix shows which populations are more differentiated from one another. These values can be visualized using heat maps, clustering diagrams, or multidimensional scaling. Pairwise comparisons are useful for examining geographic structure, identifying isolated populations, and evaluating the effects of conservation or breeding management. However, pairwise values are not necessarily independent because comparisons may share populations and samples.
  • Genetic differentiation is particularly important in conservation genetics. Small and fragmented populations may lose genetic diversity through drift and inbreeding, while reduced gene flow can increase differentiation among population fragments. FST and related measures help conservation researchers evaluate population connectivity, identify genetically distinct management units, and monitor changes in genetic structure. These results can guide decisions about habitat corridors, translocations, captive breeding, and the maintenance of genetic diversity. Nevertheless, management decisions should also consider ecological, behavioral, demographic, and disease-related information.
  • In animal and plant breeding, genetic differentiation can help describe differences among breeds, strains, lines, and varieties. Livestock breeds may have distinct allele frequencies because of geographic isolation, artificial selection, founder effects, and historical breeding practices. FST analysis can identify genetically differentiated breeds and genomic regions that may have experienced selection. In crop research, it can help characterize diversity among landraces, breeding populations, and wild relatives. This information supports germplasm conservation, parent selection, introgression, and the design of breeding populations.
  • In human population genetics, FST is used to investigate patterns of genetic variation among geographically defined or sampled populations. It can help study migration, demographic history, genetic drift, and population structure. However, human genetic variation is generally distributed in overlapping and often continuous patterns rather than sharply separated biological divisions. Population labels are influenced by sampling design and social or geographic categories, and they should not be treated as fixed genetic boundaries. Genetic differentiation statistics should therefore be interpreted carefully and should not be used to support simplistic assumptions about human groups.
  • FST also has applications in genome scans for natural selection. Researchers may calculate differentiation at thousands or millions of genomic markers and identify regions with unusually high or low values relative to an appropriate background. High differentiation may indicate local adaptation or divergent selection, while low differentiation may reflect shared ancestry, gene flow, or balancing selection. However, high-FST regions are not automatically selected regions. Population bottlenecks, background selection, recombination-rate variation, and sampling differences can generate similar patterns. Selection scans should therefore use demographic models, complementary statistics, and independent functional evidence.
  • Several bioinformatics tools support genetic differentiation analysis. PLINK and PLINK 2 provide functions for genotype quality control, population structure analysis, and relatedness assessment. VCFtools and bcftools support variant filtering and population-genetic workflows. R packages such as hierfstat, adegenet, and SNPRelate provide methods for estimating population differentiation, calculating genetic distances, and visualizing genetic structure. scikit-allel is a Python library for population-genetic analysis of genomic variation. ANGSD supports population-genomic analyses that can account for genotype uncertainty, particularly when working with sequencing data of variable coverage.
  • The following Python example calculates a simplified heterozygosity-based FST estimate for two equally weighted populations at one biallelic locus:

# Allele A frequencies in two populations

p1 = 0.8

p2 = 0.2

# Alternative allele frequencies

q1 = 1 – p1

q2 = 1 – p2

# Expected heterozygosity within each population

H1 = 2 * p1 * q1

H2 = 2 * p2 * q2

# Average within-population heterozygosity

Hs = (H1 + H2) / 2

# Pooled allele frequency, assuming equal population weights

p_total = (p1 + p2) / 2

q_total = 1 – p_total

# Expected heterozygosity in the pooled population

Ht = 2 * p_total * q_total

# Simplified FST calculation

FST = (Ht – Hs) / Ht if Ht > 0 else 0

print(“Within-population diversity (Hs):”, Hs)

print(“Total diversity (Ht):”, Ht)

print(“FST:”, FST)

  • This example illustrates the relationship between within-population and total expected heterozygosity. It is not a substitute for a multilocus estimator designed for real population-genomic datasets. In practical analyses, researchers should account for sample sizes, population weights, missing genotypes, relatedness, and the properties of the chosen estimator.
  • An R example using the hierfstat package demonstrates how F-statistics can be estimated from a genetic dataset:

# Install the package if required

install.packages(“hierfstat”)

library(hierfstat)

# Example data structure:

# First column = population identifier

# Remaining columns = genotype data

#

# The genotype data must be encoded in a format

# accepted by hierfstat.

# Replace population_data with your actual dataset

# containing population and genotype information.

# Estimate basic F-statistics

basic_stats <- basic.stats(population_data)

# Display F-statistics

print(basic_stats$overall)

# Estimate pairwise population differentiation

pairwise_fst <- pairwise.WCfst(population_data)

print(pairwise_fst)

  • The dataset must be prepared using the genotype coding and data structure required by hierfstat. The exact function options and input requirements should be checked against the installed package documentation. For large SNP datasets, PLINK, PLINK 2, VCFtools, or specialized population-genomic tools may be more suitable.
  • A simplified PLINK 2 command for estimating population differentiation from a binary genotype dataset is:

plink2 \

  –bfile population_data \

  –fst \

  –out fst_results

  • The available FST options and output details depend on the PLINK 2 version and dataset configuration. Researchers should consult the current documentation and specify the appropriate population grouping information for their analysis. For pairwise comparisons or specialized estimators, other tools may be more suitable.
  • Several limitations must be considered when interpreting FST. Estimates can be affected by sample size, allele frequencies, marker ascertainment, missing data, population subdivision, and the number and distribution of sampled loci. Rare variants can behave differently from common variants, and comparisons based on different marker panels may not be directly comparable. Strongly uneven sample sizes can also affect some estimators. In addition, FST may be relatively low even when populations have experienced substantial divergence if within-population genetic diversity is high.
  • Another important limitation is that FST does not provide a direct estimate of migration rate without additional assumptions. Under a highly simplified island model with neutral loci, equal-sized populations, migration–drift equilibrium, and other restrictive assumptions, FST has sometimes been related to effective migration through an expression such as: FST ≈ 1 / (4Nem + 1)
  • Here, Ne represents effective population size and m represents the migration rate per generation. This relationship is model-dependent and should not be used as a universal conversion from FST to migration rate. Real populations often violate the assumptions of the island model, and demographic history, unequal population sizes, selection, and population structure can produce different relationships between differentiation and migration.
  • For reliable and reproducible analysis, researchers should document the population definitions, sampling locations, sample sizes, marker types, genome assembly, variant filters, allele-frequency criteria, FST estimator, and software versions. They should also explain whether estimates were calculated across the whole genome, within selected regions, or between particular population pairs. When investigating selection, demographic history, or conservation status, FST should be interpreted alongside complementary analyses such as nucleotide diversity, heterozygosity, PCA, admixture analysis, linkage disequilibrium, and demographic modeling.
  • Genetic differentiation and FST are essential concepts for understanding how genetic variation is distributed within and among populations. Genetic differentiation reflects the evolutionary and demographic processes that cause populations to become genetically distinct, while FST provides a quantitative measure of this differentiation based on genetic diversity. Their applications extend across evolutionary biology, human population genetics, conservation, agriculture, animal breeding, and genome-wide studies of natural selection. Understanding the assumptions, estimators, interpretation, and limitations of FST is essential for drawing reliable conclusions from population-genomic data.
Author:

Leave a Reply

Your email address will not be published. Required fields are marked *