- Mixed models and relatedness correction are important statistical approaches in bioinformatics, particularly in genome-wide association studies (GWAS), population genetics, genomic prediction, and other studies involving genetically related individuals. Biological datasets frequently contain samples that are not statistically independent. Individuals may share genetic material because of family relationships, common ancestry, population structure, or membership in the same breeding population. If these relationships are ignored, statistical analyses may produce inflated association signals, biased effect estimates, and excessive false-positive findings. Mixed models provide a framework for accounting for these dependencies and improving the reliability of genetic association and prediction analyses.
- A conventional linear or logistic regression model generally assumes that observations are independent after accounting for the explanatory variables included in the model. However, this assumption is often violated in genetic studies. For example, siblings share approximately half of their segregating genetic variation on average, while more distant relatives share smaller proportions. Individuals from the same ancestral population may also share genetic variants because of historical ancestry, genetic drift, and population-specific allele frequencies. These similarities can produce correlations between their phenotypes even when the genetic marker being investigated has no direct effect on the outcome. This phenomenon is particularly important in large-scale genetic association studies, where millions of variants may be tested simultaneously.
- A mixed model, also known as a mixed-effects model, contains both fixed effects and random effects. Fixed effects represent factors whose specific effects are of direct interest or are included to adjust for known influences. Examples include genotype, age, sex, treatment, environmental exposure, and experimental batch. Random effects represent sources of variation associated with groups, individuals, families, populations, or other sampling structures. They account for correlations among observations that share a common source of variation. In genetic association studies, a random genetic effect can be used to model the similarity between individuals based on their genome-wide genetic relationships.
- A general linear mixed model can be expressed as:
[
\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\mathbf{Z}\mathbf{u}+\boldsymbol{\varepsilon}
]
- Here, (\mathbf{y}) is the vector of observed phenotypes, (\mathbf{X}) is the design matrix for fixed effects, and (\boldsymbol{\beta}) is the vector of fixed-effect coefficients. The matrix (\mathbf{Z}) connects observations to random effects, while (\mathbf{u}) represents the random effects. The residual error is represented by (\boldsymbol{\varepsilon}). A common assumption is that the random effects and residual errors follow mean-zero multivariate normal distributions with specified covariance structures.
- In a simple genomic mixed model, the random genetic effect is assumed to have a covariance structure proportional to a genomic relationship matrix. The model can be written as:
[
\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\mathbf{g}+\boldsymbol{\varepsilon}
]
- where:
[
\mathbf{g}\sim N(0,\mathbf{K}\sigma_g^2)
]
- and:
[
\boldsymbol{\varepsilon}\sim N(0,\mathbf{I}\sigma_e^2)
]
- Here, (\mathbf{K}) is the genomic relationship matrix, (\sigma_g^2) is the variance associated with the modelled genetic effect, (\mathbf{I}) is an identity matrix, and (\sigma_e^2) is the residual variance. The covariance between the genetic effects of two individuals is therefore determined by their estimated genomic relationship. Individuals with greater genetic similarity are modelled as having more strongly correlated random genetic effects.
- The genomic relationship matrix, also called the genomic kinship matrix or genomic relationship matrix (GRM), is central to many relatedness-correction methods. It summarizes genetic similarity between pairs of individuals using genome-wide marker information. Unlike a pedigree relationship matrix, which is calculated from recorded ancestry, a genomic relationship matrix is estimated from observed genetic markers. It can therefore capture realized genetic similarity, including differences among individuals that have the same expected pedigree relationship. For example, two full siblings have an expected relatedness of approximately 0.5, but their realized genetic similarity may differ because of random inheritance and recombination.
- A commonly used approach to construct a genomic relationship matrix is based on standardized genotype values. Suppose (M_{ij}) represents the genotype dosage of individual (i) at marker (j), coded as 0, 1, or 2 copies of a reference allele. Let (p_j) represent the estimated allele frequency of that allele at marker (j). A standardized genotype can be calculated as:
[
Z_{ij}=\frac{M_{ij}-2p_j}{\sqrt{2p_j(1-p_j)}}
]
- The genomic relationship matrix can then be estimated using a marker-based relationship calculation, such as:
[
\mathbf{K}=\frac{\mathbf{Z}\mathbf{Z}^{T}}{m}
]
- where (\mathbf{Z}) is the standardized genotype matrix and (m) is the number of markers included in the calculation. This is a simplified representation of one common approach. Alternative estimators, allele-frequency adjustments, missing-genotype procedures, marker filtering, and scaling conventions can produce different relationship matrices. The exact method should therefore be documented when reporting genomic analyses.
- Relatedness correction is necessary because genetic similarity can create correlations between phenotypes. Consider a GWAS involving individuals from several families. If a genetic variant is more common in one family and that family also has a higher average value for a particular trait because of shared genetic background or environment, a simple regression may incorrectly attribute the difference to the tested variant. A mixed model can account for genome-wide relatedness through the GRM, reducing the risk that family structure or broader genetic similarity produces spurious associations.
- Population stratification is another important source of confounding. It occurs when a study includes individuals from populations with different ancestry backgrounds and allele frequencies. If phenotype distributions also differ between these populations, genetic markers may appear associated with the phenotype because they distinguish ancestry groups rather than because they have a direct relationship with the biological trait. Principal component analysis (PCA) is commonly used to summarize major axes of genetic variation, and these principal components can be included as fixed-effect covariates. Mixed models using a genomic relationship matrix provide an additional way to account for genetic similarity among individuals. In many studies, both approaches are used together because they address related but not identical aspects of population structure.
- A widely used linear mixed model for genome-wide association analysis is:
[
\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\mathbf{x}_j\alpha_j+\mathbf{u}+\boldsymbol{\varepsilon}
]
- where (\mathbf{x}_j) represents the genotype values for the genetic marker currently being tested, (\alpha_j) is its estimated effect, and (\mathbf{u}) is the random polygenic effect whose covariance is modelled using the genomic relationship matrix. The model tests whether the marker contributes to variation in the phenotype after accounting for genome-wide genetic similarity and other fixed effects. This approach is commonly referred to as a genome-wide association mixed model or linear mixed model GWAS.
- Mixed models can also be applied to binary and other non-continuous outcomes. For binary disease status, a generalized linear mixed model (GLMM) can use a logistic link function while incorporating random effects to account for relatedness or other grouping structures. The model estimates the probability of the outcome while accounting for correlations among related individuals. Generalized mixed models can also be developed for count, ordinal, and other response types, although the choice of distribution and link function must match the data and research question.
- Several statistical methods are used to estimate the variance components of mixed models. Restricted maximum likelihood (REML) is widely used for estimating variance components because it accounts for the loss of degrees of freedom associated with estimating fixed effects. Maximum likelihood (ML) is another estimation approach and is useful for model comparison in appropriate settings. The variance components describe how much variation is attributed to different random sources, such as genome-wide genetic effects, family effects, environmental groups, and residual variation. Accurate variance-component estimation is important because the estimated covariance structure influences the correction for relatedness and the precision of association estimates.
- The heritability of a quantitative trait can also be estimated using a genomic mixed model. In a simplified model with one additive genomic random effect, SNP-based heritability may be expressed as:
[
h^2_{\text{SNP}}=\frac{\sigma_g^2}{\sigma_g^2+\sigma_e^2}
]
- Here, (\sigma_g^2) represents the variance captured by the genetic markers included in the model, and (\sigma_e^2) represents residual variance. This estimate is generally specific to the marker set, population, phenotype definition, and modelling assumptions. It should not automatically be interpreted as the total heritability attributable to all genetic variation, because variants not captured by the markers may not contribute to the estimated genomic component.
- Mixed models are especially important in animal and plant breeding, where populations frequently contain related individuals and selection decisions are made across generations. Pedigree-based best linear unbiased prediction (BLUP) uses a relationship matrix derived from recorded ancestry to estimate breeding values. Genomic BLUP (GBLUP) replaces or supplements the pedigree relationship matrix with a genomic relationship matrix derived from molecular markers. This allows genomic information to improve estimates of genetic merit and can capture realized relationships more accurately than pedigree expectations alone. Genomic prediction is used to estimate the genetic merit of selection candidates, including young animals or plants that have not yet accumulated extensive performance records.
- The distinction between pedigree and genomic relatedness is important. A pedigree relationship matrix estimates expected genetic relationships based on recorded parentage. It is useful when complete and accurate pedigree information is available, but it may not capture the exact proportion of genetic material shared by particular individuals. A genomic relationship matrix estimates similarity from observed markers and can detect realized differences in relatedness, hidden relationships, and some forms of pedigree error. Combining pedigree and genomic information can be beneficial in populations with incomplete genotyping or incomplete ancestry records.
- Several software tools are widely used for mixed-model analysis and relatedness correction. GEMMA is designed for linear mixed models and genome-wide association analysis. EMMAX implements an efficient mixed-model association approach. GCTA provides tools for estimating genetic relationships, heritability, and genetic associations. BOLT-LMM uses linear mixed-model methods designed for large-scale genetic association studies. REGENIE provides a scalable two-step framework for association analysis, while SAIGE uses a generalized mixed-model approach designed to handle large studies and imbalanced case-control phenotypes. PLINK 2 provides functions for genotype processing, relationship estimation, and association analysis, although its available mixed-model functionality and options should be checked against the version being used. The appropriate software depends on sample size, phenotype type, genetic data, computational resources, and study design.
- The following Python example demonstrates how a genomic relationship matrix can be calculated from a simplified genotype matrix. The values are illustrative and represent genotype dosages for several individuals across a small number of markers. A real analysis requires quality-controlled genotype data, appropriate allele-frequency estimates, missing-data handling, and a carefully selected relationship estimator.
import numpy as np
# Rows represent individuals; columns represent genetic markers.
# Genotypes are coded as 0, 1, or 2.
genotypes = np.array([
[0, 1, 2, 0, 1],
[0, 1, 1, 0, 1],
[2, 0, 0, 1, 0],
[1, 1, 2, 1, 2]
], dtype=float)
# Estimate allele frequencies from genotype dosages.
allele_frequencies = np.mean(genotypes, axis=0) / 2
# Standardize genotype values.
denominator = np.sqrt(
2 * allele_frequencies * (1 – allele_frequencies)
)
# Avoid division by zero for monomorphic markers.
valid_markers = denominator > 0
standardized = (
genotypes[:, valid_markers]
– 2 * allele_frequencies[valid_markers]
) / denominator[valid_markers]
# Calculate a simple genomic relationship matrix.
num_markers = standardized.shape[1]
grm = standardized @ standardized.T / num_markers
print(“Genomic relationship matrix:”)
print(np.round(grm, 3))
- This example illustrates the basic calculation but is not a complete production-ready genomic analysis pipeline. In practical datasets, genotype quality control, marker selection, allele-frequency estimation, missing genotype treatment, population composition, and the chosen GRM scaling method must all be considered. Some estimators also use different denominators or allele-frequency corrections.
- The following R example illustrates the general structure of a linear mixed model using the lme4 package. It models a continuous phenotype while accounting for a grouping effect, such as family or population. This example demonstrates a conventional grouping-based random effect; it does not directly use a genomic relationship matrix.
library(lme4)
# Illustrative phenotype dataset
data <- data.frame(
phenotype = c(10.2, 11.1, 9.8, 12.3, 11.7, 10.9),
genotype = c(0, 1, 0, 2, 1, 2),
age = c(25, 30, 28, 35, 32, 40),
family = factor(c(“F1”, “F1”, “F2”, “F2”, “F3”, “F3”))
)
# Fit a linear mixed model with family as a random effect
model <- lmer(
phenotype ~ genotype + age + (1 | family),
data = data
)
# Display model results
summary(model)
- For genome-wide association studies, specialized software is generally required to incorporate a large genomic relationship matrix efficiently. The following is an illustrative GEMMA command showing the general stages of calculating a relatedness matrix and fitting a linear mixed model. Exact options and input requirements should be checked against the installed GEMMA version and its documentation.
# Calculate a centered relatedness matrix
gemma -bfile genotype_data -gk 1 -o relatedness
# Run a linear mixed model association analysis
gemma -bfile genotype_data \
-k output/relatedness.cXX.txt \
-lmm 4 \
-pheno phenotype.txt \
-o mixed_model_gwas
- The performance of mixed models depends on the quality of the genetic relationship matrix and the assumptions used to construct it. Poor-quality genotypes, population-specific allele-frequency differences, relatedness estimation errors, and unaccounted batch effects can influence the covariance structure. Including too many highly correlated markers or failing to account for differences in allele frequency may also affect the estimated relationships. Researchers should examine the distribution of relatedness estimates, check sample identities and population structure, and compare results under appropriate alternative modelling strategies.
- There are also computational challenges associated with mixed models. A genomic relationship matrix for (n) individuals contains (n \times n) entries, so its storage and computation requirements increase rapidly as sample size grows. Traditional mixed-model methods may become computationally expensive for studies involving hundreds of thousands or millions of individuals. Efficient algorithms, low-rank approximations, sparse methods, and specialized software have therefore been developed to make large-scale analyses practical. Methods such as BOLT-LMM and REGENIE use computational strategies designed for large genetic datasets, while SAIGE addresses challenges associated with binary traits and imbalanced case-control studies.
- Relatedness correction does not eliminate every source of confounding. A genomic relationship matrix captures genetic similarity represented by the markers used to construct it, but it may not fully account for environmental similarities, household effects, geographic structure, recruitment bias, or technical artifacts. In some studies, additional fixed or random effects are needed to represent these influences. Researchers should also avoid indiscriminately including covariates that may be consequences of the exposure or lie on the causal pathway, because inappropriate adjustment can introduce bias.
- Mixed models should be selected according to the study design and biological question. A simple linear regression may be adequate for a well-designed study involving unrelated individuals from a relatively homogeneous population. A mixed model is more appropriate when substantial relatedness, population structure, repeated measurements, family clustering, or other dependencies are present. For quantitative traits, linear mixed models are commonly used; for binary traits, generalized linear mixed models or specialized association methods may be required. The choice should consider phenotype distribution, sample size, case-control balance, population composition, and computational feasibility.
- In reproducible bioinformatics research, investigators should report the relationship matrix construction method, genetic markers used, allele-frequency estimation procedure, genotype quality-control steps, fixed and random effects, variance-component estimation method, software name and version, and relevant model settings. They should also describe how population structure and relatedness were assessed and corrected. Where possible, sensitivity analyses can compare results from models with and without relatedness correction, different covariate sets, or alternative relationship estimators. Such comparisons help identify whether major findings are robust to modelling decisions.
- Mixed models and relatedness correction are essential components of modern statistical genetics. They allow researchers to analyse genetically structured populations while accounting for correlations among individuals and improving the reliability of association estimates. Their applications extend beyond GWAS to genomic prediction, heritability estimation, animal and plant breeding, family-based studies, and longitudinal biological research. By combining fixed effects, random effects, and appropriate covariance structures, mixed models provide a flexible framework for analysing complex biological data and reducing false-positive findings caused by relatedness and population structure.