- Duplicate reads and library complexity are important concepts in the quality assessment and analysis of high-throughput sequencing data. During sequencing, multiple reads may originate from the same original DNA or RNA molecule. These reads are commonly referred to as duplicate reads. Their presence can result from PCR amplification, optical duplication, or the natural occurrence of identical DNA fragments. Library complexity describes the diversity of distinct original molecules represented in a sequencing library. A highly complex library contains many different molecular fragments, whereas a low-complexity library contains relatively few distinct fragments, often with repeated representation of the same molecules. Understanding duplicate reads and library complexity is essential for interpreting sequencing depth, detecting genetic variants, estimating gene expression, assessing experimental quality, and avoiding misleading biological conclusions.
- Sequencing libraries are prepared by converting DNA or RNA molecules into fragments suitable for sequencing. Depending on the protocol, these fragments may undergo end repair, adapter ligation, amplification, size selection, and other processing steps. During PCR amplification, a single original molecule can generate many copies. If these copies are sequenced independently, they may produce multiple reads with identical or nearly identical sequences. These reads can represent PCR duplicates rather than independent observations of different original molecules. Consequently, a dataset containing a large number of reads does not necessarily contain an equally large number of unique biological molecules.
- PCR duplicates are generated when the same original library fragment is amplified multiple times during polymerase chain reaction. The number of duplicate reads depends on factors such as the amount of starting material, the number of amplification cycles, PCR efficiency, library preparation conditions, and sequencing depth. Libraries prepared from limited quantities of DNA or RNA may require additional amplification and can therefore be more susceptible to duplication. Excessive PCR amplification can reduce the effective molecular diversity of a library, even when the final sequencing dataset contains a large number of reads.
- Optical duplicates are a different type of duplication that can arise during imaging-based sequencing. In some sequencing systems, the same physical cluster or signal may be detected more than once because of imaging, cluster identification, or processing issues. These duplicates do not necessarily result from PCR amplification. Their identification may depend on information such as sequencing lane, tile, and cluster coordinates. Optical duplicate detection is particularly associated with certain short-read sequencing workflows and is not equivalent to identifying duplicate molecules solely from their alignment positions.
- Duplicate reads can also occur naturally. Different original DNA molecules may have identical sequences or align to the same genomic coordinates, especially in regions with limited sequence diversity, highly expressed genes, repetitive regions, or targeted sequencing panels. In RNA sequencing, highly expressed transcripts can generate many independent fragments with identical or similar alignment coordinates. In amplicon sequencing, different molecules derived from the same target region may also produce identical reads. Therefore, identical sequences or alignment positions do not always prove that reads are PCR duplicates.
- Duplicate identification is commonly performed after reads have been aligned to a reference genome. Reads are compared using their mapping positions, orientation, and, where available, information about fragment boundaries. For paired-end sequencing, both reads in a pair and the inferred fragment coordinates can help identify duplicate fragments. Reads with matching alignment characteristics may be grouped into duplicate sets, and one representative read may be retained or the group may be marked for downstream handling. The exact criteria depend on the software and the sequencing application.
- In paired-end sequencing, duplicate identification should consider both mates rather than treating each read independently. Two pairs are more likely to represent the same original fragment when their alignment positions and orientations match. Insert size and fragment boundaries can provide additional information. For single-end sequencing, duplicate detection is generally based on the alignment start position, strand, and other available alignment information. However, coordinate-based methods cannot always distinguish PCR duplicates from independent molecules that happen to share the same endpoints.
- Duplicate reads are usually identified using specialized tools such as Picard MarkDuplicates, SAMtools markdup, and other alignment-processing programs. These tools generally operate on SAM or BAM alignment files and can mark duplicate reads by setting a duplicate flag in the alignment record. Marking duplicates allows downstream programs to recognize them without necessarily deleting them from the file. This is often preferable because different analyses may require different duplicate-handling strategies.
- The following example illustrates duplicate marking using Picard after alignment and sorting:
picard MarkDuplicates \
I=sorted_alignment.bam \
O=marked_duplicates.bam \
M=duplicate_metrics.txt \
CREATE_INDEX=true
- The input file should be coordinate-sorted and prepared according to the requirements of the Picard version being used. The output BAM file contains duplicate flags, while the metrics file summarizes duplicate-related statistics. The duplicate reads remain in the alignment file but can be excluded by downstream tools when appropriate.
- SAMtools provides another widely used approach for duplicate marking. Its markdup command generally requires a coordinate-sorted and appropriately prepared alignment file. The following example illustrates a simplified workflow:
samtools sort -o sorted_alignment.bam alignment.bam
samtools fixmate -m sorted_alignment.bam fixmate.bam
samtools sort -o fixmate_sorted.bam fixmate.bam
samtools markdup fixmate_sorted.bam marked_duplicates.bam
samtools index marked_duplicates.bam
- For paired-end data, the preparation steps help provide the mate information required by samtools markdup. The precise workflow should follow the documentation for the installed SAMtools version and the characteristics of the alignment file. Duplicate marking should be checked through the resulting BAM flags and summary statistics.
- A duplicate rate is commonly calculated as the proportion of reads or read pairs identified as duplicates relative to the relevant total. The exact calculation depends on the software, whether the analysis counts individual reads or templates, and whether optical duplicates are included. For example, if 20 million read pairs are examined and 5 million are classified as duplicate pairs, the duplicate fraction is 25%. This value should not be interpreted as a universal measure of sequencing failure. A high duplicate fraction may indicate low library complexity, excessive PCR amplification, or deep sequencing of a limited molecular pool, but it can also be expected in certain biological and experimental contexts.
- Library complexity refers to the number and diversity of distinct molecules present in a sequencing library. A highly complex library contains a large number of unique fragments and can support the generation of many independent sequencing observations. A low-complexity library contains fewer unique molecules, so additional sequencing is increasingly likely to generate reads from molecules that have already been sampled. Library complexity is influenced by the quantity and quality of starting material, DNA fragmentation, RNA integrity, library preparation, PCR amplification, target enrichment, and the number of molecules successfully incorporated into the final library.
- A library may have low complexity even when its total sequencing read count is high. For example, a library containing one million unique fragments may be sequenced to produce ten million reads. Many of the resulting reads will represent repeated observations of the same original molecules. Increasing sequencing depth in this situation may provide limited additional information because the available molecular diversity has already been sampled extensively. In contrast, a highly complex library may continue to yield new unique fragments as sequencing depth increases.
- Library complexity is closely related to sequencing saturation. Sequencing saturation describes the extent to which additional sequencing is expected to recover new unique molecules or new information. At low sequencing depth, most reads may represent previously unobserved fragments. As sequencing continues, the probability of encountering a new molecule generally decreases. When a library approaches saturation, additional sequencing produces a progressively larger proportion of duplicate observations. Saturation is therefore an important consideration when deciding whether deeper sequencing is likely to improve an analysis.
- The relationship between sequencing depth and library complexity can be examined using a library saturation or rarefaction curve. Such a curve plots the number of unique molecules, unique fragments, or detected features against the number of reads sampled. A steep initial increase indicates that additional sequencing is recovering many new observations. A curve that gradually approaches a plateau suggests that most of the available diversity has been sampled. The shape of the curve depends on the distribution of molecule abundance and the method used to define uniqueness.
- In RNA sequencing, library complexity is often assessed in relation to transcript diversity and expression levels. Highly expressed genes can generate many reads, including multiple fragments with identical alignment coordinates. This may produce high duplication rates even when the library contains a large number of independent molecules. Therefore, duplicate rates in RNA-seq should be interpreted alongside gene expression distribution, transcript abundance, library preparation method, and other quality metrics. Removing all coordinate-identical reads from RNA-seq data can lead to underestimation of expression for highly expressed genes.
- In whole-genome sequencing, duplicate reads may result from PCR amplification or optical duplication, and duplicate marking can be useful when estimating coverage and calling genetic variants. However, naturally identical fragments and repetitive genomic regions can complicate duplicate identification. In targeted sequencing, duplication rates may be high because the number of possible fragment positions is restricted by the capture design or amplicon boundaries. A high duplication rate in a small gene panel does not necessarily indicate poor library preparation. It may instead reflect the limited number of possible unique fragments within the targeted regions.
- In whole-exome sequencing, hybridization capture concentrates sequencing reads within a relatively small portion of the genome. This can lead to repeated sequencing of the same captured fragments, particularly when the starting DNA quantity is limited or sequencing depth is high. Duplicate metrics should therefore be evaluated together with target coverage, uniformity, library input, and the proportion of target bases covered at the required depth. A high duplicate rate may reduce the effective number of independent molecules available for variant detection.
- In single-cell RNA sequencing, duplicate reads have a different interpretation because many protocols use unique molecular identifiers. UMIs are short sequences attached to original RNA-derived molecules before amplification. Reads with the same cell barcode, UMI, and compatible gene or transcript assignment can be grouped into molecular families. The number of distinct UMI-defined molecules can then be used to estimate transcript abundance more accurately than raw read counts alone. This approach helps distinguish repeated sequencing of amplified copies from independent original molecules.
- Unique molecular identifiers are especially valuable in applications where accurate molecule counting is required. They are used in single-cell RNA sequencing, targeted sequencing, circulating tumour DNA analysis, and other workflows where PCR amplification may distort apparent molecule abundance. A UMI is not the same as a sample index or cell barcode. A sample index identifies a library or sample in a multiplexed sequencing run, a cell barcode identifies a cell or partition, and a UMI identifies an original molecular template within the relevant sample or cell. Some protocols combine all three types of identifiers.
- UMI-based deduplication involves extracting barcode and UMI sequences, correcting sequencing errors in the UMI where appropriate, grouping reads into molecular families, and generating consensus sequences or molecule counts. Tools such as UMI-tools support UMI extraction, grouping, deduplication, and related analyses. Other tools and platform-specific pipelines may perform similar operations. UMI correction is important because sequencing errors in barcode sequences can cause reads from the same original molecule to be assigned to different groups. Conversely, UMI collisions can cause distinct molecules to be grouped together if they receive the same UMI sequence.
- The following example illustrates a simplified UMI extraction and deduplication workflow using UMI-tools. The barcode pattern and file names are illustrative and must be adapted to the actual library design:
umi_tools extract \
–bc-pattern=NNNNNNNN \
–stdin=sample_R1.fastq.gz \
–stdout=sample_R1_umi.fastq.gz \
–read2-in=sample_R2.fastq.gz \
–read2-out=sample_R2_umi.fastq.gz
umi_tools dedup \
–stdin=sorted_alignment.bam \
–stdout=deduplicated_alignment.bam \
–log=deduplication.log
- The extraction pattern specifies the position and length of the UMI in the input read. It should not be copied without checking the actual sequencing protocol. In practical workflows, the extracted UMI information must be retained in a format that can be recognized during alignment and deduplication. The deduplication stage also requires suitable alignment and grouping information.
- Library complexity can be estimated using several approaches. Coordinate-based duplicate metrics provide a practical indication of repeated fragment representation, but they do not always measure true molecular diversity. UMI-based molecule counting can provide a more direct estimate when UMIs are incorporated before amplification. Downsampling and rarefaction analysis can also help estimate how many unique fragments are likely to be recovered at different sequencing depths. Tools such as Picard, preseq, UMI-tools, and platform-specific sequencing analysis pipelines can support these assessments.
- Picard provides tools for estimating library complexity and summarizing duplicate metrics. Its EstimateLibraryComplexity program examines sequencing data to estimate the diversity of molecular fragments. The following command illustrates its use:
picard EstimateLibraryComplexity \
I=sorted_alignment.bam \
O=library_complexity_metrics.txt
- The input and interpretation requirements depend on the library type and the program’s supported data characteristics. Estimates based on alignment coordinates can be affected by target capture, repetitive sequences, fragment-size distributions, and naturally identical molecules. They should therefore be treated as estimates rather than exact counts of all original molecules.
- The preseq software can be used to estimate library complexity and predict the number of unique molecules expected as sequencing depth increases. Such predictions can help researchers decide whether additional sequencing is likely to provide useful new information. Preseq methods use observed duplicate and sampling patterns to estimate unseen molecular diversity. Their predictions depend on assumptions about the sampling process and may be less reliable when the library or sequencing process differs substantially from those assumptions.
- The following example illustrates a basic preseq command:
preseq lc_extrap \
-o complexity_prediction.txt \
-B \
sorted_alignment.bam
- The exact input requirements and options should be checked against the installed preseq version. The resulting estimates can be used to examine the relationship between sequencing effort and expected unique molecular observations. Researchers should avoid treating extrapolated values as direct measurements of the true library size.
- Duplicate reads and library complexity are also important in variant calling. If many reads originate from the same original DNA molecule, they do not represent an equivalent number of independent observations of a genetic variant. Counting all duplicate reads as independent evidence can exaggerate confidence in a variant call, particularly when sequencing errors or PCR errors occur early in amplification. Duplicate marking or UMI-based consensus analysis can help reduce this problem. However, variant callers differ in their treatment of duplicate reads, so the preprocessing strategy should be consistent with the selected caller and analysis protocol.
- In cancer sequencing and liquid biopsy analysis, molecular duplication and UMI-based consensus methods are especially important because variant allele frequencies may be very low. A rare mutation supported by several PCR copies of one original molecule provides less independent evidence than the same mutation observed in multiple distinct original molecules. Molecular barcoding can help estimate the number of independent molecules supporting a variant and reduce the influence of amplification and sequencing errors. Consensus-based methods may also improve accuracy, although their performance depends on UMI design, family size, sequencing depth, and error-correction procedures.
- Duplicate handling in metagenomics requires particular care. In shotgun metagenomic sequencing, identical reads may represent PCR duplicates, but they may also originate from abundant microbial genomes or highly conserved genomic regions. Removing duplicates indiscriminately can distort microbial abundance estimates and reduce the representation of genuinely abundant organisms. In amplicon sequencing, identical sequences may be the expected result of repeatedly amplifying the same biological marker from different organisms or molecules. UMI-based approaches can help distinguish original molecules from amplification copies when the library protocol includes suitable molecular identifiers.
- The interpretation of duplication metrics also depends on sequencing technology. Short-read sequencing workflows commonly identify duplicates through alignment coordinates and paired-end fragment information. Long-read sequencing can produce repeated observations of similar molecules, but its error profiles and library preparation strategies differ. PacBio circular consensus sequencing, for example, generates multiple observations of a circularized template to produce a high-accuracy consensus read. These internal passes are part of the consensus-generation process and should not be confused with ordinary PCR duplicate reads in a short-read alignment file. Oxford Nanopore workflows also require platform-specific consideration of read identity, basecalling, and library design.
- Several factors can reduce library complexity. Low starting DNA or RNA quantity can limit the number of original molecules available for sequencing. Degraded nucleic acids can reduce the number of usable fragments. Excessive PCR amplification can increase the representation of a small subset of molecules. Inefficient fragmentation, size selection, adapter ligation, or target capture can also reduce molecular diversity. In targeted sequencing, the number of possible unique fragments may be inherently limited by the size and design of the target regions. Researchers should therefore assess library complexity in relation to the experimental protocol rather than applying a single universal threshold.
- Improving library complexity begins with good sample preparation. Sufficient quantities of high-quality DNA or RNA should be used whenever possible. Nucleic acid extraction should minimize degradation and contamination. Library preparation protocols should be optimized to reduce unnecessary amplification, and PCR cycle numbers should be kept as low as practical while still producing sufficient library material. Efficient adapter ligation, appropriate size selection, and careful handling can also improve the proportion of original molecules represented in the final library. When supported by the protocol, PCR-free library preparation can reduce amplification-related duplication.
- UMI-based library preparation can improve the ability to distinguish original molecules from PCR-derived copies. However, UMIs do not automatically eliminate all duplication or sequencing errors. Their effectiveness depends on the number and design of possible UMI sequences, the timing of UMI attachment, sequencing accuracy, and the quality of the computational grouping procedure. A small UMI space can lead to collisions, while sequencing errors can split one molecular family into multiple apparent UMIs. Proper UMI correction and consensus generation are therefore important.
- A useful sequencing quality assessment should examine duplication metrics alongside other measurements, including total read count, mapped read percentage, coverage depth, coverage uniformity, insert-size distribution, GC content, library concentration, and the number of unique molecules. In RNA-seq, gene expression distribution and transcript abundance should also be considered. In single-cell sequencing, cell barcode counts, UMI counts, detected genes per cell, and sequencing saturation provide additional information. No single duplication threshold is suitable for every sequencing application.
- For reproducible bioinformatics analysis, researchers should record the software and versions used for duplicate identification, the criteria applied, whether duplicates were marked or removed, and whether UMI-based processing was performed. They should preserve original sequencing files and relevant intermediate files, including alignment files and duplicate metrics. The library preparation method, input nucleic acid quantity, PCR cycle number, sequencing platform, and relevant sample information should also be documented. Workflow managers such as Snakemake and Nextflow can help standardize duplicate handling and library complexity assessment across samples.
- Duplicate reads and library complexity provide complementary information about the quality and effective molecular diversity of sequencing datasets. Duplicate metrics indicate how frequently reads represent repeated observations of fragments, while library complexity describes the diversity of original molecules available for sequencing. Their interpretation depends on the sequencing technology, library preparation method, biological material, and research objective. By combining appropriate duplicate identification, UMI-based molecular counting where available, complexity estimation, and careful experimental design, researchers can make better use of sequencing data and avoid overestimating the amount of independent biological information represented by raw read counts.