Skip to content
BGI-flexlabPublic

About

A Tool for integrated Quality Control and Preprocessing on FASTQ or BAM/CRAM files

Resources

Stars

118 stars

Watchers

8 watching

Forks

Repository files navigation

SOAPnuke 2.1.10

SOAPnuke3 is available

SOAPnuke3 is the new Rust-based generation of SOAPnuke. The WGS benchmark below compares its preprocessing performance and downstream variant-calling results with five other tools. See the SOAPnuke3 repository for installation and the migration guide when updating existing workflows.

HG002 whole-genome benchmark

We compared six tools on the complete 430,952,721-pair HG002 WGS dataset (58.144 GB of compressed input, approximately 41.6× nominal depth), with three preprocessing runs at each of 8, 16 and 32 allocated physical cores.

At 32 cores, SOAPnuke 3.0.0 completed in 6.27 minutes, compared with 8.46 minutes for fastp 1.4.0: 25.8% less elapsed time. Median peak RSS was 0.225 GiB, compared with 0.533 GiB for fastp. These are results for this dataset and the explicit profiles below.

Six-tool HG002 preprocessing throughput, peak memory and effective CPU use across 8, 16 and 32 allocated cores

Lines show three-run medians; faint points show individual observations. Throughput uses compressed-input MB/s. Memory uses a logarithmic axis; CPU is measured effective cores, not requested threads.

Preprocessing at 32 cores

Tool Wall time (s) CPU (%) Peak RSS (GiB) Input (MB/s) Retained pairs
SOAPnuke 3.0.0 376.49 1,223.0 0.225 154.44 430,883,998
SOAPnuke 2.1.10 1,201.45 3,135.4 8.530 48.39 430,883,993
SOAPnuke 1.6.5 11,833.00 234.1 0.647 4.91 430,883,993
fastp 1.4.0 507.37 1,157.6 0.533 114.60 430,889,028
Trimmomatic 0.41 15,475.00 161.5 9.749 3.76 430,889,593
Cutadapt 5.2 751.98 806.4 0.081 77.32 429,492,691

CPU 100% represents one effective core. Each metric is its own three-run median. See all 18 cells and ranges and all 54 observations.

SOAPnuke3, fastp, Trimmomatic and Cutadapt used gzip level 4; v1/v2 retained their historical native compression settings. Trimmomatic and Cutadapt performed quality-end trimming, while the SOAPnuke and fastp profiles used read filtering. Exact parameters and boundary differences are in the methods.

Downstream variant-calling accuracy

Retained paired reads entered a common BWA-MEM2 → GATK duplicate marking/BQSR → HaplotypeCaller → hard-filtering workflow. SNP and INDEL calls were evaluated with hap.py/RTG vcfeval against GIAB HG002 GRCh38 v4.2.1 in the same confident regions.

Tool SNP precision (%) SNP recall (%) SNP F1 (%) INDEL precision (%) INDEL recall (%) INDEL F1 (%)
Cutadapt 5.2 99.6849 98.0568 98.8641 99.4418 99.0469 99.2440
SOAPnuke 3.0.0 99.6876 98.0504 98.8622 99.4573 99.0279 99.2422
SOAPnuke 1.6.5 99.6883 98.0510 98.8629 99.4559 99.0273 99.2411
SOAPnuke 2.1.10 99.6875 98.0505 98.8622 99.4568 99.0277 99.2418
fastp 1.4.0 99.6880 98.0505 98.8625 99.4562 99.0277 99.2415
Trimmomatic 0.41 99.6879 98.0485 98.8614 99.4559 99.0300 99.2425

These results show closely similar observed calling accuracy on this sample. The differences are small and describe this single sample. Download the accuracy table as SVG. Formal SOAPnuke 3.0.0 and fastp 1.4.0 calling results were reused after both retained input mates were confirmed byte-identical to the already analyzed files.

All measurements used a shared dual-socket Xeon Gold 6442Y host and Lustre storage. CPU placement and measurement dates differ across campaign stages; cores were not exclusively reserved. See the complete methods, commands and provenance for the scope and reproducibility details.

About SOAPnuke 2.1.10

SOAPnuke 2.1.10 is a maintenance release of the v2 C++ implementation. It fixes large-count QC percentiles, positional-statistics aggregation and discarded-base reporting while preserving the existing filtering policy. See release notes and build instructions.

Modules

Module Purpose
filter Trim/filter single-end or paired-end FASTQ and produce QC statistics.
filterHts Process BAM/CRAM; enable optional HTSlib support when building. Paired inputs must be sorted by read ID.
filterStLFR Process StLFR FASTQ with barcode detection and list input.
filtersRNA Small-RNA preprocessing.
filterMeta Metagenomic-read preprocessing.

The WGS results above cover ordinary paired-end filtering, not the specialist modules. V3 interfaces and supported workflows differ; this maintenance release retains the v2 modules.

Getting started

Requirements

GNU Make, a C++11 compiler, zlib development headers and pthreads. HTSlib is needed only when enabling USEHTS=true; the default build keeps HTS support disabled. See BUILD.md.

Install

git clone https://github.com/BGI-flexlab/SOAPnuke.git
cd SOAPnuke
make
./SOAPnuke -v

For BAM/CRAM support, install HTSlib and set USEHTS=true in the Makefile before building. The default is USEHTS=false.

QuickStart

All usages start with executable file SOAPnuke, and different modules are invoked with different sub-commands. Here are some usage examples:

    #filter:

	#QC the input fastq and extract 10M clean reads to the output files.
	echo "totalReadsNum=10000000" >config.txt

    SOAPnuke filter -1 test.r1.fq.gz -2 test.r2.fq.gz -C clean_1.fq.gz -D clean_2.fq.gz -o result -T 8 -c config.txt
    
    
    #filterHts:
    
    SOAPnuke filterHts --ref chr21.fa -1 input.bam -2 output.cram  -o result
	SOAPnuke filterHts -1 input.bam  -2 output.bam -o result


    #filterStLFR:

    SOAPnuke filterStLFR -1 fq1.list -2 fq2.list -C clean1.gz -D clean2.gz -o result -T 8 -c config

Detailed QC steps

If set trim-related parameters(no trim if not set), do trimming first:


Read ID

If parameter “index” set in config file, remove index sequence from read ID.

Once “index” is set, if seqType is 0(default value), read ID would be expected like:

@FCD1PB1ACXX:4:1101:1799:2201#GAAGCACG/2, 

“#GAAGCACG” would be removed then.

If seqType is 1, read ID would be expected like:

@HISEQ:310:C5MH9ANXX:1:1101:3517:20432:N:0:TCGGTCAC, 

“:TCGGTCAC” would be removed then.

Read sequence and quality

First, the cutting length of all trimming type would be calculated, including hard trim, low quality end trim, adapter trim and tail-polyG trim. The longest cutting would be performed.

  • hard trim: directly remove a certain length sequence from head or tail on read sequence
  • low quality end trim: remove low quality base starting from end until quality higher than cutoff
  • adapter trim: when adapter was found, the base sequence and quality sequence would be trimmed from the start position which match adapter
  • tail-polyG trim: if polyG number is greater than cutoff, then these polyG sequence in tail would be trimmed

Then do filtering:

Note that the read pair would be both discarded both when any of which fails to pass QC.

Priority(High to Low):

  • Tile, may be used in some types of BGI data.

        If you want to discard reads with certain tile ID, set the parameter like “1101-1104,1205”.

  • Fov, may be used in data from zebra-platform.

        If you want to discard reads with certain FOV ID, set the parameter like “C001R003,C003R004”.

  • Minimal read length

        Discard a read with sequence length shorter than the parameter.

  • Maximal read length

        Discard a read with sequence length longer than the parameter.

  • N ratio

        Discard a read with N base ratio not smaller than the parameter.

  • High A ratio

        Discard a read with A base ratio not smaller than the parameter.

  • polyX number (X means any one base)

        Discard a read with poly-X number not smaller than the parameter.

  • Low quality base ratio

        Discard a read with low-quality bases ratio not smaller than the parameter.

  • Mean quality

        Discard a read of which mean quality of sequence smaller than the parameter.

  • Overlapped length if PE

        Discard a read pair which is suspected to be overlapped longer then the parameter.

  • Adapter

        Discard a read which contains an adapter.

Parameter

Commonly used parameters


filter module

  • -1 / --fq1

fq1 file(required), .gz or normal text format are both supported

  • -2 / --fq2

fq2 file(used when process PE data), format should be same as fq1 file, both are gz or both are normal text

  • -C / --cleanFq1

reads which passed QC from fq1 file would output to this file

  • -D / --cleanFq2

reads which passed QC from fq2 file would output to this file

  • -o / --out

Output directory. Processed fq files and statistical results would be output to here

  • -f / --adapter1

adapter sequence or list file of read1

  • -r / --adapter2

adapter sequence or list file of read2

  • -J / --ada_trim

trim read when find adapter, it’s a bool parameter, default is false which means discard the read when find adapter

  • -T / --thread

threads number used in process, default value is 6

  • -c / --configFile

config file which include uncommonly used parameters. Each line contains a parameter, e.g., for value needed parameter: adaMis=2, for bool parameter: contam_trim, which means set mode as discard when find contaminant sequence

  • -l / --lowQual

low quality threshold, default value is 5

  • -q / --qualRate

low quality rate threshold, default value is 0.5

  • -n / --nRate

N rate threshold, default value is 0.05

  • -m / --mean

low average quality threshold, if you want discard reads with low average quality, you can set a value. The software do NOT check this item by default

  • -p / --highA

ratio of A threshold in a read, the software do NOT check this item by default

  • -g / --polyG_tail

polyG number threshold in read tail, the software do NOT check this item by default

  • -X / --polyX

polyX number threshold, the software do NOT check this item by default

  • -4 / --minReadLen

read minimal length, default value is 30

  • -h / --help

Show help information

  • -v / --version

Show version information


filterHts module

Here we only present options different from filter module.

  • -E / --ref

reference file(required when process cram format)

  • -1

input bam/cram file(required)

  • -2

output bam/cram file(required)


filterStLFR module

Here we only present options different from filter module.

  • -1 / --fq1

Support FASTQ files list as input

  • -2 / --fq2

Support FASTQ files list as input


Uncommonly used parameters

  • ctMatchR

Contaminant sequence shortest consistent matching ratio [default:0.2]

  • seqType

Sequence fq name type, 0->old fastq name, 1->new fastq name [0]

old fastq name: @FCD1PB1ACXX:4:1101:1799:2201#GAAGCACG/2

new fastq name: @HISEQ:310:C5MH9ANXX:1:1101:3517:2043 2:N:0:TCGGTCAC

  • trimFq1

trim fq1 file name(gz format) [optional]

  • trimFq2

trim fq2 file name [optional]. If trim related parameters were set on, these output files would include the total reads which only do trimming. For example, if read A failed QC after trimming, it will still output to -R/-W, but not to -C/-D

  • tile

tile number to ignore reads, such as [1101-1104,1205]

  • fov

fov number to ignore reads (only for zebra-platform data), such as [C001R003,C003R004]

  • barcodeListPath

barcode list of two columns:sequence and barcodeID

  • barcodeRegionStr

barcode regions, such as: 101_10,117_10,145_10 or 101_10,117_10,133_10

  • notCutNoLFR

do not cut sequence when fail found barcode

  • inputAsList

input file list not a file

  • tenX

output tenX format

  • outFileType

output file format: fastq or fasta[default: fastq]

  • index

remove index

  • totalReadsNum

number/fraction of reads you want to keep in the output clean FASTQ file(cannot be assigned when -w is given). It will extract reads randomly through the total clean FASTQ file by default, you also can get the head reads for save time by add head suffix to the integer

  • trim

trim some bp of the read's head and tail, they means: (PE type:read1's head and tail and read2's head and tail [0,0,0,0]; SE type:read head and tail [0,0])

  • trimBadHead

Trim from head ends until meeting high-quality base or reach the length threshold, set (quality threshold,MaxLengthForTrim) [0,0]

  • trimBadTail

Trim from tail ends until meeting high-quality base or reach the length threshold, set (quality threshold,MaxLengthForTrim) [0,0]

  • overlap

filter the small insert size.Not filter until the value exceed 1 [-1]

  • mis

the maximum mismatch ratio when find overlap between PE reads(depend on -O) [0.1]

  • patch

Reads processed per worker batch. When omitted, ordinary FASTQ processing uses 2500 × threads; an explicit patch value overrides this default.

  • qualSys

quality system 1:64, 2:33 [default:2]

  • outQualSys

out quality system 1:64, 2:33 [default:2]

  • maxReadLen

read max length, default 49 for filtersRNA, the software do NOT check this item by default in other modules

  • cleanOutSplit

max reads number in each output clean FASTQ file

  • pe_info

Add /1, /2 at the end of FASTQ name. [default: not add]

  • baseConvert

convert base when write data, example: TtoU , means convert base T to base U in the output

  • log

log file output path


Plotting

The three scripts in src/Rscripts/ are used for plotting QC stats from SOAPnuke.

Q20Q30.R

USAGE:

Rscript src/Rscripts/Q20Q30.R Distribution_of_Q20_Q30_bases_by_read_position_1.txt Distribution_of_Q20_Q30_bases_by_read_position_2.txt q2030.png

dWZF5F.png


base.R

USAGE:

Rscript src/Rscripts/base.R Base_distributions_by_read_position_1.txt Base_distributions_by_read_position_2.txt raw.png clean.png

dWZiUU.png


quality.R

Rscript src/Rscripts/quality.R Base_quality_value_distribution_by_read_position_1.txt Base_quality_value_distribution_by_read_position_2.txt rawQuality.png cleanQuality.png 0 0

dWZAC4.png


Availability

SOAPnuke is released under GPLv3. The latest source code is freely available at github.

Citing SOAPnuke

  • Chen Y, Chen Y, Shi C, et al. SOAPnuke: a MapReduce acceleration-supported software for integrated quality control and preprocessing of high-throughput sequencing data. Gigascience. 2018;7(1):1-6. doi:10.1093/gigascience/gix120 [PMID: 29220494]

About

A Tool for integrated Quality Control and Preprocessing on FASTQ or BAM/CRAM files

Resources

Stars

118 stars

Watchers

8 watching

Forks

Releases

Packages

Contributors

Languages