Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

133 Commits

Folders and files

Repository files navigation

fluscan-nf

This is a Nextflow pipeline for Influenza A mutation analysis from either short-read FASTQ data or pre-built consensus FASTA files. The short-read workflow performs trimming, assembly, mapping, LoFreq variant calling, SnpEff annotation, and consensus generation. In FASTA mode (--fasta_input), the pipeline currently launches both a PAFTOOLS branch and a NEXTCLADE branch: PAFTOOLS validates the supplied FASTA, selects a subtype-matched reference with BLAST, aligns per segment with minimap2, calls differences with paftools.js, annotates with SnpEff, and fixes multi-nucleotide variant codons; NEXTCLADE validates the same input, batches sequences by segment for nextclade run, and converts the results to VCF, annotating substitutions that share a codon with their combined codon change.

Analyses

Short-read workflow

  • Read trimming & QC: fastp
  • Primer removal with cutadapt
  • FASTQ quality reporting with FastQC
  • Aggregate the reports with MultiQC
  • De-novo assembly
  • Automated reference selection from built-in database via BLAST
  • Reference-based read mapping (BWA or minimap2)
  • Nucleotide variant calling using LoFreq
  • Annotation of amino acid variant calls using SnpEff
  • MNV codon correction
  • Detection of mutations of interest via customisable watchlist scanning of the final VCF
  • Consensus sequence generation
  • Detection of influenza molecular markers with FluMut

PAFTOOLS workflow (consensus FASTA input)

  • Consensus FASTA validation and header normalisation
  • Automated reference selection from built-in database via BLAST
  • Per-segment alignment using minimap2 (asm20 mode)
  • Variant calling using paftools.js
  • Annotation of amino acid variant calls using SnpEff
  • MNV codon correction
  • Alignment coverage QC with run-level aggregation
  • Detection of influenza molecular markers with FluMut

NEXTCLADE workflow (consensus FASTA input)

  • Consensus FASTA validation and header normalisation
  • Per-segment batching across samples
  • Per-segment nextclade run against built-in datasets
  • Per-segment QC from the Nextclade alignment; a segment that fails gets no VCF
  • Conversion to freebayes-style VCF, with substitutions that share a codon annotated as one codon change
  • Detection of influenza molecular markers with FluMut
  • Optional (--merge_flumut_mutations, default off): reference coordinate conversion — builds a run-level, idempotent nucleotide/amino-acid coordinate map lifting flumut's fixed reference onto the selected Nextclade reference dataset coordinate system (bin/build_reference_coordinate_map.py) — followed by FluMutDB annotation of the VCF, which tags each fluscan amino-acid call that matches a curated FluMutDB marker with a FLUMUT= INFO field carrying the mutation name and its published effects, split into those directly attributable to that mutation and those associated with it through a multi-mutation marker, each qualified by the subtype it was observed in. No rows are added or removed, so a marker is only reported where fluscan called a difference from the Nextclade reference. Because FluMutDB numbers HA by its HA1/HA2 subunits, HA markers are only annotated when the Nextclade dataset also exposes that split — either as two dedicated HA1/HA2 CDS features, or as mature_protein_region_of_CDS children of a single HA CDS (the NCBI/RefSeq model, which the GsGD dataset uses). A dataset offering only a bare HA CDS is flagged stale_annotation and passes through with its HA rows unannotated
flowchart TD
    input[Consensus FASTA input directory]
    dispatch[main.nf FASTA dispatch]

    input --> dispatch
    dispatch --> prepare

    subgraph FASTA_PREP[FASTA_PREP workflow]
        direction TB
        prepare[prepare_consensus]
        prep_flumut[flumut]

        prepare --> prep_flumut
    end

    subgraph PAFTOOLS[PAFTOOLS workflow]
        direction TB
        blastn[blastn reference selection]
        covagg[aggregate coverage summary]
        paf_final[Final PAFTOOLS VCFs]

        subgraph paftools_variants[paftools_variants, one task per sample]
            direction TB
            minimap[minimap2 asm20]
            covqc[coverage QC]
            paftools_call[paftools.js call]
            snpeff[SnpEff]
            paf2fb[paftools_to_freebayes_vcf.py]
            fixmnv_paf[fix_mnv_codons.py]

            minimap --> covqc
            minimap --> paftools_call
            paftools_call --> snpeff
            snpeff --> paf2fb
            covqc --> paf2fb
            paf2fb --> fixmnv_paf
        end

        prepare --> blastn
        prepare --> minimap
        blastn --> minimap
        covqc --> covagg
        fixmnv_paf --> paf_final
    end

    subgraph NEXTCLADE[NEXTCLADE workflow]
        direction TB
        groupseg[collect each segment across samples: splitFasta + collectFile]
        nc_run[nextclade run batch]
        nc2vcf[nextclade_to_freebayes_vcf.py: QC + VCF]
        nc_final[Final NEXTCLADE VCFs]
        coord_map[coordinate_conversion: build_reference_coordinate_map.py]
        coord_final[aggregate/nextclade/db/reference_remap]
        flumut_merge[merge_flumut_mutations: merge_flumut_mutations.py]
        flumut_annotated[FluMutDB-annotated VCFs]

        prepare --> groupseg
        groupseg --> nc_run
        nc_run --> nc2vcf
        nc2vcf --> nc_final
        dispatch -.->|"flumutdb reference + db nextclade datasets, sample-independent (if merge_flumut_mutations)"| coord_map
        coord_map --> coord_final
        coord_map -.->|"per-segment shard (if merge_flumut_mutations)"| flumut_merge
        nc_final -.-> flumut_merge
        flumut_merge --> flumut_annotated
    end
Loading

Input

Short-read workflow: Illumina paired-end FASTQ files (*.fastq.gz, *.fq.gz, *.fastq, *.fq), or a sample sheet (below).

PAFTOOLS/NEXTCLADE workflows: Consensus FASTA files (one multi-FASTA per sample, one sequence per influenza segment). Two header formats are accepted:

  • Underscore-delimited (legacy): >SAMPLE_ID_segmentN_SEGMENT e.g. >WIN-AH-2024-FAV-0415_segment1_PB2
  • Pipe-delimited (GISAID-style): >EPI_ISL_ACCESSION|subtype|SEGMENT|...|clade_SEGMENT e.g. >EPI_ISL_20399898|A_/_H5N1|NA|6|EPI5261179|2.3.4.4b_NA

Cleaned output headers ({sample_id}.fasta) are normalised to {sample_id}_{SEGMENT} (e.g. EPI-ISL-20399898_NA).

Sample sheet input

--samplesheet_input takes a CSV in place of --fastq_input or --fasta_input. Its header picks the workflow:

ID,R1,R2
SAMPLE-01,/path/to/SAMPLE-01_R1.fastq.gz,/path/to/SAMPLE-01_R2.fastq.gz
SAMPLE-02,/path/to/SAMPLE-02_R1.fastq.gz,/path/to/SAMPLE-02_R2.fastq.gz

runs the short-read workflow, and

ID,FASTA
SAMPLE-01,/path/to/SAMPLE-01.fasta
SAMPLE-02,/path/to/SAMPLE-02.fasta

runs PAFTOOLS and NEXTCLADE. A FASTA sheet may add a DB column naming each sample's reference bundle, so one run can mix subtypes:

ID,FASTA,DB
SAMPLE-01,/path/to/SAMPLE-01.fasta,/path/to/refs/gsgd
SAMPLE-02,/path/to/SAMPLE-02.fasta,/path/to/refs/wisconsin588

A blank DB cell falls back to --db; see docs/database_bundles.md. Other columns are ignored.

  • Each ID is cleaned by the same rule as a FASTA filename (see the note under Usage), so 24-1234-IP7-SAMPLE_01 is reported as SAMPLE-01. After cleaning, an ID may contain only letters, digits, . and -.
  • Relative paths resolve against the directory nextflow run is launched from; absolute paths are safer.
  • The run stops before any task starts if a column or value is missing, a file does not exist, or two IDs clean to the same sample name.
  • run_name, which prefixes the run-level reports, is the sample sheet's filename without .csv.
  • aggregate/sample_ids.csv maps each input ID (or FASTA filename) to its cleaned sample_id and bundle name.
  • The short-read workflow takes a single --db and rejects a sheet with a DB column.
  • Give only one of --fastq_input, --fasta_input and --samplesheet_input.

Reference databases

Every reference the pipeline needs for one reference set lives in a single directory — a bundle — passed as --db:

<db>/
├── db.yml            optional metadata (name, version, …)
├── blastn_db.fa      nt reference; headers accession|subtype|segment
├── nextclade/        one uppercase directory per segment (HA/, NA/, …)
└── snpeff/           snpEff.config + <genome>/

Bundles may be partial — each workflow validates only the components it consumes, and fails at the start of the run, before any task is submitted, naming the path it expected. See docs/database_bundles.md for the full layout, the db.yml schema, and how to build and validate a bundle.

scripts/validate_db.py /path/to/refs/gsgd

Usage

Example command (short-read):

nextflow run BCCDC-PHL/fluscan-nf \
  -r v0.1.0 \
  -profile conda \
  --cache ~/.conda/envs \
  --fastq_input /path/to/your_fastqs \
  --db /path/to/refs/gsgd \
  --outdir /path/to/output_dir

To run from a sample sheet, replace --fastq_input (or --fasta_input below) with --samplesheet_input /path/to/samplesheet.csv; see Sample sheet input.

Optional short-read parameters:

  • --aligner — bwa (default) or minimap2

Optional in every workflow:

  • --watchlist_dir — directory of *watchlist.csv files (gene,mutation[,note]) scanned out of the final VCF into a published TSV. Defaults to assets/watchlists/; set to NO_FILE to disable. HA1/HA2 entries are matched against the reference annotation's mature-peptide coordinates, so they work whether or not the VCF annotates the subunits directly. See docs/watchlists.md

FASTA-mode example (runs both PAFTOOLS and NEXTCLADE):

nextflow run BCCDC-PHL/fluscan-nf \
  -r v0.1.0 \
  -profile conda \
  --cache ~/.conda/envs \
  --fasta_input /path/to/consensus_fastas \
  --db /path/to/refs/gsgd \
  --merge_flumut_mutations \
  --outdir /path/to/output_dir

Note: In FASTA mode the sample name comes from the input filename (or the sample sheet ID, cleaned by the same rule), and it names every output. The filename is taken without its extensions, a trailing _consensus is dropped, underscores become dashes, and a leading submitter prefix of the form 24-1234- or 24-1234-IP7- is stripped — so 24-1234-IP7-SAMPLE_A_consensus.fasta is reported as SAMPLE-A. Input FASTA headers are read only for the segment label; they do not affect naming.

FASTA mode launches both branches, so the bundle must carry all three components: PAFTOOLS requires blastn_db.fa and snpeff/, NEXTCLADE requires nextclade/ with all eight segment datasets.

In the short-read workflow only blastn_db.fa is required. SnpEff annotation — and the MNV correction and watchlist scan that read its output — runs when the bundle carries a snpeff/ directory, and is skipped when it does not; consensus and FluMut complete either way.

--merge_flumut_mutations (boolean, default false) gates the two final NEXTCLADE steps — coordinate conversion and FluMutDB annotation of the VCF (*_flumut_annotated.vcf). The coordinate_conversion step lifts FluMut's own reference, read from the flumutdb conda package, onto the --db datasets. The marker set comes from the same package, so the two always match and the annotation step does not consume the per-sample flumut output.

--emit_no_mature_vcf (boolean, default false) additionally writes each NEXTCLADE VCF a second time with no mature-peptide ANN= entries, for consumers that require HA annotated as HA0 only. The two files carry the same records, alleles and order; only the ANN= sub-field differs, and segments without mature peptides in their annotation come out identical. The published copy lands in nextclade/no_mature_peptide/. Nothing downstream in the pipeline reads it: mutation_watch and merge_flumut_mutations resolve HA markers through the HA1/HA2 Feature_ID and stay on the standard VCF.

Package management:

  • Conda is used for dependency management. The -profile conda flag and --cache parameter are required on BCCDC systems.
  • If environment resolution fails on first run, ensure mamba is available (the pipeline uses conda.useMamba = true).

Output

Outputs are written to the directory specified with the --outdir parameter. Sample-level outputs are written to one folder per sample, and some workflows also emit run-level aggregate outputs.

Short-read workflow outputs (per sample)

File Description
*.fastp.html Per-sample fastp HTML report
*.fastp.json Per-sample fastp JSON report
*_contigs.fasta De-novo assembled contigs from SPAdes
*_consensus.fasta Multi-FASTA containing the consensus sequences for each segment
blast/*_ref.fasta Multi-FASTA containing the optimal reference sequences selected for each segment
blast/*_blastn.tsv BLAST search results of assembled contigs against the reference database
blast/*_blastn_filter.tsv Filtered BLAST search results
*.sort.filter.bam Sorted, filtered alignment BAM used for downstream calling and coverage calculations
*_coverage.png Line plots describing mapping coverage across all 8 flu segments
*_pileup.tsv Pileup TSV with raw counts of all 4 nucleotides and gaps
*_depth.tsv Per-position alignment depth (samtools depth)
*_low_coverage.bed Low-coverage regions masked during consensus generation
short_read/*_final.vcf Per-segment final VCF after SnpEff annotation and MNV codon correction
watchlist/<list>/*_<list>_sr_watchlist_hits.tsv Optional (--watchlist_dir): watchlist substitutions found in the final VCF, one file per segment
flumut/*_flumut_markers.tsv FluMut table of molecular markers identified in the consensus sequences
flumut/*_flumut_mutations.tsv FluMut long-format table of amino acid mutations at positions of interest
flumut/*_flumut_mutations_wide.tsv FluMut wide-format table of amino acid mutations at positions of interest
flumut/*_flumut_literature.tsv FluMut literature references associated with the identified markers
*_short_read_*_provenance.yml Aggregated per-sample provenance file for the short-read workflow

The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder FORMAT/GQ as an integer.

PAFTOOLS workflow outputs (per sample, consensus FASTA input)

File Description
${sample_id}.fasta Validated consensus FASTA with headers normalised to {sample_id}_{SEGMENT}
qc/*_qc_report.tsv QC report from consensus preparation
blast/*_ref.fasta Multi-FASTA containing the optimal reference sequences selected for each segment
blast/*_blastn.tsv BLAST search results against the reference database
blast/*_blastn_filter.tsv Filtered BLAST search results
qc/*_coverage_qc.tsv Per-segment PAFTOOLS consensus coverage QC with coverage percentage and PASS/WARN/FAIL status. A segment with no BLAST reference, or that minimap2 cannot align, is FAIL and has no VCF
paftools/*_final.vcf Per-segment final VCF after paftools.js, SnpEff, freebayes-style conversion, and MNV codon correction
watchlist/<list>/*_<list>_pf_watchlist_hits.tsv Optional (--watchlist_dir): watchlist substitutions found in the final VCF, one file per segment
flumut/*_flumut_markers.tsv FluMut table of molecular markers identified in the consensus sequences
flumut/*_flumut_mutations.tsv FluMut long-format table of amino acid mutations at positions of interest
flumut/*_flumut_mutations_wide.tsv FluMut wide-format table of amino acid mutations at positions of interest
flumut/*_flumut_literature.tsv FluMut literature references associated with the identified markers
*_paftools_*_provenance.yml Aggregated per-sample provenance file for the PAFTOOLS workflow

NEXTCLADE workflow outputs (per sample, consensus FASTA input)

File Description
nextclade/*_final.vcf Per-segment final VCF from the Nextclade-to-VCF conversion, written only for segments that do not FAIL QC (see aggregate/nextclade_qc_summary.tsv). Rows in a codon holding more than one substitution carry the combined change, plus MNV_CORRECTED and MNV_ORIGINAL_HGVS_P where it differs from the call before codon correction (Nextclade's, otherwise the single-base change)
nextclade/no_mature_peptide/*_nc_nomat_final.vcf Optional (--emit_no_mature_vcf): the same records annotated without mature peptides, so HA carries only its HA0 ANN= entry and no HA1/HA2 ones
nextclade/flumut_annotated/*_flumut_annotated.vcf Optional (--merge_flumut_mutations): the same VCF with a FLUMUT= INFO field (Protein|AA_Pos|Alt_AA|Mutation_Name|DirectEffect|AssociatedEffect) on rows whose amino-acid change matches a curated FluMutDB marker
watchlist/<list>/*_<list>_nc_watchlist_hits.tsv Optional (--watchlist_dir): watchlist substitutions found in the final VCF, one file per segment, beside the PAFTOOLS (_pf_) ones. Independent of --merge_flumut_mutations: FLUMUT= is a curated annotation written into the VCF, this is a custom list scanned out of it
flumut/*_flumut_markers.tsv FluMut table of molecular markers identified in the consensus sequences
flumut/*_flumut_mutations.tsv FluMut long-format table of amino acid mutations at positions of interest
flumut/*_flumut_mutations_wide.tsv FluMut wide-format table of amino acid mutations at positions of interest
flumut/*_flumut_literature.tsv FluMut literature references associated with the identified markers
*_nextclade_*_provenance.yml Aggregated per-sample provenance file for the NEXTCLADE workflow

Run-level outputs

File Description
aggregate/${run_name}_fluscan-nf_multiqc_report.html MultiQC report summarising fastp, cutadapt, and FastQC results for all samples (short-read workflow only)
aggregate/paftools_qc_summary.tsv Run-level tabular summary of PAFTOOLS alignment coverage across all FASTA samples and segments
aggregate/nextclade_qc_summary.tsv One row per sample and segment Nextclade saw (NEXTCLADE workflow). status is FAIL, with no VCF written, when Nextclade could not align the segment (not_aligned), reports a frameshift (frameshift), or scores its stop codons bad (stop_codons); WARN when coverage_pct is below --consensus_coverage_pass_pct (default 90); otherwise PASS. coverage_pct is the share of the reference covered by called (non-N) bases, so a truncated or N-heavy segment WARNs but keeps its VCF. identity_pct is exact matches over the called bases plus insertions; it has no threshold and is low for a segment from a different lineage than the reference
aggregate/sample_ids.csv input_id,sample_id,db: each input's sample sheet ID (or FASTA filename), its cleaned sample_id, and the bundle it ran against (FASTA mode)
aggregate/consensus_qc_summary.tsv Every *_qc_report.tsv row from consensus preparation, concatenated across the run. A sample rejected at preparation — duplicate segment, or headers naming more than one sample — contributes no rows and is reported only in the Nextflow log
aggregate/nextclade/<db>/<SEGMENT>/batch.* Raw nextclade run --output-all output for one bundle and segment across every sample in the run (NEXTCLADE workflow)
aggregate/nextclade/<db>/reference_remap/coordinate_map.json Optional (--merge_flumut_mutations): primary machine-readable nucleotide/amino-acid coordinate map lifting flumut's reference (source) onto bundle <db>'s Nextclade reference datasets (target)
aggregate/nextclade/<db>/reference_remap/coordinate_map.nt.tsv Flat per-position nucleotide coordinate map (source/flumut position/base -> target/nextclade position/base, per segment)
aggregate/nextclade/<db>/reference_remap/coordinate_map.aa.tsv Flat per-residue amino-acid coordinate map (source/flumut position/residue -> target/nextclade position/residue, per segment/gene)
aggregate/nextclade/<db>/reference_remap/coordinate_map.qc.tsv Per-segment/per-gene alignment QC: identity, coverage, the number of co-optimal alignments the map's gap placement was chosen from, and status (ok/warning/missing_in_source/extra_in_source/stale_annotation)
aggregate/nextclade/<db>/reference_remap/flumut_alignment.fasta Target/source (nextclade/flumut) nucleotide alignment blocks, one per segment
aggregate/nextclade/<db>/reference_remap/flumut_aa_alignment.fasta Target/source (nextclade/flumut) amino-acid alignment blocks, one per gene
${run_name}_fluscan-nf_nextflow_report.html Nextflow execution report
${run_name}_fluscan-nf_nextflow_timeline.html Nextflow execution timeline
${run_name}_fluscan-nf_nextflow_trace.txt Nextflow task trace

Provenance files

All three workflows publish one aggregated ${sample_id}_<workflow>_<timestamp>_provenance.yml file per sample, where <workflow> is short_read, paftools, or nextclade. FASTA mode runs both PAFTOOLS and NEXTCLADE over the same input, so each sample gets two files; the workflow label is what tells them apart.

Every record opens with the pipeline and reference-database header, followed by the SHA-256 of each input file — the paired FASTQs in short-read mode, the consensus FASTA in FASTA mode. A sample whose prepare_consensus or blastn step fails produces no provenance file at all, since those processes run errorStrategy 'ignore' and the collection joins are inner joins.

Note the example below is contrived.

- pipeline_name: BCCDC-PHL/fluscan-nf
  pipeline_version: 0.1.0
  nextflow_session_id: 59fdd919-fc28-4af5-99e0-60355b11807c
  nextflow_run_name: hopeful_bhaskara
  timestamp_analysis_start: 2024-07-15T16:30:11.150887-07:00
- input_filename: sample-01_R1.fastq.gz
  file_type: fastq_input
  sha256: 2c1d5b310a5ca11cc2b1665c094a064fc3aa597e06f392985dac639bd2ab4d81
- input_filename: sample-01_R2.fastq.gz
  file_type: fastq_input
  sha256: 73745eed4badc3594cdd8554e90c75ae4b9b4599ca255064415ded433e598749
- process_name: fastp
  tools:
    - tool_name: fastp
      tool_version: 0.23.2
- process_name: cutadapt
  tools:
    - tool_name: cutadapt
      tool_version: 4.4
- process_name: assembly
  tools:
    - tool_name: spades
      tool_version: v3.15.3
- process_name: lofreq
  tools:
    - tool_name: samtools
      tool_version: 1.18
    - tool_name: lofreq
      tool_version: 2.1.5
    - tool_name: bcftools
      tool_version: 1.18
- process_name: flumut
  tools:
    - tool_name: flumut
      tool_version: 0.8.0

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages