Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,10 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and
package, so they cannot drift apart.
- `mutation_watch` scans each final VCF against the custom watchlists in
`--watchlist_dir` (default `assets/watchlists/`) and publishes one
`*_watchlist_hits.tsv` per final VCF and watchlist (header-only when nothing
matches), named by segment from the VCF filename. It runs in all
three workflows and is independent of `--merge_flumut_mutations`.
`watchlist/<list>/*_watchlist_hits.tsv` per final VCF and watchlist (header-only
when nothing matches), named by segment from the VCF filename and tagged
`pf`/`nc`/`sr` by workflow, since FASTA mode puts both branches in one list dir.
It runs in all three workflows and is independent of `--merge_flumut_mutations`.

## Repository map

Expand Down Expand Up @@ -108,8 +109,7 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and
`--db`. In FASTA mode the bundle is the sample sheet's `DB` cell, else `--db`,
so a run can hold several; their names (`db.yml` `name`, else the directory)
must differ, because they key the per-bundle channels and the run-level
outputs `nextclade/<db>/`, `nextclade/status/<db>/` and
`aggregate/reference_remap/<db>/`.
outputs under `aggregate/nextclade/<db>/`.
- The nucleotide reference FASTA used for BLAST has headers
`accession|subtype|segment`.
- Nextclade datasets live in uppercase per-segment subdirectories (`HA/`, `PB2/`,
Expand Down
37 changes: 19 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ flowchart TD
nc2vcf[nextclade_to_freebayes_vcf.py: QC + VCF]
nc_final[Final NEXTCLADE VCFs]
coord_map[coordinate_conversion: build_reference_coordinate_map.py]
coord_final[aggregate/reference_remap, per bundle]
coord_final[aggregate/nextclade/db/reference_remap]
flumut_merge[merge_flumut_mutations: merge_flumut_mutations.py]
flumut_annotated[FluMutDB-annotated VCFs]

Expand Down Expand Up @@ -238,16 +238,16 @@ Sample-level outputs are written to one folder per sample, and some workflows al
| `*.fastp.json` | Per-sample fastp JSON report |
| `*_contigs.fasta` | De-novo assembled contigs from SPAdes |
| `*_consensus.fasta` | Multi-FASTA containing the consensus sequences for each segment |
| `*_ref.fasta` | Multi-FASTA containing the optimal reference sequences selected for each segment |
| `*_blastn.tsv` | BLAST search results of assembled contigs against the reference database |
| `*_blastn_filter.tsv` | Filtered BLAST search results |
| `blast/*_ref.fasta` | Multi-FASTA containing the optimal reference sequences selected for each segment |
| `blast/*_blastn.tsv` | BLAST search results of assembled contigs against the reference database |
| `blast/*_blastn_filter.tsv` | Filtered BLAST search results |
| `*.sort.filter.bam` | Sorted, filtered alignment BAM used for downstream calling and coverage calculations |
| `*_coverage.png` | Line plots describing mapping coverage across all 8 flu segments |
| `*_pileup.tsv` | Pileup TSV with raw counts of all 4 nucleotides and gaps |
| `*_depth.tsv` | Per-position alignment depth (samtools depth) |
| `*_low_coverage.bed` | Low-coverage regions masked during consensus generation |
| `short_read/*_final.vcf` | Per-segment final VCF after SnpEff annotation and MNV codon correction |
| `short_read/*_watchlist_hits.tsv` | *Optional* (`--watchlist_dir`): watchlist substitutions found in the final VCF |
| `watchlist/<list>/*_<list>_sr_watchlist_hits.tsv` | *Optional* (`--watchlist_dir`): watchlist substitutions found in the final VCF, one file per segment |
| `flumut/*_flumut_markers.tsv` | FluMut table of molecular markers identified in the consensus sequences |
| `flumut/*_flumut_mutations.tsv` | FluMut long-format table of amino acid mutations at positions of interest |
| `flumut/*_flumut_mutations_wide.tsv` | FluMut wide-format table of amino acid mutations at positions of interest |
Expand All @@ -261,13 +261,13 @@ The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder `FORMAT/GQ` as
| File | Description |
|------|-------------|
| `${sample_id}.fasta` | Validated consensus FASTA with headers normalised to `{sample_id}_{SEGMENT}` |
| `*_qc_report.tsv` | QC report from consensus preparation |
| `*_ref.fasta` | Multi-FASTA containing the optimal reference sequences selected for each segment |
| `*_blastn.tsv` | BLAST search results against the reference database |
| `*_blastn_filter.tsv` | Filtered BLAST search results |
| `qc/*_qc_report.tsv` | QC report from consensus preparation |
| `blast/*_ref.fasta` | Multi-FASTA containing the optimal reference sequences selected for each segment |
| `blast/*_blastn.tsv` | BLAST search results against the reference database |
| `blast/*_blastn_filter.tsv` | Filtered BLAST search results |
| `qc/*_coverage_qc.tsv` | Per-segment PAFTOOLS consensus coverage QC with coverage percentage and PASS/WARN/FAIL status. A segment with no BLAST reference, or that minimap2 cannot align, is FAIL and has no VCF |
| `paftools/*_final.vcf` | Per-segment final VCF after `paftools.js`, SnpEff, freebayes-style conversion, and MNV codon correction |
| `paftools/*_watchlist_hits.tsv` | *Optional* (`--watchlist_dir`): watchlist substitutions found in the final VCF |
| `watchlist/<list>/*_<list>_pf_watchlist_hits.tsv` | *Optional* (`--watchlist_dir`): watchlist substitutions found in the final VCF, one file per segment |
| `flumut/*_flumut_markers.tsv` | FluMut table of molecular markers identified in the consensus sequences |
| `flumut/*_flumut_mutations.tsv` | FluMut long-format table of amino acid mutations at positions of interest |
| `flumut/*_flumut_mutations_wide.tsv` | FluMut wide-format table of amino acid mutations at positions of interest |
Expand All @@ -281,7 +281,7 @@ The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder `FORMAT/GQ` as
| `nextclade/*_final.vcf` | Per-segment final VCF from the Nextclade-to-VCF conversion, written only for segments that do not FAIL QC (see `aggregate/nextclade_qc_summary.tsv`). Rows in a codon holding more than one substitution carry the combined change, plus `MNV_CORRECTED` and `MNV_ORIGINAL_HGVS_P` where it differs from the call before codon correction (Nextclade's, otherwise the single-base change) |
| `nextclade/no_mature_peptide/*_nc_nomat_final.vcf` | *Optional* (`--emit_no_mature_vcf`): the same records annotated without mature peptides, so HA carries only its HA0 `ANN=` entry and no HA1/HA2 ones |
| `nextclade/flumut_annotated/*_flumut_annotated.vcf` | *Optional* (`--merge_flumut_mutations`): the same VCF with a `FLUMUT=` INFO field (`Protein\|AA_Pos\|Alt_AA\|Mutation_Name\|DirectEffect\|AssociatedEffect`) on rows whose amino-acid change matches a curated FluMutDB marker |
| `nextclade/*_watchlist_hits.tsv` | *Optional* (`--watchlist_dir`): watchlist substitutions found in the final VCF. Independent of `--merge_flumut_mutations`: `FLUMUT=` is a curated annotation written *into* the VCF, this is a custom list scanned *out* of it |
| `watchlist/<list>/*_<list>_nc_watchlist_hits.tsv` | *Optional* (`--watchlist_dir`): watchlist substitutions found in the final VCF, one file per segment, beside the PAFTOOLS (`_pf_`) ones. Independent of `--merge_flumut_mutations`: `FLUMUT=` is a curated annotation written *into* the VCF, this is a custom list scanned *out* of it |
| `flumut/*_flumut_markers.tsv` | FluMut table of molecular markers identified in the consensus sequences |
| `flumut/*_flumut_mutations.tsv` | FluMut long-format table of amino acid mutations at positions of interest |
| `flumut/*_flumut_mutations_wide.tsv` | FluMut wide-format table of amino acid mutations at positions of interest |
Expand All @@ -292,17 +292,18 @@ The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder `FORMAT/GQ` as

| File | Description |
|------|-------------|
| `${run_name}_fluscan-nf_multiqc_report.html` | MultiQC report summarising fastp, cutadapt, and FastQC results for all samples (short-read workflow only) |
| `aggregate/${run_name}_fluscan-nf_multiqc_report.html` | MultiQC report summarising fastp, cutadapt, and FastQC results for all samples (short-read workflow only) |
| `aggregate/paftools_qc_summary.tsv` | Run-level tabular summary of PAFTOOLS alignment coverage across all FASTA samples and segments |
| `aggregate/nextclade_qc_summary.tsv` | One row per sample and segment Nextclade saw (NEXTCLADE workflow). `status` is FAIL, with no VCF written, when Nextclade could not align the segment (`not_aligned`), reports a frameshift (`frameshift`), or scores its stop codons `bad` (`stop_codons`); WARN when `coverage_pct` is below `--consensus_coverage_pass_pct` (default 90); otherwise PASS. `coverage_pct` is the share of the reference covered by called (non-N) bases, so a truncated or N-heavy segment WARNs but keeps its VCF. `identity_pct` is exact matches over the called bases plus insertions; it has no threshold and is low for a segment from a different lineage than the reference |
| `aggregate/sample_ids.csv` | `input_id,sample_id,db`: each input's sample sheet `ID` (or FASTA filename), its cleaned `sample_id`, and the bundle it ran against (FASTA mode) |
| `aggregate/consensus_qc_summary.tsv` | Every `*_qc_report.tsv` row from consensus preparation, concatenated across the run. A sample rejected at preparation — duplicate segment, or headers naming more than one sample — contributes no rows and is reported only in the Nextflow log |
| `aggregate/reference_remap/<db>/coordinate_map.json` | Primary machine-readable nucleotide/amino-acid coordinate map lifting flumut's reference (source) onto bundle `<db>`'s Nextclade reference datasets (target), one directory per bundle (NEXTCLADE workflow only) |
| `aggregate/reference_remap/<db>/coordinate_map.nt.tsv` | Flat per-position nucleotide coordinate map (source/flumut position/base -> target/nextclade position/base, per segment) |
| `aggregate/reference_remap/<db>/coordinate_map.aa.tsv` | Flat per-residue amino-acid coordinate map (source/flumut position/residue -> target/nextclade position/residue, per segment/gene) |
| `aggregate/reference_remap/<db>/coordinate_map.qc.tsv` | Per-segment/per-gene alignment QC: identity, coverage, the number of co-optimal alignments the map's gap placement was chosen from, and status (`ok`/`warning`/`missing_in_source`/`extra_in_source`/`stale_annotation`) |
| `aggregate/reference_remap/<db>/flumut_alignment.fasta` | Target/source (nextclade/flumut) nucleotide alignment blocks, one per segment |
| `aggregate/reference_remap/<db>/flumut_aa_alignment.fasta` | Target/source (nextclade/flumut) amino-acid alignment blocks, one per gene |
| `aggregate/nextclade/<db>/<SEGMENT>/batch.*` | Raw `nextclade run --output-all` output for one bundle and segment across every sample in the run (NEXTCLADE workflow) |
| `aggregate/nextclade/<db>/reference_remap/coordinate_map.json` | *Optional* (`--merge_flumut_mutations`): primary machine-readable nucleotide/amino-acid coordinate map lifting flumut's reference (source) onto bundle `<db>`'s Nextclade reference datasets (target) |
| `aggregate/nextclade/<db>/reference_remap/coordinate_map.nt.tsv` | Flat per-position nucleotide coordinate map (source/flumut position/base -> target/nextclade position/base, per segment) |
| `aggregate/nextclade/<db>/reference_remap/coordinate_map.aa.tsv` | Flat per-residue amino-acid coordinate map (source/flumut position/residue -> target/nextclade position/residue, per segment/gene) |
| `aggregate/nextclade/<db>/reference_remap/coordinate_map.qc.tsv` | Per-segment/per-gene alignment QC: identity, coverage, the number of co-optimal alignments the map's gap placement was chosen from, and status (`ok`/`warning`/`missing_in_source`/`extra_in_source`/`stale_annotation`) |
| `aggregate/nextclade/<db>/reference_remap/flumut_alignment.fasta` | Target/source (nextclade/flumut) nucleotide alignment blocks, one per segment |
| `aggregate/nextclade/<db>/reference_remap/flumut_aa_alignment.fasta` | Target/source (nextclade/flumut) amino-acid alignment blocks, one per gene |
| `${run_name}_fluscan-nf_nextflow_report.html` | Nextflow execution report |
| `${run_name}_fluscan-nf_nextflow_timeline.html` | Nextflow execution timeline |
| `${run_name}_fluscan-nf_nextflow_trace.txt` | Nextflow task trace |
Expand Down
5 changes: 2 additions & 3 deletions docs/database_bundles.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,7 @@ SAMPLE-02,/path/to/SAMPLE-02.fasta,/path/to/refs/wisconsin588
A blank `DB` cell, or a sheet without the column, falls back to `--db`. Every bundle is
validated before any task runs, and each sample is analysed against its own bundle only.
Bundle names (`db.yml` `name`, else the directory name) must differ within a run, because
run-level outputs are filed under them: `nextclade/<name>/`, `nextclade/status/<name>/`
and `aggregate/reference_remap/<name>/`. The short-read workflow takes one `--db` and
run-level outputs are filed under them in `aggregate/nextclade/<name>/`. The short-read workflow takes one `--db` and
rejects a sheet with a `DB` column.

### Bundles may be partial
Expand Down Expand Up @@ -93,7 +92,7 @@ Quote `version`. Unquoted `2025-07-07` parses as a *date*, not a string, in both
YAML reader the pipeline uses and the one that reads the provenance file back.

`nextclade_ref_name` is what appears as `target_reference` in
`aggregate/reference_remap/<name>/coordinate_map.json`. It is separate from `name` so a bundle
`aggregate/nextclade/<name>/reference_remap/coordinate_map.json`. It is separate from `name` so a bundle
can be renamed or relocated on disk without changing labels already written into
published coordinate maps.

Expand Down
3 changes: 2 additions & 1 deletion docs/watchlists.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,8 @@ would be a wrong answer with nothing to show it was wrong.

## Output

`${outdir}/${sample_id}/${workflow}/${sample_id}_${segment}_${list}_watchlist_hits.tsv`:
`${outdir}/${sample_id}/watchlist/${list}/${sample_id}_${segment}_${list}_${wf}_watchlist_hits.tsv`,
where `${wf}` is `pf` (PAFTOOLS), `nc` (NEXTCLADE) or `sr` (SHORT_READ):

```
sample_id segment gene position ref_aa alt_aa mutation chrom pos hgvs_p note
Expand Down
3 changes: 1 addition & 2 deletions modules/nextclade/coordinate_conversion.nf
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,7 @@ process coordinate_conversion {

conda "${projectDir}/environments/fluscan.yml"

publishDir "${params.outdir}/aggregate/reference_remap", pattern: "reference_remap", mode: 'copy',
saveAs: { db_name }
publishDir "${params.outdir}/aggregate/nextclade/${db_name}", pattern: "reference_remap", mode: 'copy'

input:
tuple val(db_name), val(target_name), path(target_dataset_dir, stageAs: 'target_dataset')
Expand Down
3 changes: 1 addition & 2 deletions modules/nextclade/merge_flumut_mutations.nf
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,8 @@ process merge_flumut_mutations {
publishDir "${params.outdir}", mode: 'copy',
saveAs: { filename ->
if (filename.endsWith('_flumut_annotated.vcf')) {
def wf = task.ext.workflow ?: 'nextclade'
def sample_id = filename.split('__')[0]
return "${sample_id}/${wf}/flumut_annotated/${filename}"
return "${sample_id}/nextclade/flumut_annotated/${filename}"
}
// Provenance is not published from here: collect_provenance concatenates it
// with the rest of each sample's records and publishes that single file.
Expand Down
10 changes: 3 additions & 7 deletions modules/nextclade/nextclade.nf
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,12 @@ process nextclade_run_batch {
publishDir "${params.outdir}", mode: 'copy',
saveAs: { filename ->
if (filename.endsWith('_final.vcf')) {
def workflow = task.ext.workflow ?: 'unknown_workflow'
def sample_id = filename.split('__')[0]
def subdir = filename.endsWith('_nc_nomat_final.vcf') ? 'no_mature_peptide/' : ''
return "${sample_id}/${workflow}/${subdir}${filename}"
return "${sample_id}/nextclade/${subdir}${filename}"
}
if (filename =~ /nc_output.*/) {
def newname = filename.replaceAll(/nc_output/, "${segment}_output")
return "nextclade/${db_name}/${newname}"
}
return null
// Not named ${segment} in the work dir: the staged dataset already holds that name.
return filename == 'nc_output' ? "aggregate/nextclade/${db_name}/${segment}" : null
}

input:
Expand Down
4 changes: 1 addition & 3 deletions modules/shared/blast.nf
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,7 @@ process blastn {

conda "${projectDir}/environments/fluscan.yml"

publishDir "${params.outdir}/${sample_id}/", pattern: "${sample_id}*blastn.tsv" , mode:'copy'
publishDir "${params.outdir}/${sample_id}/", pattern: "${sample_id}*filter.tsv" , mode:'copy'
publishDir "${params.outdir}/${sample_id}/", pattern: "${sample_id}*fasta" , mode:'copy'
publishDir "${params.outdir}/${sample_id}/blast", pattern: "${sample_id}*.{tsv,fasta}", mode:'copy'

input:
tuple val(sample_id), path(query), path(blast_db)
Expand Down
2 changes: 1 addition & 1 deletion modules/shared/consensus_qc.nf
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ process prepare_consensus {
conda "${projectDir}/environments/fluscan.yml"

publishDir "${params.outdir}/${sample_id}", pattern: "*.fasta", mode: 'copy'
publishDir "${params.outdir}/${sample_id}", pattern: "*_qc_report.tsv", mode: 'copy'
publishDir "${params.outdir}/${sample_id}/qc", pattern: "*_qc_report.tsv", mode: 'copy'

input:
tuple val(sample_id), path(fasta_input, stageAs: 'input/*')
Expand Down
Loading
Loading