Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 19 additions & 10 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and
`--fastq_input`, `--fasta_input` and `--samplesheet_input` may be set.
`--fastq_input` runs `SHORT_READ` and `--fasta_input` launches both `PAFTOOLS`
and `NEXTCLADE`. For `--samplesheet_input` the header decides: `ID,R1,R2` runs
`SHORT_READ`, `ID,FASTA` runs the two FASTA branches.
`SHORT_READ`, `ID,FASTA` runs the two FASTA branches. An optional `DB` column
gives each FASTA sample its own bundle, so one run can mix subtypes.
- `SHORT_READ` performs read QC/primer trimming, downsampling, assembly,
BLAST-based reference selection, mapping, LoFreq calling, SnpEff annotation,
MNV correction, consensus generation, and FluMut screening.
Expand All @@ -32,13 +33,13 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and
task per sample aligns each segment with minimap2, calls differences with
`paftools.js`, annotates with SnpEff, converts to the common VCF shape, and
corrects MNV codons.
- `NEXTCLADE` batches samples by segment, runs Nextclade once per segment,
- `NEXTCLADE` batches samples by bundle and segment, runs Nextclade once per batch,
splits per-sample results, and converts JSON to the common VCF shape,
annotating substitutions that share a codon with the combined change.
- `--merge_flumut_mutations` is off by default. When enabled in the Nextclade
branch, a run-level map lifts coordinates from FluMut's own reference
branch, a per-bundle map lifts coordinates from FluMut's own reference
(source, the `references`/`annotations` tables of the installed `flumutdb`
sqlite) onto the selected bundle's Nextclade datasets (`--db`, target), then
sqlite) onto each bundle's Nextclade datasets (target), then
`bin/merge_flumut_mutations.py` adds `FLUMUT=` annotations to matching
existing VCF rows. The markers and the source coordinates come from the same
package, so they cannot drift apart.
Expand All @@ -64,8 +65,8 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and
| `docs/vcf_annotation_and_flumut_insertion.md` | Detailed Nextclade VCF and FluMutDB semantics |
| `docs/watchlists.md` | Custom watchlist schema, matching rules, and the HA1/HA numbering trap |
| `assets/watchlists/` | The shipped `*watchlist.csv` files; `--watchlist_dir` defaults here |
| `modules/shared/sample_id.nf` | The only place a consensus or sample sheet `sample_id` is derived; holds the FASTA discovery channel `FASTA_PREP` calls and `samplesheet_channel`, which all three workflows call for `--samplesheet_input`. Changing `clean_sample_id` renames every output |
| `modules/shared/database.nf` | The only place that knows the `--db` bundle layout; `resolve_db` validates it up front. Also holds the canonical segment list, `@Field DB_SEGMENTS`, which it uses to check each segment's dataset |
| `modules/shared/sample_id.nf` | The only place a consensus or sample sheet `sample_id` is derived; holds `fasta_sample_channel` (FASTA input with each sample's bundle), `input_db_paths` (every bundle the input names) and `samplesheet_channel` (SHORT_READ). Changing `clean_sample_id` renames every output |
| `modules/shared/database.nf` | The only place that knows the bundle layout; `resolve_db` validates one bundle up front and `resolve_dbs` every bundle a run uses, keyed by name. Also holds the canonical segment list, `@Field DB_SEGMENTS`, which it uses to check each segment's dataset |
| `scripts/validate_db.py` | Deeper, contents-level bundle check; run after building or moving a bundle |
| `docs/database_bundles.md` | Bundle layout, `db.yml` schema, how to build and validate one |
| `README.md` | User-facing inputs, commands, outputs, and architecture |
Expand Down Expand Up @@ -97,11 +98,16 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and
underscore, and colliding pairs are dropped silently by `.unique`; sample sheet
FASTQ IDs go through `clean_sample_id` instead. Avoid introducing or silently
accepting colliding names.
- All reference data comes from one bundle directory, `--db`, in the fixed
- Each sample's reference data comes from one bundle directory, in the fixed
layout `blastn_db.fa` + `nextclade/` + `snpeff/{snpEff.config,<genome>/}` plus an
optional `db.yml`. Resolve it through `resolve_db()` in
`modules/shared/database.nf` rather than composing paths at the call site;
that function is the only thing that should know the layout.
that function is the only thing that should know the layout. SHORT_READ uses
`--db`. In FASTA mode the bundle is the sample sheet's `DB` cell, else `--db`,
so a run can hold several; their names (`db.yml` `name`, else the directory)
must differ, because they key the per-bundle channels and the run-level
outputs `nextclade/<db>/`, `nextclade/status/<db>/` and
`aggregate/reference_remap/<db>/`.
- The nucleotide reference FASTA used for BLAST has headers
`accession|subtype|segment`.
- Nextclade datasets live in uppercase per-segment subdirectories (`HA/`, `PB2/`,
Expand Down Expand Up @@ -130,9 +136,12 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and
explicitly unless a local config supplies it.
- `--samplesheet_input` is parsed eagerly by `samplesheet_channel`, so column,
blank-cell, ID and file-existence errors all fail before any task is submitted.
`FASTA_PREP` chooses between it and `consensus_fasta_channel`. `run_name` is the sheet's filename without `.csv`.
`fasta_sample_channel` chooses between it and `consensus_fasta_channel`, and
`FASTA_PREP` publishes `aggregate/sample_ids.csv` (`input_id,sample_id,db`), the
only record of how an input ID was cleaned. `run_name` is the sheet's filename without `.csv`.
- Channel tuple keys are part of the design. Preserve `sample_id` and `segment`
through `join`, `combine`, `transpose`, and batching operations, and comment
through `join`, `combine`, `transpose`, and batching operations (and in FASTA
mode the bundle name, which keys per-bundle tasks and provenance), and comment
non-obvious tuple shapes next to the transformation.
- Preserve `publishDir` destinations and filenames when refactoring. Outputs
are primarily `${outdir}/${sample_id}/...`, with run-level data under
Expand Down
28 changes: 20 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ flowchart TD
nc2vcf[nextclade_to_freebayes_vcf.py]
nc_final[Final NEXTCLADE VCFs]
coord_map[coordinate_conversion: build_reference_coordinate_map.py]
coord_final[aggregate/reference_remap]
coord_final[aggregate/reference_remap, per bundle]
flumut_merge[merge_flumut_mutations: merge_flumut_mutations.py]
flumut_annotated[FluMutDB-annotated VCFs]

Expand Down Expand Up @@ -140,12 +140,23 @@ SAMPLE-01,/path/to/SAMPLE-01.fasta
SAMPLE-02,/path/to/SAMPLE-02.fasta
```

runs PAFTOOLS and NEXTCLADE. Other columns are ignored.
runs PAFTOOLS and NEXTCLADE. A FASTA sheet may add a `DB` column naming each sample's
reference bundle, so one run can mix subtypes:

```csv
ID,FASTA,DB
SAMPLE-01,/path/to/SAMPLE-01.fasta,/path/to/refs/gsgd
SAMPLE-02,/path/to/SAMPLE-02.fasta,/path/to/refs/wisconsin588
```

A blank `DB` cell falls back to `--db`; see [docs/database_bundles.md](docs/database_bundles.md#several-bundles-in-one-run). Other columns are ignored.

- Each `ID` is cleaned by the same rule as a FASTA filename (see the note under Usage), so `24-1234-IP7-SAMPLE_01` is reported as `SAMPLE-01`. After cleaning, an ID may contain only letters, digits, `.` and `-`.
- Relative paths resolve against the directory `nextflow run` is launched from; absolute paths are safer.
- The run stops before any task starts if a column or value is missing, a file does not exist, or two IDs clean to the same sample name.
- `run_name`, which prefixes the run-level reports, is the sample sheet's filename without `.csv`.
- `aggregate/sample_ids.csv` maps each input ID (or FASTA filename) to its cleaned `sample_id` and bundle name.
- The short-read workflow takes a single `--db` and rejects a sheet with a `DB` column.
- Give only one of `--fastq_input`, `--fasta_input` and `--samplesheet_input`.

## Reference databases
Expand Down Expand Up @@ -289,13 +300,14 @@ The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder `FORMAT/GQ` as
|------|-------------|
| `${run_name}_fluscan-nf_multiqc_report.html` | MultiQC report summarising fastp, cutadapt, and FastQC results for all samples (short-read workflow only) |
| `aggregate/paftools_qc_summary.tsv` | Run-level tabular summary of PAFTOOLS alignment coverage across all FASTA samples and segments |
| `aggregate/sample_ids.csv` | `input_id,sample_id,db`: each input's sample sheet `ID` (or FASTA filename), its cleaned `sample_id`, and the bundle it ran against (FASTA mode) |
| `aggregate/consensus_qc_summary.tsv` | Every `*_qc_report.tsv` row from consensus preparation, concatenated across the run. A sample rejected at preparation — duplicate segment, or headers naming more than one sample — contributes no rows and is reported only in the Nextflow log |
| `aggregate/reference_remap/coordinate_map.json` | Primary machine-readable nucleotide/amino-acid coordinate map lifting flumut's reference (source) onto the selected Nextclade reference dataset (target) (NEXTCLADE workflow only) |
| `aggregate/reference_remap/coordinate_map.nt.tsv` | Flat per-position nucleotide coordinate map (source/flumut position/base -> target/nextclade position/base, per segment) |
| `aggregate/reference_remap/coordinate_map.aa.tsv` | Flat per-residue amino-acid coordinate map (source/flumut position/residue -> target/nextclade position/residue, per segment/gene) |
| `aggregate/reference_remap/coordinate_map.qc.tsv` | Per-segment/per-gene alignment QC: identity, coverage, the number of co-optimal alignments the map's gap placement was chosen from, and status (`ok`/`warning`/`missing_in_source`/`extra_in_source`/`stale_annotation`) |
| `aggregate/reference_remap/flumut_alignment.fasta` | Target/source (nextclade/flumut) nucleotide alignment blocks, one per segment |
| `aggregate/reference_remap/flumut_aa_alignment.fasta` | Target/source (nextclade/flumut) amino-acid alignment blocks, one per gene |
| `aggregate/reference_remap/<db>/coordinate_map.json` | Primary machine-readable nucleotide/amino-acid coordinate map lifting flumut's reference (source) onto bundle `<db>`'s Nextclade reference datasets (target), one directory per bundle (NEXTCLADE workflow only) |
| `aggregate/reference_remap/<db>/coordinate_map.nt.tsv` | Flat per-position nucleotide coordinate map (source/flumut position/base -> target/nextclade position/base, per segment) |
| `aggregate/reference_remap/<db>/coordinate_map.aa.tsv` | Flat per-residue amino-acid coordinate map (source/flumut position/residue -> target/nextclade position/residue, per segment/gene) |
| `aggregate/reference_remap/<db>/coordinate_map.qc.tsv` | Per-segment/per-gene alignment QC: identity, coverage, the number of co-optimal alignments the map's gap placement was chosen from, and status (`ok`/`warning`/`missing_in_source`/`extra_in_source`/`stale_annotation`) |
| `aggregate/reference_remap/<db>/flumut_alignment.fasta` | Target/source (nextclade/flumut) nucleotide alignment blocks, one per segment |
| `aggregate/reference_remap/<db>/flumut_aa_alignment.fasta` | Target/source (nextclade/flumut) amino-acid alignment blocks, one per gene |
| `${run_name}_fluscan-nf_nextflow_report.html` | Nextflow execution report |
| `${run_name}_fluscan-nf_nextflow_timeline.html` | Nextflow execution timeline |
| `${run_name}_fluscan-nf_nextflow_trace.txt` | Nextflow task trace |
Expand Down
27 changes: 23 additions & 4 deletions docs/database_bundles.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# Reference database bundles

Every reference the pipeline needs for one reference set lives in a single directory — a
*bundle* — passed as `--db`. The FluMut coordinate reference used by the optional merge
is not a bundle: it is read from the `flumutdb` conda package.
*bundle* — passed as `--db`. In FASTA mode a sample sheet's `DB` column can instead give
each sample its own bundle (see [Several bundles in one run](#several-bundles-in-one-run)).
The FluMut coordinate reference used by the optional merge is not a bundle: it is read from
the `flumutdb` conda package.

```
nextflow run BCCDC-PHL/fluscan-nf \
Expand Down Expand Up @@ -34,7 +36,24 @@ nests them one level deeper under `data/`; a bundle does not, because `snpeff/`
already the per-bundle directory that level provided.

The filenames are fixed and not configurable. `modules/shared/database.nf` is the only
place that knows this layout; `--db` is the only path you supply.
place that knows this layout; the bundle root is the only path you supply.

### Several bundles in one run

In FASTA mode a sample sheet may carry a `DB` column holding a bundle root per row:

```csv
ID,FASTA,DB
SAMPLE-01,/path/to/SAMPLE-01.fasta,/path/to/refs/gsgd
SAMPLE-02,/path/to/SAMPLE-02.fasta,/path/to/refs/wisconsin588
```

A blank `DB` cell, or a sheet without the column, falls back to `--db`. Every bundle is
validated before any task runs, and each sample is analysed against its own bundle only.
Bundle names (`db.yml` `name`, else the directory name) must differ within a run, because
run-level outputs are filed under them: `nextclade/<name>/`, `nextclade/status/<name>/`
and `aggregate/reference_remap/<name>/`. The short-read workflow takes one `--db` and
rejects a sheet with a `DB` column.

### Bundles may be partial

Expand Down Expand Up @@ -74,7 +93,7 @@ Quote `version`. Unquoted `2025-07-07` parses as a *date*, not a string, in both
YAML reader the pipeline uses and the one that reads the provenance file back.

`nextclade_ref_name` is what appears as `target_reference` in
`aggregate/reference_remap/coordinate_map.json`. It is separate from `name` so a bundle
`aggregate/reference_remap/<name>/coordinate_map.json`. It is separate from `name` so a bundle
can be renamed or relocated on disk without changing labels already written into
published coordinate maps.

Expand Down
23 changes: 14 additions & 9 deletions main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,14 @@ include { SHORT_READ } from './workflows/short-read.nf'
include { FASTA_PREP } from './workflows/fasta-prep.nf'
include { PAFTOOLS } from './workflows/paftools.nf'
include { NEXTCLADE } from './workflows/nextclade.nf'
include { resolve_db } from './modules/shared/database.nf'
include { db_summary } from './modules/shared/database.nf'
include { resolve_dbs } from './modules/shared/database.nf'
include { db_summary } from './modules/shared/database.nf'
include { input_db_paths } from './modules/shared/sample_id.nf'

// Resolved with no required components: this only checks that --db points at something
// that exists, so a typo fails before the banner. Each workflow re-resolves with the
// components it actually consumes.
def db = resolve_db(params.db, [])
// Resolved with no required components: this only checks that every bundle the input names
// exists, so a typo fails before the banner. Each workflow re-resolves with the components
// it actually consumes.
def dbs = resolve_dbs(input_db_paths(), [])

// prints to the screen and to the log
log.info """
Expand All @@ -26,7 +27,7 @@ log.info """
fastqInputDir : ${params.fastq_input}
fastaInputDir : ${params.fasta_input}
samplesheet : ${params.samplesheet_input}
database : ${db_summary(db)}
database : ${dbs.values().collect { db_summary(it) }.join('\n ')}
primers : ${params.primers}
outdir : ${params.outdir}
pipeline run name : ${workflow.runName}
Expand All @@ -50,12 +51,16 @@ workflow {
? (file(params.samplesheet_input, checkIfExists: true).splitCsv(strip: true)[0] ?: [])
: []

if ('R1' in header && 'DB' in header) {
error "ERROR: the short-read workflow takes one --db bundle; remove the DB column from ${params.samplesheet_input}."
}

if (params.fastq_input != 'NO_FILE' || 'R1' in header) {
SHORT_READ()
} else if (params.fasta_input != 'NO_FILE' || 'FASTA' in header) {
FASTA_PREP()
PAFTOOLS(FASTA_PREP.out.cleaned, FASTA_PREP.out.provenance)
NEXTCLADE(FASTA_PREP.out.cleaned, FASTA_PREP.out.provenance)
PAFTOOLS(FASTA_PREP.out.cleaned, FASTA_PREP.out.db, FASTA_PREP.out.provenance)
NEXTCLADE(FASTA_PREP.out.cleaned, FASTA_PREP.out.db, FASTA_PREP.out.provenance)
} else if (params.samplesheet_input != 'NO_FILE') {
error "ERROR: --samplesheet_input needs an ID,R1,R2 header (short-read workflow) or an ID,FASTA header (PAFTOOLS/NEXTCLADE workflow)."
} else {
Expand Down
15 changes: 8 additions & 7 deletions modules/nextclade/coordinate_conversion.nf
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// nextclade/coordinate_conversion.nf - Build a FluMut->Nextclade reference coordinate map.
//
// Runs once per pipeline invocation, not per sample. Translates nucleotide and amino-acid
// Runs once per bundle, not per sample. Translates nucleotide and amino-acid
// indices from flumut's fixed reference (source, read from the installed flumutdb package)
// onto the selected Nextclade dataset (target), so merge_flumut_mutations can place FluMutDB
// positions in Nextclade coordinates. Idempotent: the same flumutdb and dataset always give
Expand All @@ -12,16 +12,17 @@ process coordinate_conversion {

conda "${projectDir}/environments/fluscan.yml"

publishDir "${params.outdir}/aggregate", pattern: "reference_remap", mode: 'copy'
publishDir "${params.outdir}/aggregate/reference_remap", pattern: "reference_remap", mode: 'copy',
saveAs: { db_name }

input:
tuple val(target_name), path(target_dataset_dir, stageAs: 'target_dataset')
tuple val(db_name), val(target_name), path(target_dataset_dir, stageAs: 'target_dataset')

output:
path("reference_remap"), emit: map_dir
path("reference_remap/coordinate_map.json"), emit: json
path("reference_remap/coordinate_map.*.json"), emit: json_shards
path("reference_remap/coordinate_conversion_provenance.yml"), emit: provenance
tuple val(db_name), path("reference_remap"), emit: map_dir
tuple val(db_name), path("reference_remap/coordinate_map.json"), emit: json
tuple val(db_name), path("reference_remap/coordinate_map.*.json"), emit: json_shards
tuple val(db_name), path("reference_remap/coordinate_conversion_provenance.yml"), emit: provenance

script:
"""
Expand Down
8 changes: 4 additions & 4 deletions modules/nextclade/merge_flumut_mutations.nf
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

process merge_flumut_mutations {

tag { segment }
tag { "${db_name}/${segment}" }

conda "${projectDir}/environments/fluscan.yml"

Expand All @@ -27,11 +27,11 @@ process merge_flumut_mutations {
}

input:
tuple val(segment), path(vcfs, stageAs: 'vcf/*'), path(coord_json)
tuple val(db_name), val(segment), path(vcfs, stageAs: 'vcf/*'), path(coord_json)

output:
tuple val(segment), path("*_flumut_annotated.vcf"), emit: vcf
tuple val(segment), path("${segment}_merge_flumut_provenance.yml"), emit: provenance
tuple val(db_name), val(segment), path("*_flumut_annotated.vcf"), emit: vcf
tuple val(db_name), val(segment), path("${segment}_merge_flumut_provenance.yml"), emit: provenance

script:
"""
Expand Down
Loading
Loading