From 8fccdd4578595cc33e6361dbcf2d8f7405d446d0 Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Tue, 29 Sep 2026 16:48:27 -0400 Subject: [PATCH 01/11] Correct MNV codons in the Nextclade VCF converter The converter now translates each codon with all of its substitutions applied and writes MNV_CORRECTED/MNV_ORIGINAL_HGVS_P itself, so the Nextclade branch no longer runs fix_mnv_codons.py. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 18 ++- README.md | 13 +- bin/nextclade_to_freebayes_vcf.py | 137 +++++++++++--------- docs/vcf_annotation_and_flumut_insertion.md | 33 ++--- modules/nextclade/nextclade.nf | 20 ++- modules/shared/post_process.nf | 73 +---------- nextflow.config | 6 - tests/test_nextclade_to_freebayes_vcf.py | 63 ++++++++- workflows/nextclade.nf | 33 ++--- 9 files changed, 194 insertions(+), 202 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 118915d..c544893 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,8 +31,8 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and minimap2, calls differences with `paftools.js`, sorts and annotates VCFs, converts them to the common VCF shape, and corrects MNV codons. - `NEXTCLADE` batches samples by segment, runs Nextclade once per segment, - splits per-sample results, converts JSON to the common VCF shape, and - corrects MNV codons. + splits per-sample results, and converts JSON to the common VCF shape, + annotating substitutions that share a codon with the combined change. - `--merge_flumut_mutations` is off by default. When enabled in the Nextclade branch, a run-level map lifts coordinates from FluMut's own reference (source, the `references`/`annotations` tables of the installed `flumutdb` @@ -145,8 +145,12 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and - Consensus-derived Nextclade VCFs are deliberately “freebayes-shaped.” Depth, quality, and genotype fields are placeholders; biological meaning is carried by coordinates, alleles, and the 16-field SnpEff-compatible `ANN=`. -- Individual SNVs in one codon can have incorrect amino-acid annotations until - `bin/fix_mnv_codons.py` runs. Final VCFs must remain downstream of that step. +- In the SnpEff-annotated PAFTOOLS and SHORT_READ VCFs, individual SNVs in one + codon can have incorrect amino-acid annotations until `bin/fix_mnv_codons.py` + runs; their final VCFs must remain downstream of that step. The Nextclade + converter translates each codon with all its substitutions applied (and keeps + Nextclade's own call where `nucToAaMuts` has one), writing the same + `MNV_CORRECTED`/`MNV_ORIGINAL_HGVS_P` fields itself. - `bin/merge_flumut_mutations.py` is the only implementation. It may add `FLUMUT=` to matching rows, but must not add, remove, reorder, or otherwise rewrite variant rows. The earlier implementation that inserted synthetic rows @@ -156,9 +160,9 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and gone too. - Rows flagged `MNV_UNRESOLVED` by `bin/fix_mnv_codons.py` are skipped, not matched: their HGVS.p is known to be wrong, so a marker hit there would be a - false positive. Every member row of a *resolved* MNV group carries the same - corrected HGVS.p and so the same `FLUMUT=` entry — downstream marker counts - must de-duplicate. + false positive. Every row of a codon holding more than one substitution + carries the same combined HGVS.p and so the same `FLUMUT=` entry — downstream + marker counts must de-duplicate. - **Watchlist hits never enter the VCF.** `FLUMUT=` comes from a versioned database, so the same consensus always yields the same annotation; a watchlist is a CSV we edit, and merging its hits in would make the same consensus diff --git a/README.md b/README.md index 1a28778..2bb1f37 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # fluscan-nf This is a Nextflow pipeline for Influenza A mutation analysis from either short-read FASTQ data or pre-built consensus FASTA files. -The short-read workflow performs trimming, assembly, mapping, LoFreq variant calling, SnpEff annotation, and consensus generation. In FASTA mode (`--fasta_input`), the pipeline currently launches both a PAFTOOLS branch and a NEXTCLADE branch: PAFTOOLS validates the supplied FASTA, selects a subtype-matched reference with BLAST, aligns per segment with `minimap2`, calls differences with `paftools.js`, sorts with `bcftools`, annotates with SnpEff, and fixes multi-nucleotide variant codons; NEXTCLADE validates the same input, batches sequences by segment for `nextclade run`, converts the results to VCF, and applies the same MNV codon correction step. +The short-read workflow performs trimming, assembly, mapping, LoFreq variant calling, SnpEff annotation, and consensus generation. In FASTA mode (`--fasta_input`), the pipeline currently launches both a PAFTOOLS branch and a NEXTCLADE branch: PAFTOOLS validates the supplied FASTA, selects a subtype-matched reference with BLAST, aligns per segment with `minimap2`, calls differences with `paftools.js`, sorts with `bcftools`, annotates with SnpEff, and fixes multi-nucleotide variant codons; NEXTCLADE validates the same input, batches sequences by segment for `nextclade run`, and converts the results to VCF, annotating substitutions that share a codon with their combined codon change. ## Analyses @@ -36,8 +36,7 @@ The short-read workflow performs trimming, assembly, mapping, LoFreq variant cal - Per-segment batching across samples - Per-segment `nextclade run` against built-in datasets - Per-sample JSON/TSV splitting from the batch results -- Conversion to freebayes-style VCF -- MNV codon correction +- Conversion to freebayes-style VCF, with substitutions that share a codon annotated as one codon change - Detection of influenza molecular markers with [FluMut](https://izsvenezie-virology.github.io/FluMut/docs/usage/usage-cli) - *Optional* (`--merge_flumut_mutations`, default off): reference coordinate conversion — builds a run-level, idempotent nucleotide/amino-acid coordinate map lifting flumut's fixed reference onto the selected Nextclade reference dataset coordinate system (`bin/build_reference_coordinate_map.py`) — followed by FluMutDB annotation of the VCF, which tags each fluscan amino-acid call that matches a curated FluMutDB marker with a `FLUMUT=` INFO field carrying the mutation name and its published effects, split into those directly attributable to that mutation and those associated with it through a multi-mutation marker, each qualified by the subtype it was observed in. No rows are added or removed, so a marker is only reported where fluscan called a difference from the Nextclade reference. Because FluMutDB numbers HA by its HA1/HA2 subunits, HA markers are only annotated when the Nextclade dataset also exposes that split — either as two dedicated `HA1`/`HA2` CDS features, or as `mature_protein_region_of_CDS` children of a single HA CDS (the NCBI/RefSeq model, which the GsGD dataset uses). A dataset offering only a bare `HA` CDS is flagged `stale_annotation` and passes through with its HA rows unannotated @@ -91,7 +90,6 @@ flowchart TD nc_run[nextclade run batch] split_results[split per-sample JSON and TSV] nc2vcf[nextclade_to_freebayes_vcf.py] - fixmnv_nc[fix_mnv_codons.py] nc_final[Final NEXTCLADE VCFs] coord_map[coordinate_conversion: build_reference_coordinate_map.py] coord_final[aggregate/reference_remap] @@ -104,8 +102,7 @@ flowchart TD groupseg --> nc_run nc_run --> split_results split_results --> nc2vcf - nc2vcf --> fixmnv_nc - fixmnv_nc --> nc_final + nc2vcf --> nc_final dispatch -.->|"flumutdb reference + db nextclade datasets, sample-independent (if merge_flumut_mutations)"| coord_map coord_map --> coord_final coord_map -.->|"per-segment shard (if merge_flumut_mutations)"| flumut_merge @@ -214,7 +211,7 @@ nextflow run BCCDC-PHL/fluscan-nf \ > > `--merge_flumut_mutations` (boolean, default `false`) gates the two final NEXTCLADE steps — coordinate conversion and FluMutDB annotation of the VCF (`*_flumut_annotated.vcf`). The `coordinate_conversion` step lifts FluMut's own reference, read from the `flumutdb` conda package, onto the `--db` datasets. The marker set comes from the same package, so the two always match and the annotation step does not consume the per-sample `flumut` output. > -> `--emit_no_mature_vcf` (boolean, default `false`) additionally writes each NEXTCLADE VCF a second time with no mature-peptide `ANN=` entries, for consumers that require HA annotated as HA0 only. The two files carry the same records, alleles and order; only the `ANN=` sub-field differs, and segments without mature peptides in their annotation come out identical. The published copy lands in `nextclade/no_mature_peptide/` and gets the same MNV codon correction as the standard VCF. Nothing downstream in the pipeline reads it: `mutation_watch` and `merge_flumut_mutations` resolve HA markers through the HA1/HA2 `Feature_ID` and stay on the standard VCF. +> `--emit_no_mature_vcf` (boolean, default `false`) additionally writes each NEXTCLADE VCF a second time with no mature-peptide `ANN=` entries, for consumers that require HA annotated as HA0 only. The two files carry the same records, alleles and order; only the `ANN=` sub-field differs, and segments without mature peptides in their annotation come out identical. The published copy lands in `nextclade/no_mature_peptide/`. Nothing downstream in the pipeline reads it: `mutation_watch` and `merge_flumut_mutations` resolve HA markers through the HA1/HA2 `Feature_ID` and stay on the standard VCF. **Package management:** - Conda is used for dependency management. The `-profile conda` flag and `--cache` parameter are required on BCCDC systems. @@ -275,7 +272,7 @@ The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder `FORMAT/GQ` as |------|-------------| | `nextclade/json/*_nextclade.json` | Per-segment Nextclade JSON split out from the cross-sample batch result | | `nextclade/tsv/*_nextclade.tsv` | Per-segment Nextclade TSV split out from the cross-sample batch result | -| `nextclade/*_final.vcf` | Per-segment final VCF after Nextclade-to-VCF conversion and MNV codon correction | +| `nextclade/*_final.vcf` | Per-segment final VCF from the Nextclade-to-VCF conversion. Rows in a codon holding more than one substitution carry the combined change, plus `MNV_CORRECTED` and `MNV_ORIGINAL_HGVS_P` where it differs from the single-base call | | `nextclade/no_mature_peptide/*_nc_nomat_final.vcf` | *Optional* (`--emit_no_mature_vcf`): the same records annotated without mature peptides, so HA carries only its HA0 `ANN=` entry and no HA1/HA2 ones | | `nextclade/flumut_annotated/*_flumut_annotated.vcf` | *Optional* (`--merge_flumut_mutations`): the same VCF with a `FLUMUT=` INFO field (`Protein\|AA_Pos\|Alt_AA\|Mutation_Name\|DirectEffect\|AssociatedEffect`) on rows whose amino-acid change matches a curated FluMutDB marker | | `nextclade/*_watchlist_hits.tsv` | *Optional* (`--watchlist_dir`): watchlist substitutions found in the final VCF. Independent of `--merge_flumut_mutations`: `FLUMUT=` is a curated annotation written *into* the VCF, this is a custom list scanned *out* of it | diff --git a/bin/nextclade_to_freebayes_vcf.py b/bin/nextclade_to_freebayes_vcf.py index 62aa654..d6df3bf 100755 --- a/bin/nextclade_to_freebayes_vcf.py +++ b/bin/nextclade_to_freebayes_vcf.py @@ -17,7 +17,8 @@ --output-no-mature writes a second VCF alongside the first, carrying the same records but no mature-peptide ANN entries: HA is annotated as HA0 only, with no HA1/HA2 entries. -Pipeline position: after nextclade run, before fix_mnv_codons.py. +Substitutions sharing a codon are annotated with the combined codon change; rows whose +annotation differs from the single-base one carry MNV_CORRECTED and MNV_ORIGINAL_HGVS_P. """ import argparse @@ -332,10 +333,11 @@ def _cds_to_genomic(cds_pos_0: int, segments: list, strand: str) -> Optional[int return None -def _synonymous_aa_change(ref_seq: str, tx: dict, pos_0: int, ref_nt: str, alt_nt: str) -> tuple[str, str, int]: - """Return (refAa, altAa, aa_pos_1) by direct codon translation from the reference. +def _codon_aa_change(ref_seq: str, tx: dict, pos_0: int, qry_subs: dict[int, str]) -> tuple[str, str, int, int]: + """Return (refAa, altAa, aa_pos_1, n_subs) for the codon holding pos_0, translated with + every substitution in qry_subs applied; n_subs counts the substitutions inside that codon. - Falls back to ('X','X',0) when the codon cannot be determined (e.g. minus strand, + Falls back to ('X','X',0,1) when the codon cannot be determined (e.g. minus strand, segment-boundary ambiguity, or out-of-range position). """ segs = tx["segments"] @@ -350,25 +352,21 @@ def _synonymous_aa_change(ref_seq: str, tx: dict, pos_0: int, ref_nt: str, alt_n break running += (e - b) if cds_local_0 is None: - return "X", "X", 0 + return "X", "X", 0, 1 - codon_idx = cds_local_0 // 3 - codon_offset = cds_local_0 % 3 - aa_pos_1 = codon_idx + 1 + codon_idx = cds_local_0 // 3 + aa_pos_1 = codon_idx + 1 # Genomic positions for all 3 codon bases gpos = [_cds_to_genomic(codon_idx * 3 + i, segs, strand) for i in range(3)] - if None in gpos: - return "X", "X", aa_pos_1 + if None in gpos or strand != "+": + # minus strand reads the codon reversed; just fall back + return "X", "X", aa_pos_1, 1 - if strand == "+": - ref_codon = "".join(ref_seq[g] for g in gpos) - alt_codon = ref_codon[:codon_offset] + alt_nt + ref_codon[codon_offset + 1:] - else: - # minus strand: reading direction is reversed; just fall back - return "X", "X", aa_pos_1 - - return _translate_codon(ref_codon), _translate_codon(alt_codon), aa_pos_1 + ref_codon = "".join(ref_seq[g] for g in gpos) + alt_codon = "".join(qry_subs.get(g, ref_seq[g]) for g in gpos) + n_subs = sum(g in qry_subs for g in gpos) + return _translate_codon(ref_codon), _translate_codon(alt_codon), aa_pos_1, n_subs # --------------------------------------------------------------------------- # HGVS formatters (1-letter amino-acid codes, matching paftools/snpEff output) @@ -546,6 +544,15 @@ def build_header(chrom: str, ref_len: int, ref_path: str, sample: str) -> str: " cDNA.length | CDS.pos / CDS.length | AA.pos / AA.length | Distance |" " ERRORS / WARNINGS / INFO'\">" ) + lines.append( + '##INFO=' + ) + lines.append( + '##INFO=' + ) for fid, number, typ, desc in _FREEBAYES_FORMAT: lines.append(f'##FORMAT=') lines.append(f"#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\t{sample}") @@ -555,6 +562,29 @@ def build_header(chrom: str, ref_len: int, ref_path: str, sample: str) -> str: # Per-variant ANN field builders # --------------------------------------------------------------------------- +def _coding_entry( + tx: dict, ref_nt: str, alt_nt: str, cds_local_1: int, rank_str: str, aa_change: tuple, +) -> tuple[str, str]: + """Return (ANN entry, HGVS.p) for a substitution inside a coding transcript.""" + ref_aa, alt_aa, aa_pos_1 = aa_change[:3] + annotation, impact = _classify_aa_consequence(ref_aa, alt_aa, aa_pos_1) + hgvs_p = (_format_hgvs_p_syn(ref_aa, aa_pos_1) + if annotation == "synonymous_variant" + else _format_hgvs_p_sub(ref_aa, aa_pos_1, alt_aa)) + entry = _build_ann_entry( + allele=alt_nt, annotation=annotation, impact=impact, + gene_name=tx["gene_name"], gene_id=tx["gene_id"], + feature_id=tx["feature_id"], + transcript_biotype="protein_coding", + hgvs_c=_format_hgvs_c_snp(cds_local_1, ref_nt, alt_nt), hgvs_p=hgvs_p, + cds_pos=cds_local_1, cds_len=tx["cds_length"], + aa_pos=aa_pos_1, aa_len=tx["aa_length"], + rank_str=rank_str, + warnings=tx["warnings"], + ) + return entry, hgvs_p + + def _ann_field_for_sub( pos_0: int, ref_nt: str, @@ -563,9 +593,20 @@ def _ann_field_for_sub( genes: list, transcripts: list, ref_seq: str, + qry_subs: dict[int, str], ) -> str: - """Return 'ANN=...' INFO sub-field for a nucleotide substitution.""" + """Return the 'ANN=...' INFO sub-field for a nucleotide substitution, followed by + MNV_CORRECTED/MNV_ORIGINAL_HGVS_P when a shared codon changes its annotation.""" entries: list[str] = [] + # (entry, single-base entry, single-base HGVS.p, feature) for codons holding 2+ substitutions + mnv: list[tuple[str, str, str, str]] = [] + + def add_coding(tx, cds_local_1, rank_str, aa_change, single_base): + entry, _ = _coding_entry(tx, ref_nt, alt_nt, cds_local_1, rank_str, aa_change) + entries.append(entry) + if aa_change[3] > 1: + mnv.append((entry, *_coding_entry(tx, ref_nt, alt_nt, cds_local_1, rank_str, single_base), + tx["feature_id"])) # --- 1. Main CDS transcripts: in-CDS or intronic --- for tx in transcripts: @@ -574,32 +615,17 @@ def _ann_field_for_sub( cds_local_1, exon_rank, exon_count = _cds_pos_from_genomic(tx, pos_0) if cds_local_1 is not None: rank_str = f"{exon_rank}/{exon_count}" + codon = _codon_aa_change(ref_seq, tx, pos_0, qry_subs) + # Nextclade's own call translates the query codon, so it needs no correction. aa_changes = [c for c in nuc_to_aa.get(str(pos_0), []) if c["cdsName"] == tx["protein_id"]] if aa_changes: - ch = aa_changes[0] - ref_aa = ch["refAa"] - alt_aa = ch["qryAa"] - aa_pos_1 = ch["pos"] + 1 + ch = aa_changes[0] + codon = (ch["refAa"], ch["qryAa"], ch["pos"] + 1, codon[3]) + single_base = codon else: - ref_aa, alt_aa, aa_pos_1 = _synonymous_aa_change(ref_seq, tx, pos_0, ref_nt, alt_nt) - - annotation, impact = _classify_aa_consequence(ref_aa, alt_aa, aa_pos_1) - hgvs_c = _format_hgvs_c_snp(cds_local_1, ref_nt, alt_nt) - hgvs_p = (_format_hgvs_p_syn(ref_aa, aa_pos_1) - if annotation == "synonymous_variant" - else _format_hgvs_p_sub(ref_aa, aa_pos_1, alt_aa)) - - entries.append(_build_ann_entry( - allele=alt_nt, annotation=annotation, impact=impact, - gene_name=tx["gene_name"], gene_id=tx["gene_id"], - feature_id=tx["feature_id"], - transcript_biotype="protein_coding", - hgvs_c=hgvs_c, hgvs_p=hgvs_p, - cds_pos=cds_local_1, cds_len=tx["cds_length"], - aa_pos=aa_pos_1, aa_len=tx["aa_length"], - rank_str=rank_str, - )) + single_base = _codon_aa_change(ref_seq, tx, pos_0, {pos_0: alt_nt}) + add_coding(tx, cds_local_1, rank_str, codon, single_base) continue # Check for intron position in spliced transcripts @@ -630,23 +656,9 @@ def _ann_field_for_sub( if cds_local_1 is None: continue rank_str = f"{exon_rank}/{exon_count}" - ref_aa, alt_aa, aa_pos_1 = _synonymous_aa_change(ref_seq, tx, pos_0, ref_nt, alt_nt) - annotation, impact = _classify_aa_consequence(ref_aa, alt_aa, aa_pos_1) - hgvs_c = _format_hgvs_c_snp(cds_local_1, ref_nt, alt_nt) - hgvs_p = (_format_hgvs_p_syn(ref_aa, aa_pos_1) - if annotation == "synonymous_variant" - else _format_hgvs_p_sub(ref_aa, aa_pos_1, alt_aa)) - entries.append(_build_ann_entry( - allele=alt_nt, annotation=annotation, impact=impact, - gene_name=tx["gene_name"], gene_id=tx["gene_id"], - feature_id=tx["feature_id"], - transcript_biotype="protein_coding", - hgvs_c=hgvs_c, hgvs_p=hgvs_p, - cds_pos=cds_local_1, cds_len=tx["cds_length"], - aa_pos=aa_pos_1, aa_len=tx["aa_length"], - rank_str=rank_str, - warnings=tx["warnings"], - )) + add_coding(tx, cds_local_1, rank_str, + _codon_aa_change(ref_seq, tx, pos_0, qry_subs), + _codon_aa_change(ref_seq, tx, pos_0, {pos_0: alt_nt})) elif pos_0 < seg_begin: distance = seg_begin - pos_0 hgvs_c = f"c.-{distance}{ref_nt}>{alt_nt}" @@ -692,7 +704,11 @@ def _ann_field_for_sub( if not entries: return "" - return "ANN=" + ",".join(entries) + ann = "ANN=" + ",".join(entries) + if any(entry != single for entry, single, _, _ in mnv): + original = "|".join(f"{feature}:{hgvs_p}" for _, _, hgvs_p, feature in mnv) + ann += f";MNV_CORRECTED;MNV_ORIGINAL_HGVS_P={original}" + return ann def _ann_field_for_del( @@ -895,12 +911,13 @@ def variants_from_json_result( for i in result.get("aaInsertions", []) ] frame_shifts = result.get("frameShifts", []) + qry_subs = {sub["pos"]: sub["qryNuc"] for sub in result.get("substitutions", [])} for sub in result.get("substitutions", []): vcf_pos = sub["pos"] + 1 ann = _ann_field_for_sub( sub["pos"], sub["refNuc"], sub["qryNuc"], - nuc_to_aa, genes, transcripts, ref_seq, + nuc_to_aa, genes, transcripts, ref_seq, qry_subs, ) rows.append((vcf_pos, variant_row(chrom, vcf_pos, sub["refNuc"], sub["qryNuc"], qual, ann=ann))) diff --git a/docs/vcf_annotation_and_flumut_insertion.md b/docs/vcf_annotation_and_flumut_insertion.md index 9c309fd..7ffd6e4 100644 --- a/docs/vcf_annotation_and_flumut_insertion.md +++ b/docs/vcf_annotation_and_flumut_insertion.md @@ -21,9 +21,7 @@ which round-trip a VCF byte-for-byte. `bin/` holds only entry points — Nextflo > sequences and protein coordinates from it, and the merge takes the marker catalogue. ``` -consensus.fasta ─► nextclade_run_batch ─► nextclade_to_freebayes_vcf.py ─► *_nc.vcf - │ - fix_mnv_codons.py (post_process) ─► *_final.vcf +consensus.fasta ─► nextclade_run_batch ─► nextclade_to_freebayes_vcf.py ─► *_final.vcf │ coordinate_conversion (flumut→Nextclade map) ─┐ │ FluMutDB (flumutdb conda package) ────────────┼─► merge_flumut_mutations.py ─► *_flumut_annotated.vcf @@ -120,13 +118,13 @@ For each substitution the script walks every transcript and appends an `ANN` ent feature it touches (so a base in an overlap region yields several entries): 1. **Inside a main CDS** — take the AA change from `nucToAaMuts` when present, otherwise - recompute it by translating the reference codon (`_synonymous_aa_change`). Classify the + translate the codon with every substitution in it applied (`_codon_aa_change`). Classify the consequence (`_classify_aa_consequence`: `synonymous_variant` / `missense_variant` / `stop_gained` / `stop_lost` / `start_lost`, with LOW/MODERATE/HIGH impact) and format `HGVS.c` (`c.>`) and `HGVS.p`. `Rank` = exon/exon-count. 2. **Intronic** (a gap between the exons of a spliced transcript) — `intron_variant`, MODIFIER, `HGVS.c` in `c.+>` form. -3. **Mature-peptide region** — same AA logic, tagged with +3. **Mature-peptide region** — AA change from `_codon_aa_change`, tagged with `WARNING_TRANSCRIPT_NO_START_CODON`; positions 5′ of the region become `upstream_gene_variant` with a `Distance`. 4. **Upstream of a main CDS** (`pos < CDS start`) — `upstream_gene_variant`, MODIFIER, @@ -141,14 +139,16 @@ that distinguish in-frame indels (`conservative_inframe_deletion/insertion`) fro `##fileformat=VCFv4.2`, `PASS` + `AmbiguousInsert` filters, `fileDate`, `source`, `reference`, a single `##contig`, the complete freebayes `##INFO`/`##FORMAT` definitions, -and the `##INFO=` definition. +and the `ANN`, `MNV_CORRECTED` and `MNV_ORIGINAL_HGVS_P` `##INFO` definitions. -### Known caveat — multi-nucleotide codons +### Substitutions sharing a codon -Each SNV is annotated independently, assuming the rest of its codon matches the reference. -When two consensus SNVs fall in the same codon, their individual `HGVS.p` values can be -wrong. This is corrected downstream by `fix_mnv_codons.py` (the `post_process_vcf*` -processes), which recomputes the real codon before the `*_final.vcf` is produced. +`nucToAaMuts` and `_codon_aa_change` both translate the whole query codon, so every +substitution in a codon carries the combined change. Where an entry differs from the change +its substitution would make alone, the row also carries `MNV_CORRECTED` and +`MNV_ORIGINAL_HGVS_P` (`featureID:HGVS.p` for every feature whose codon holds more than one +substitution, pipe-separated): the same fields `fix_mnv_codons.py` writes on the PAFTOOLS +and SHORT_READ VCFs. --- @@ -245,7 +245,7 @@ merge_flumut_mutations.py --vcf --coord-map .json`) | `coordinate_conversion` | flumut→Nextclade **AA position** mapping, per gene | | FluMutDB sqlite | `flumutdb` conda package | the marker catalogue; located via `flumut.DbReader.get_db_file()`, overridable with `--flumut-db` | @@ -267,8 +267,9 @@ one are independent. pair columns 1:1, so the inverse is well defined; a collision would be warned about and resolved to the lowest source position. 4. **Skip rows flagged `MNV_UNRESOLVED`** (`has_info_flag`). `fix_mnv_codons.py` sets that - flag when it identified a multi-SNV codon but could not compute the combined change, - leaving the row's `HGVS.p` at a value the pipeline has already determined to be wrong; + flag on PAFTOOLS and SHORT_READ VCFs when it identified a multi-SNV codon but could not + compute the combined change (the Nextclade converter always has the reference codon and + never sets it), leaving the row's `HGVS.p` at a value the pipeline has already determined to be wrong; matching a curated marker against it would manufacture a false positive. 5. **Per row, per `ANN` entry** (`row_flumut_field` → `aa_changes_from_info`): take `Gene_Name` and `Feature_ID` — resolved as described under *Gene names* below — and the @@ -369,8 +370,8 @@ flag rows that were in fact annotated correctly. FluMutDB 6.6's mapped mutations are in this state (`PB2:M535I`, `NA-1:V149A`). - FluMutDB holds **substitutions only** (`mutations.type` is `SNP` throughout), so indel and non-coding rows never match. -- **Resolved MNV groups repeat an entry** — every member row of a corrected multi-SNV codon - carries the same corrected `HGVS.p`, so the same `FLUMUT=` entry is emitted on each of them. +- **Shared codons repeat an entry** — every row of a codon holding more than one substitution + carries the same combined `HGVS.p`, so the same `FLUMUT=` entry is emitted on each of them. Any downstream count of "markers found" must de-duplicate. ### The retired insertion-based implementation diff --git a/modules/nextclade/nextclade.nf b/modules/nextclade/nextclade.nf index 0cef37e..16731e7 100644 --- a/modules/nextclade/nextclade.nf +++ b/modules/nextclade/nextclade.nf @@ -69,7 +69,7 @@ process nextclade_run_batch { } -// Converts per-sample nextclade JSON files (produced by nextclade_run_batch) into +// Converts per-sample nextclade JSON files (produced by nextclade_run_batch) into the final // freebayes-style VCFs, one per sample, for the given segment. process nextclade_to_vcf_batch { @@ -77,17 +77,27 @@ process nextclade_to_vcf_batch { tag { segment } + publishDir "${params.outdir}", mode: 'copy', + saveAs: { filename -> + def workflow = task.ext.workflow ?: 'unknown_workflow' + def sample_id = filename.split('__')[0] + if (filename.endsWith('_nc_nomat_final.vcf')) { + return "${sample_id}/${workflow}/no_mature_peptide/${filename}" + } + return "${sample_id}/${workflow}/${filename}" + } + input: tuple val(segment), val(sample_ids), path(json_files, stageAs: 'json/*'), path(dataset_dir) output: - tuple val(segment), path('*_nc.vcf'), path(dataset_dir), emit: vcf - tuple val(segment), path('*_nc_nomat.vcf'), path(dataset_dir), optional: true, emit: vcf_no_mature + tuple val(segment), path('*_nc_final.vcf'), emit: vcf + tuple val(segment), path('*_nc_nomat_final.vcf'), optional: true, emit: vcf_no_mature script: - def no_mature_arg = params.emit_no_mature_vcf ? "--output-no-mature \"\${sample_id}__\${REFERENCE_NAME}__\${SUBTYPE}__\${SEGMENT}_nc_nomat.vcf\"" : "" + def no_mature_arg = params.emit_no_mature_vcf ? "--output-no-mature \"\${sample_id}__\${REFERENCE_NAME}__\${SUBTYPE}__\${SEGMENT}_nc_nomat_final.vcf\"" : "" """ REFERENCE_NAME=\$(head -n1 ${dataset_dir}/reference.fasta | sed -E 's/>(\\S+).*/\\1/g') SUBTYPE=\$(head -n1 ${dataset_dir}/reference.fasta | sed -E 's/.+\\(\\S+\\((\\S+)\\)\\S*\\).+/\\1/g') @@ -100,7 +110,7 @@ process nextclade_to_vcf_batch { nextclade_to_freebayes_vcf.py \\ -i "\$json_file" \\ -d ${dataset_dir} \\ - -o "\${sample_id}__\${REFERENCE_NAME}__\${SUBTYPE}__\${SEGMENT}_nc.vcf" \\ + -o "\${sample_id}__\${REFERENCE_NAME}__\${SUBTYPE}__\${SEGMENT}_nc_final.vcf" \\ ${no_mature_arg} \\ --sample "\$sample_id" done diff --git a/modules/shared/post_process.nf b/modules/shared/post_process.nf index bcc2be8..8f73b06 100644 --- a/modules/shared/post_process.nf +++ b/modules/shared/post_process.nf @@ -1,6 +1,6 @@ -// shared/post_process.nf - MNV codon correction applied to annotated VCFs (shared by all workflows) +// shared/post_process.nf - MNV codon correction applied to SnpEff-annotated VCFs (PAFTOOLS and SHORT_READ) -// Per-sample batched form: processes all 8 segments in one task (paftools workflow). +// Processes all segments of one sample in one task. // VCF files named {sample_id}__{ref}__{subtype}__{segment}_paf.vcf; // references named {sample_id}_ref_{segment}.s.fasta — matched by segment in the loop. process post_process_vcf { @@ -60,72 +60,3 @@ EOF ./run_fix_mnv_sample.sh {} '${sample_id}' '${min_af}' """ } - - -// Per-segment cross-sample batch form: processes all samples for one segment in one task (nextclade workflow). -// VCF files named {sample_id}__{ref}__{subtype}__{segment}_nc.vcf; all share the segment reference. -// publishDir uses saveAs to route each sample's VCF to its own output directory. -process post_process_vcf_batch { - tag { segment } - - publishDir "${params.outdir}", mode: 'copy', - saveAs: { filename -> - def workflow = task.ext.workflow ?: 'unknown_workflow' - def sample_id = filename.split('__')[0] - // Checked first: a no-mature VCF also ends in _final.vcf, and belongs in its - // own subfolder rather than beside the VCF the rest of the workflow consumes. - if (filename.endsWith('_nomat_final.vcf')) { - return "${sample_id}/${workflow}/no_mature_peptide/${filename}" - } - if (filename.endsWith('_final.vcf')) { - return "${sample_id}/${workflow}/${filename}" - } - return null - } - - input: - tuple val(segment), - path(vcfs, stageAs: 'vcf/*'), - path(dataset_dir) - - output: - tuple val(segment), path("*_final.vcf"), emit: vcf - tuple val(segment), path("post_process_${segment}_provenance.yml"), emit: provenance - - script: - def min_af = params.mnv_min_af ?: 0.5 - def parallel_jobs = task.cpus ?: 1 - """ - printf -- "- process_name: post_process_vcf\\n" > post_process_${segment}_provenance.yml - printf -- " tools: \\n" >> post_process_${segment}_provenance.yml - printf -- " - tool_name: python\\n tool_version: \$(python --version 2>&1 | cut -d' ' -f2)\\n" >> post_process_${segment}_provenance.yml - printf -- " - tool_name: parallel\\n tool_version: \$(parallel --version | head -n1 | cut -d' ' -f3)\\n" >> post_process_${segment}_provenance.yml - printf -- " - tool_name: fix_mnv_codons.py\\n tool_version: 1.0.0\\n" >> post_process_${segment}_provenance.yml - - cp ${dataset_dir}/reference.fasta reference_renamed.fasta - sed -i -E 's/^>([^|]+)\\|.+/>\\1/g' reference_renamed.fasta - - cat > run_fix_mnv_batch.sh <<'EOF' -#!/usr/bin/env bash -set -euo pipefail - -vcf_file="\$1" -min_af="\$2" - -vcf_basename=\$(basename "\$vcf_file" .vcf) - -echo "Processing \$vcf_file with min_af=\${min_af}" - -fix_mnv_codons.py \ - --input "\$vcf_file" \ - --reference reference_renamed.fasta \ - --output-vcf "\${vcf_basename}_final.vcf" \ - --min-af "\${min_af}" -EOF - chmod +x run_fix_mnv_batch.sh - - find vcf -maxdepth 1 -name '*.vcf' -print0 | \ - parallel -0 --will-cite --jobs ${parallel_jobs} --halt soon,fail=1 \ - ./run_fix_mnv_batch.sh {} '${min_af}' - """ -} diff --git a/nextflow.config b/nextflow.config index 9e9638a..3f7a843 100644 --- a/nextflow.config +++ b/nextflow.config @@ -143,12 +143,6 @@ process { memory = '32G' } - // Regex so the post_process_vcf_batch_no_mature alias picks up the same settings. - withName: 'post_process_vcf_batch.*' { - cpus = 16 - memory = '32G' - } - withName: coordinate_conversion { cpus = 2 memory = '4G' diff --git a/tests/test_nextclade_to_freebayes_vcf.py b/tests/test_nextclade_to_freebayes_vcf.py index f1bc881..eb4f2ad 100644 --- a/tests/test_nextclade_to_freebayes_vcf.py +++ b/tests/test_nextclade_to_freebayes_vcf.py @@ -1,4 +1,4 @@ -"""Unit tests for bin/nextclade_to_freebayes_vcf.py, covering the two-output split. +"""Unit tests for bin/nextclade_to_freebayes_vcf.py: the two-output split and shared codons. Run with: pytest tests/test_nextclade_to_freebayes_vcf.py @@ -66,10 +66,13 @@ def _write_dataset(directory: Path, gff3: str) -> Path: return directory -def _write_json(path: Path, positions=SUB_POSITIONS, sample: str = "SAMPLE01") -> Path: - """A minimal nextclade v3 result carrying one substitution per position.""" +def _write_json(path: Path, positions=SUB_POSITIONS, sample: str = "SAMPLE01", + qry: dict[int, str] | None = None) -> Path: + """A minimal nextclade v3 result carrying one substitution per position (qry sets the base).""" + qry = qry or {} substitutions = [ - {"pos": pos, "refNuc": REF_SEQ[pos], "qryNuc": "A" if REF_SEQ[pos] != "A" else "C"} + {"pos": pos, "refNuc": REF_SEQ[pos], + "qryNuc": qry.get(pos, "A" if REF_SEQ[pos] != "A" else "C")} for pos in positions ] path.write_text(json.dumps({ @@ -195,6 +198,58 @@ def test_no_mature_output_matches_a_dataset_without_mature_peptides(tmp_path, ne assert _records(from_flag) == _records(from_gff) +# --------------------------------------------------------------------------- +# Substitutions sharing a codon +# --------------------------------------------------------------------------- + +# 0-based 107-109 is a TAC (Tyr) codon: CDS codon 30, HA1 codon 29. T>G alone gives GAC (Asp), +# A>C alone gives TCC (Ser), together GCC (Ala). +MNV_POSITIONS = [107, 108, 400] +MNV_QRY = {107: "G", 108: "C"} + + +def _info(record: str) -> dict[str, str]: + return dict(token.partition("=")[::2] for token in record.split("\t")[7].split(";")) + + +def _hgvs_p(record: str) -> dict[str, str]: + return {fields[6]: fields[10] for fields in _ann_entries(record) if fields[10]} + + +@pytest.fixture +def mnv_records(tmp_path, dataset): + json_path = _write_json(tmp_path / "mnv.json", positions=MNV_POSITIONS, qry=MNV_QRY) + out = tmp_path / "mnv.vcf" + _run(dataset, json_path, out) + return _records(out) + + +def test_shared_codon_is_annotated_with_the_combined_change(mnv_records): + for record in mnv_records[:2]: + assert _hgvs_p(record) == {"TESTLOCUS": "p.Y30A", "HA1": "p.Y29A"} + assert "missense_variant" in record + + +def test_shared_codon_records_the_single_base_calls(mnv_records): + first, second, _ = mnv_records + assert "MNV_CORRECTED" in _info(first) + assert _info(first)["MNV_ORIGINAL_HGVS_P"] == "TESTLOCUS:p.Y30D|HA1:p.Y29D" + assert _info(second)["MNV_ORIGINAL_HGVS_P"] == "TESTLOCUS:p.Y30S|HA1:p.Y29S" + + +def test_lone_substitution_carries_no_mnv_fields(mnv_records): + info = _info(mnv_records[2]) + assert "MNV_CORRECTED" not in info and "MNV_ORIGINAL_HGVS_P" not in info + + +def test_mnv_fields_are_declared(tmp_path, dataset, nextclade_json): + out = tmp_path / "out.vcf" + _run(dataset, nextclade_json, out) + header = out.read_text() + assert "##INFO= [sample_id, segment, vcf]. Shared by the watchlist scan below // and, when enabled, by the flumut merge further down. - ch_final_vcf = post_process_vcf_batch.out.vcf + ch_final_vcf = nextclade_to_vcf_batch.out.vcf .transpose() .map { segment, vcf -> def sample_id = vcf.name.split('__')[0] [sample_id, segment, vcf] } - // --- Step 6a: The same VCFs without mature-peptide (HA1/HA2) annotations (optional) --- - // A parallel output for consumers that require HA annotated as HA0 only. Nothing - // downstream in this workflow reads it: mutation_watch and merge_flumut_mutations - // resolve HA markers through the HA1/HA2 Feature_ID and stay on ch_final_vcf. - if (params.emit_no_mature_vcf) { - post_process_vcf_batch_no_mature(nextclade_to_vcf_batch.out.vcf_no_mature) - - ch_final_vcf_no_mature = post_process_vcf_batch_no_mature.out.vcf - .transpose() - .map { segment, vcf -> - def sample_id = vcf.name.split('__')[0] - [sample_id, segment, vcf] - } - } - - // --- Step 6b: Scan the final VCF for a custom watchlist (optional) --- + // --- Step 6: Scan the final VCF for a custom watchlist (optional) --- // Independent of --merge_flumut_mutations: FLUMUT= is a curated annotation written // into the VCF, this is a custom list scanned out of it into a separate TSV. if (params.watchlist_dir != 'NO_FILE') { @@ -213,8 +196,8 @@ workflow NEXTCLADE { merge_flumut_mutations(ch_merge_in) } - // Per-sample provenance. post_process_vcf_batch is keyed by segment rather than by - // sample, so it has no per-sample record to join here. + // Per-sample provenance. The Nextclade steps are keyed by segment rather than by + // sample, so they have no per-sample record to join here. ch_provenance = ch_fasta_input.map{ it -> it[0] } ch_provenance = ch_provenance.combine(ch_pipeline_provenance).map{ it -> [it[0], [it[1]]] } ch_provenance = ch_provenance.join(hash_files.out.provenance).map{ it -> [it[0], it[1] << it[2]] } From c37b49f43c9d386ee023c9c0620c3cd9495f4512 Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Tue, 29 Sep 2026 16:53:41 -0400 Subject: [PATCH 02/11] Convert a whole Nextclade batch in one converter call nextclade_run_batch now writes every sample's VCF from the batch JSON, replacing the per-sample loop in nextclade_to_vcf_batch. A segment where no sample aligns publishes no VCFs instead of failing the task. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- bin/nextclade_to_freebayes_vcf.py | 122 +++++--------------- docs/vcf_annotation_and_flumut_insertion.md | 17 +-- modules/nextclade/nextclade.nf | 72 ++++-------- nextflow.config | 5 - tests/test_nextclade_to_freebayes_vcf.py | 105 ++++++++--------- workflows/nextclade.nf | 18 ++- 7 files changed, 113 insertions(+), 228 deletions(-) diff --git a/README.md b/README.md index 2bb1f37..4446cce 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,7 @@ flowchart TD split_consensus --> groupseg groupseg --> nc_run nc_run --> split_results - split_results --> nc2vcf + nc_run --> nc2vcf nc2vcf --> nc_final dispatch -.->|"flumutdb reference + db nextclade datasets, sample-independent (if merge_flumut_mutations)"| coord_map coord_map --> coord_final diff --git a/bin/nextclade_to_freebayes_vcf.py b/bin/nextclade_to_freebayes_vcf.py index d6df3bf..8cd1bed 100755 --- a/bin/nextclade_to_freebayes_vcf.py +++ b/bin/nextclade_to_freebayes_vcf.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Convert nextclade JSON output to a freebayes-format VCF with snpEff-compatible ANN= annotations. +"""Convert nextclade JSON output to freebayes-format VCFs with snpEff-compatible ANN= annotations. Reads a per-segment GFF3 annotation (the same file Nextclade consumed) to resolve CDS structure, then annotates each variant with per-transcript HGVS.c / HGVS.p entries for @@ -11,11 +11,13 @@ spliced CDSes (M2, NEP, PA-X) natively from GFF3 segment data. Requires: - -i Nextclade v3 JSON (per-segment) - -d Nextclade reference dataset directory (must contain reference.fasta and genome_annotation.gff3) + -i Nextclade v3 JSON for one segment, any number of samples + -d Nextclade reference dataset directory (must contain reference.fasta and genome_annotation.gff3) + --suffix each sample's VCF is written to ---output-no-mature writes a second VCF alongside the first, carrying the same records but -no mature-peptide ANN entries: HA is annotated as HA0 only, with no HA1/HA2 entries. +A sequence Nextclade could not align gets no VCF. --no-mature-suffix writes a second VCF +per sample, carrying the same records but no mature-peptide ANN entries: HA is annotated +as HA0 only, with no HA1/HA2 entries. Substitutions sharing a codon are annotated with the combined codon change; rows whose annotation differs from the single-base one carry MNV_CORRECTED and MNV_ORIGINAL_HGVS_P. @@ -24,7 +26,6 @@ import argparse import json import logging -import re import sys from datetime import date from pathlib import Path @@ -970,29 +971,6 @@ def write_vcf( return len(rows) -def write_results( - output_path: Path, - results: list[tuple[str, list[tuple[int, str]]]], - chrom: str, - ref_len: int, - ref_path: str, - named_sample: bool, -) -> None: - """Write one VCF per result; multi-sample input appends the sample name to the stem.""" - if len(results) == 1: - sample, rows = results[0] - n = write_vcf(str(output_path), chrom, ref_len, ref_path, sample, rows) - log.info("Wrote %d records for '%s' -> %s", n, sample, output_path) - return - - if named_sample: - log.warning("--sample ignored for multi-sample input") - for sample, rows in results: - safe = re.sub(r"[^\w.-]", "_", sample) - out = output_path.parent / f"{output_path.stem}_{safe}.vcf" - n = write_vcf(str(out), chrom, ref_len, ref_path, sample, rows) - log.info("Wrote %d records for '%s' -> %s", n, sample, out) - # --------------------------------------------------------------------------- # CLI # --------------------------------------------------------------------------- @@ -1000,37 +978,24 @@ def write_results( def get_args() -> argparse.Namespace: p = argparse.ArgumentParser( description=( - "Convert nextclade JSON output to a freebayes-format VCF with " + "Convert nextclade JSON output to one freebayes-format VCF per sample with " "snpEff-compatible ANN= annotations sourced from a GFF3 annotation file." ) ) p.add_argument("-i", "--input", required=True, - help="Nextclade v3 JSON input file") + help="Nextclade v3 JSON; one VCF is written per entry in results") p.add_argument("-d", "--dataset", required=True, help="Nextclade reference dataset directory (must contain reference.fasta and genome_annotation.gff3)") - p.add_argument("-o", "--output", required=True, - help="Output VCF file (used as prefix when input has multiple samples)") - p.add_argument("--output-no-mature", default=None, dest="output_no_mature", - help="Optional second output VCF with the same records but no " + p.add_argument("--suffix", required=True, + help="Each VCF is written to in the working directory") + p.add_argument("--no-mature-suffix", default=None, dest="no_mature_suffix", + help="Also write with the same records but no " "mature-peptide (HA1/HA2) ANN entries") - p.add_argument("--sample", default=None, - help="Override sample name (single-sample inputs only)") p.add_argument("--qual", type=int, default=60, help="QUAL score for all records (default: 60)") return p.parse_args() -_SEED_COVER_RE = re.compile(r"seed alignment covers (\d+(?:\.\d+)?%)") - - -def _extract_seed_cover(error_messages: list[str]) -> Optional[str]: - for msg in error_messages: - m = _SEED_COVER_RE.search(msg) - if m: - return m.group(1) - return None - - def main() -> None: args = get_args() @@ -1056,62 +1021,27 @@ def main() -> None: # The no-mature output is the same annotation pass run against a transcript list with # the mature peptides removed, so it matches what a GFF3 without them would produce. transcripts_no_mature = [tx for tx in transcripts if not tx["is_mature"]] - if args.output_no_mature: - log.info( - "Also writing a no-mature VCF -> %s (%d mature-peptide transcript(s) omitted)", - args.output_no_mature, len(transcripts) - len(transcripts_no_mature), - ) with open(args.input) as fh: data = json.load(fh) - results: list[tuple[str, list[tuple[int, str]]]] = [] - results_no_mature: list[tuple[str, list[tuple[int, str]]]] = [] - for result in data.get("results", []): + sample = result.get("seqName", "SAMPLE") if result.get("errors"): - log.warning("Skipping %s: errors=%s", result.get("seqName", "?"), result["errors"]) + log.warning("Skipping %s: errors=%s", sample, result["errors"]) continue - sample = args.sample or result.get("seqName", "SAMPLE") - results.append(( - sample, - variants_from_json_result(result, ref_seq, chrom, genes, transcripts, args.qual), - )) - if args.output_no_mature: - results_no_mature.append(( - sample, - variants_from_json_result( - result, ref_seq, chrom, genes, transcripts_no_mature, args.qual, - ), - )) - + outputs = [(args.suffix, transcripts)] + if args.no_mature_suffix: + outputs.append((args.no_mature_suffix, transcripts_no_mature)) + for suffix, txs in outputs: + rows = variants_from_json_result(result, ref_seq, chrom, genes, txs, args.qual) + n = write_vcf(f"{sample}{suffix}", chrom, ref_len, str(ref_path), sample, rows) + log.info("Wrote %d records for '%s' -> %s%s", n, sample, sample, suffix) + + # A sequence Nextclade could not align gets no VCF; the batch status TSV records it. for error_entry in data.get("errors", []): - seq_name = error_entry.get("seqName", "UNKNOWN") - sample = args.sample or seq_name - msgs = error_entry.get("errors", []) - if any("Unable to align" in m for m in msgs): - seed_cover = _extract_seed_cover(msgs) - if seed_cover: - log.warning("Unable to align '%s' (seed alignment covers %s); writing empty VCF", - sample, seed_cover) - else: - log.warning("Unable to align '%s'; writing empty VCF", sample) - else: - log.warning("Nextclade errors for '%s': %s; writing empty VCF", - sample, "; ".join(str(m) for m in msgs)) - results.append((sample, [])) - if args.output_no_mature: - results_no_mature.append((sample, [])) - - if not results: - log.error("No results found in %s", args.input) - sys.exit(1) - - write_results(Path(args.output), results, chrom, ref_len, str(ref_path), bool(args.sample)) - if args.output_no_mature: - # named_sample=False: the first call has already warned about --sample if needed. - write_results(Path(args.output_no_mature), results_no_mature, - chrom, ref_len, str(ref_path), False) + log.warning("No VCF for '%s': %s", error_entry.get("seqName", "UNKNOWN"), + "; ".join(str(m) for m in error_entry.get("errors", []))) if __name__ == "__main__": diff --git a/docs/vcf_annotation_and_flumut_insertion.md b/docs/vcf_annotation_and_flumut_insertion.md index 7ffd6e4..7f963af 100644 --- a/docs/vcf_annotation_and_flumut_insertion.md +++ b/docs/vcf_annotation_and_flumut_insertion.md @@ -5,7 +5,7 @@ freebayes-style VCF and tag the curated FluMutDB markers among them: | Script | Role | | --- | --- | -| [`bin/nextclade_to_freebayes_vcf.py`](../bin/nextclade_to_freebayes_vcf.py) | Convert a Nextclade JSON into a freebayes-format VCF with snpEff-style `ANN=` annotations. | +| [`bin/nextclade_to_freebayes_vcf.py`](../bin/nextclade_to_freebayes_vcf.py) | Convert a Nextclade JSON into one freebayes-format VCF per sample with snpEff-style `ANN=` annotations. | | [`bin/merge_flumut_mutations.py`](../bin/merge_flumut_mutations.py) | Add a `FLUMUT=` INFO field to rows whose AA change matches a FluMutDB marker. Run by the `merge_flumut_mutations` process. | | [`bin/lib/flu_gene_names.py`](../bin/lib/flu_gene_names.py) | The canonical gene-key aliases (`NS-1`→`NS1`, `NS2`/`NS-2`→`NEP`), shared by the coordinate map builder and the merge so the two cannot drift. | | [`bin/lib/aa_markers.py`](../bin/lib/aa_markers.py) | `ANN=` extraction, both HGVS.p spellings, and the `MNV_UNRESOLVED` flag test, shared by the merge and the watchlist scanner for the same reason. | @@ -75,12 +75,13 @@ VCFs. It resolves CDS structure directly from the dataset's GFF3, handling overl ### CLI ``` -nextclade_to_freebayes_vcf.py -i -d -o - [--sample NAME] [--qual 60] +nextclade_to_freebayes_vcf.py -i -d --suffix + [--no-mature-suffix ] [--qual 60] ``` `-d` must contain `reference.fasta` and `genome_annotation.gff3` (the same files Nextclade -aligned against). `-o` is used as a filename prefix when the JSON holds multiple sequences. +aligned against). Each sample's VCF is written to `` in the working +directory; `nextclade_run_batch` passes `_______nc_final.vcf`. ### How it runs (`main`) @@ -95,9 +96,9 @@ aligned against). `-o` is used as a filename prefix when the JSON holds multiple and any `warnings`. 3. **Iterate JSON `results`** — one sequence at a time. Sequences whose result carries `errors`, or that appear under the top-level `errors` array (e.g. *"Unable to align"*), - produce an **empty VCF** (header only) with a warning rather than failing the run. -4. **Emit variants** (`variants_from_json_result`) and **write** (`write_vcf`). One output - file per sequence; single-sequence input writes directly to `-o`. + get **no VCF** and a warning; the segment's `*_nextclade_status.tsv` records them. +4. **Emit variants** (`variants_from_json_result`) and **write** (`write_vcf`), one file + per sequence. ### Turning JSON deltas into rows (`variants_from_json_result`) @@ -245,7 +246,7 @@ merge_flumut_mutations.py --vcf --coord-map .json`) | `coordinate_conversion` | flumut→Nextclade **AA position** mapping, per gene | | FluMutDB sqlite | `flumutdb` conda package | the marker catalogue; located via `flumut.DbReader.get_db_file()`, overridable with `--flumut-db` | diff --git a/modules/nextclade/nextclade.nf b/modules/nextclade/nextclade.nf index 16731e7..21b7060 100644 --- a/modules/nextclade/nextclade.nf +++ b/modules/nextclade/nextclade.nf @@ -1,7 +1,8 @@ // nextclade/nextclade.nf - Per-segment Nextclade alignment and VCF conversion processes -// Cross-sample per-segment batch: runs nextclade once for all samples' sequences -// for a given segment, then splits output into per-sample JSON and TSV files. +// Cross-sample per-segment batch: runs nextclade once for all samples' sequences for a +// given segment, splits the output into per-sample JSON and TSV files, and converts it to +// one final VCF per sample. A sample Nextclade cannot align gets no JSON and no VCF. process nextclade_run_batch { conda "${projectDir}/environments/nextclade.yml" @@ -10,6 +11,12 @@ process nextclade_run_batch { publishDir "${params.outdir}", mode: 'copy', saveAs: { filename -> + if (filename.endsWith('_final.vcf')) { + def workflow = task.ext.workflow ?: 'unknown_workflow' + def sample_id = filename.split('__')[0] + def subdir = filename.endsWith('_nc_nomat_final.vcf') ? 'no_mature_peptide/' : '' + return "${sample_id}/${workflow}/${subdir}${filename}" + } if (filename =~ /.*_nextclade_status\.tsv$/) { def workflow = task.ext.workflow ?: 'unknown_workflow' return "${workflow}/status/${filename}" @@ -34,13 +41,17 @@ process nextclade_run_batch { path(dataset_dir) output: - tuple val(segment), val(sample_ids), path("*_nextclade.json"), path(dataset_dir), emit: json_with_ref - tuple val(segment), path("*_nextclade.tsv"), emit: tsv - tuple val(segment), path("nc_output"), emit: all - tuple val(segment), path("${segment}_nextclade_status.tsv"), emit: status + tuple val(segment), path("*_nc_final.vcf"), optional: true, emit: vcf + tuple val(segment), path("*_nc_nomat_final.vcf"), optional: true, emit: vcf_no_mature + tuple val(segment), path("*_nextclade.json"), optional: true, emit: json + tuple val(segment), path("*_nextclade.tsv"), emit: tsv + tuple val(segment), path("nc_output"), emit: all + tuple val(segment), path("${segment}_nextclade_status.tsv"), emit: status script: def threads = task.cpus ?: 1 + def stem = "__\${REFERENCE_NAME}__\${SUBTYPE}__${segment}" + def no_mature_arg = params.emit_no_mature_vcf ? "--no-mature-suffix \"${stem}_nc_nomat_final.vcf\"" : "" """ # Rewrite each segment FASTA header to exactly the sample_id so nextclade # seqName matches sample_id in the result JSON (enables per-sample demux). @@ -65,54 +76,15 @@ process nextclade_run_batch { --tsv nc_output/batch.tsv \\ --segment ${segment} \\ --status-summary ${segment}_nextclade_status.tsv - """ -} - - -// Converts per-sample nextclade JSON files (produced by nextclade_run_batch) into the final -// freebayes-style VCFs, one per sample, for the given segment. -process nextclade_to_vcf_batch { - - conda "${projectDir}/environments/nextclade.yml" - - tag { segment } - publishDir "${params.outdir}", mode: 'copy', - saveAs: { filename -> - def workflow = task.ext.workflow ?: 'unknown_workflow' - def sample_id = filename.split('__')[0] - if (filename.endsWith('_nc_nomat_final.vcf')) { - return "${sample_id}/${workflow}/no_mature_peptide/${filename}" - } - return "${sample_id}/${workflow}/${filename}" - } - - input: - tuple val(segment), val(sample_ids), - path(json_files, stageAs: 'json/*'), - path(dataset_dir) - - output: - tuple val(segment), path('*_nc_final.vcf'), emit: vcf - tuple val(segment), path('*_nc_nomat_final.vcf'), optional: true, emit: vcf_no_mature - - script: - def no_mature_arg = params.emit_no_mature_vcf ? "--output-no-mature \"\${sample_id}__\${REFERENCE_NAME}__\${SUBTYPE}__\${SEGMENT}_nc_nomat_final.vcf\"" : "" - """ REFERENCE_NAME=\$(head -n1 ${dataset_dir}/reference.fasta | sed -E 's/>(\\S+).*/\\1/g') SUBTYPE=\$(head -n1 ${dataset_dir}/reference.fasta | sed -E 's/.+\\(\\S+\\((\\S+)\\)\\S*\\).+/\\1/g') if echo "\$SUBTYPE" | grep -q '^>'; then SUBTYPE="unknown"; fi - SEGMENT="${segment}" - for json_file in json/*.json; do - json_basename=\$(basename "\$json_file" _nextclade.json) - sample_id=\${json_basename%_${segment}} - nextclade_to_freebayes_vcf.py \\ - -i "\$json_file" \\ - -d ${dataset_dir} \\ - -o "\${sample_id}__\${REFERENCE_NAME}__\${SUBTYPE}__\${SEGMENT}_nc_final.vcf" \\ - ${no_mature_arg} \\ - --sample "\$sample_id" - done + nextclade_to_freebayes_vcf.py \\ + -i nc_output/batch.json \\ + -d ${dataset_dir} \\ + --suffix "${stem}_nc_final.vcf" \\ + ${no_mature_arg} """ } diff --git a/nextflow.config b/nextflow.config index 3f7a843..61cf53f 100644 --- a/nextflow.config +++ b/nextflow.config @@ -133,11 +133,6 @@ process { memory = '32G' } - withName: nextclade_to_vcf_batch { - cpus = 16 - memory = '32G' - } - withName: post_process_vcf { cpus = 16 memory = '32G' diff --git a/tests/test_nextclade_to_freebayes_vcf.py b/tests/test_nextclade_to_freebayes_vcf.py index eb4f2ad..ab558da 100644 --- a/tests/test_nextclade_to_freebayes_vcf.py +++ b/tests/test_nextclade_to_freebayes_vcf.py @@ -2,8 +2,8 @@ Run with: pytest tests/test_nextclade_to_freebayes_vcf.py -The script writes the VCF the rest of the workflow consumes and, when ---output-no-mature is given, a second VCF annotated without mature peptides -- HA as HA0 +The script writes one VCF per sample, the file the rest of the workflow consumes, and, when +--no-mature-suffix is given, a second VCF annotated without mature peptides -- HA as HA0 only, no HA1/HA2 entries. The contract the downstream consumer relies on is that the two files differ *only* in their ANN sub-field: same records, same order, same alleles. That is what is asserted here, alongside the guarantee that the standard output is byte-identical @@ -90,12 +90,15 @@ def _write_json(path: Path, positions=SUB_POSITIONS, sample: str = "SAMPLE01", return path -def _run(dataset: Path, json_path: Path, output: Path, no_mature: Path | None = None) -> None: +def _run(dataset: Path, json_path: Path, workdir: Path, no_mature: bool = False) -> tuple[Path, Path]: + """Convert json_path inside workdir; return SAMPLE01's standard and no-mature VCF paths.""" + workdir.mkdir(parents=True, exist_ok=True) cmd = [sys.executable, str(SCRIPT_PATH), "-i", str(json_path), - "-d", str(dataset), "-o", str(output), "--sample", "SAMPLE01"] - if no_mature is not None: - cmd += ["--output-no-mature", str(no_mature)] - subprocess.run(cmd, check=True, capture_output=True, text=True) + "-d", str(dataset), "--suffix", ".vcf"] + if no_mature: + cmd += ["--no-mature-suffix", "_nomat.vcf"] + subprocess.run(cmd, check=True, capture_output=True, text=True, cwd=workdir) + return workdir / "SAMPLE01.vcf", workdir / "SAMPLE01_nomat.vcf" def _records(vcf: Path) -> list[str]: @@ -126,10 +129,7 @@ def nextclade_json(tmp_path): @pytest.fixture def both_outputs(tmp_path, dataset, nextclade_json): - standard = tmp_path / "out.vcf" - no_mature = tmp_path / "out_nomat.vcf" - _run(dataset, nextclade_json, standard, no_mature) - return standard, no_mature + return _run(dataset, nextclade_json, tmp_path / "out", no_mature=True) # --------------------------------------------------------------------------- @@ -189,11 +189,8 @@ def test_no_mature_output_matches_a_dataset_without_mature_peptides(tmp_path, ne with_mature = _write_dataset(tmp_path / "with", GFF3_WITH_MATURE) without = _write_dataset(tmp_path / "without", GFF3_NO_MATURE) - from_flag = tmp_path / "from_flag.vcf" - _run(with_mature, nextclade_json, tmp_path / "ignored.vcf", from_flag) - - from_gff = tmp_path / "from_gff.vcf" - _run(without, nextclade_json, from_gff) + _, from_flag = _run(with_mature, nextclade_json, tmp_path / "flag", no_mature=True) + from_gff, _ = _run(without, nextclade_json, tmp_path / "gff") assert _records(from_flag) == _records(from_gff) @@ -219,8 +216,7 @@ def _hgvs_p(record: str) -> dict[str, str]: @pytest.fixture def mnv_records(tmp_path, dataset): json_path = _write_json(tmp_path / "mnv.json", positions=MNV_POSITIONS, qry=MNV_QRY) - out = tmp_path / "mnv.vcf" - _run(dataset, json_path, out) + out, _ = _run(dataset, json_path, tmp_path / "mnv") return _records(out) @@ -243,66 +239,61 @@ def test_lone_substitution_carries_no_mnv_fields(mnv_records): def test_mnv_fields_are_declared(tmp_path, dataset, nextclade_json): - out = tmp_path / "out.vcf" - _run(dataset, nextclade_json, out) + out, _ = _run(dataset, nextclade_json, tmp_path / "out") header = out.read_text() assert "##INFO= dict: + return {"seqName": name, "substitutions": [{"pos": 100, "refNuc": REF_SEQ[100], "qryNuc": "A"}], + "deletions": [], "insertions": [], "nucToAaMuts": {}, + "aaDeletions": [], "aaInsertions": [], "frameShifts": []} -def test_multi_sample_input_splits_both_outputs(tmp_path, dataset): - json_path = tmp_path / "multi.json" - json_path.write_text(json.dumps({ - "results": [ - {"seqName": name, "substitutions": [ - {"pos": 100, "refNuc": REF_SEQ[100], "qryNuc": "A"}], - "deletions": [], "insertions": [], "nucToAaMuts": {}, - "aaDeletions": [], "aaInsertions": [], "frameShifts": []} - for name in ("SAMPLE01", "SAMPLE02") - ], - })) +def test_batch_input_writes_one_named_vcf_per_sample(tmp_path, dataset): + json_path = tmp_path / "batch.json" + json_path.write_text(json.dumps({"results": [_result("SAMPLE01"), _result("SAMPLE02")]})) subprocess.run( [sys.executable, str(SCRIPT_PATH), "-i", str(json_path), "-d", str(dataset), - "-o", str(tmp_path / "out.vcf"), - "--output-no-mature", str(tmp_path / "out_nomat.vcf")], - check=True, capture_output=True, text=True, + "--suffix", "__REF__HA_nc_final.vcf", "--no-mature-suffix", "__REF__HA_nc_nomat_final.vcf"], + check=True, capture_output=True, text=True, cwd=tmp_path, ) for sample in ("SAMPLE01", "SAMPLE02"): - assert (tmp_path / f"out_{sample}.vcf").exists() - assert (tmp_path / f"out_nomat_{sample}.vcf").exists() - assert not _feature_ids(tmp_path / f"out_nomat_{sample}.vcf") & {"HA1", "HA2"} + standard = tmp_path / f"{sample}__REF__HA_nc_final.vcf" + assert len(_records(standard)) == 1 + assert f"FORMAT\t{sample}" in standard.read_text() + assert not _feature_ids(tmp_path / f"{sample}__REF__HA_nc_nomat_final.vcf") & {"HA1", "HA2"} -def test_unalignable_sample_writes_both_empty_vcfs(tmp_path, dataset): - json_path = tmp_path / "errors.json" +def test_unalignable_sample_gets_no_vcf(tmp_path, dataset): + json_path = tmp_path / "batch.json" json_path.write_text(json.dumps({ - "results": [], + "results": [_result("SAMPLE02")], "errors": [{"seqName": "SAMPLE01", "errors": ["Unable to align: seed alignment covers 3.0% of the query"]}], })) - standard = tmp_path / "out.vcf" - no_mature = tmp_path / "out_nomat.vcf" - _run(dataset, json_path, standard, no_mature) - assert _records(standard) == [] - assert _records(no_mature) == [] + _run(dataset, json_path, tmp_path / "out", no_mature=True) + assert sorted(p.name for p in (tmp_path / "out").glob("*.vcf")) == ["SAMPLE02.vcf", "SAMPLE02_nomat.vcf"] + + +def test_batch_where_nothing_aligned_writes_nothing(tmp_path, dataset): + json_path = tmp_path / "batch.json" + json_path.write_text(json.dumps({"results": [], "errors": [{"seqName": "SAMPLE01", "errors": ["x"]}]})) + _run(dataset, json_path, tmp_path / "out") + assert list((tmp_path / "out").glob("*.vcf")) == [] diff --git a/workflows/nextclade.nf b/workflows/nextclade.nf index 2da9828..fc46537 100644 --- a/workflows/nextclade.nf +++ b/workflows/nextclade.nf @@ -21,7 +21,6 @@ include { split_fasta as split_fasta_consensus } from '../modules/shared/consens include { consensus_identity_qc } from '../modules/nextclade/consensus_qc.nf' include { flumut } from '../modules/shared/flumut.nf' include { nextclade_run_batch } from '../modules/nextclade/nextclade.nf' -include { nextclade_to_vcf_batch } from '../modules/nextclade/nextclade.nf' include { coordinate_conversion } from '../modules/nextclade/coordinate_conversion.nf' include { merge_flumut_mutations } from '../modules/nextclade/merge_flumut_mutations.nf' include { mutation_watch } from '../modules/shared/mutation_watch.nf' @@ -115,25 +114,22 @@ workflow NEXTCLADE { return [segment, sample_ids, fastas, dataset_dir] } - // --- Step 4: Run nextclade once per segment for all samples --- - nextclade_run_batch(ch_segment_batches) - - // --- Step 5: Convert per-sample nextclade JSON to the final freebayes-style VCF --- + // --- Step 4: Run nextclade once per segment for all samples, and convert to VCF --- // With --emit_no_mature_vcf it also publishes each VCF without mature-peptide // (HA1/HA2) annotations. Nothing downstream reads that copy: mutation_watch and // merge_flumut_mutations resolve HA markers through the HA1/HA2 Feature_ID. - nextclade_to_vcf_batch(nextclade_run_batch.out.json_with_ref) + nextclade_run_batch(ch_segment_batches) // [segment, [vcfs]] -> [sample_id, segment, vcf]. Shared by the watchlist scan below // and, when enabled, by the flumut merge further down. - ch_final_vcf = nextclade_to_vcf_batch.out.vcf + ch_final_vcf = nextclade_run_batch.out.vcf .transpose() .map { segment, vcf -> def sample_id = vcf.name.split('__')[0] [sample_id, segment, vcf] } - // --- Step 6: Scan the final VCF for a custom watchlist (optional) --- + // --- Step 5: Scan the final VCF for a custom watchlist (optional) --- // Independent of --merge_flumut_mutations: FLUMUT= is a curated annotation written // into the VCF, this is a custom list scanned out of it into a separate TSV. if (params.watchlist_dir != 'NO_FILE') { @@ -145,18 +141,18 @@ workflow NEXTCLADE { mutation_watch(ch_final_vcf.combine(ch_watchlists)) } - // --- Steps 7-8: Coordinate conversion + flumut mutation merging (optional) --- + // --- Steps 6-7: Coordinate conversion + flumut mutation merging (optional) --- // Gated behind --merge_flumut_mutations (default false); otherwise the workflow // stops at the *_final.vcf produced above. if (params.merge_flumut_mutations) { - // --- Step 7: Build the flumut-to-Nextclade coordinate map --- + // --- Step 6: Build the flumut-to-Nextclade coordinate map --- // source = the flumutdb package's reference (lift FROM), target = the --db // datasets (lift ONTO). The target label comes from the bundle's db.yml, so the // published map names the reference rather than where the bundle sits. coordinate_conversion(Channel.value([db.nextclade_ref_name, db.nextclade])) - // --- Step 8: Annotate each per-sample/per-segment VCF with FluMutDB markers --- + // --- Step 7: Annotate each per-sample/per-segment VCF with FluMutDB markers --- // Additive: writes a separate *_flumut_annotated.vcf and leaves *_final.vcf // untouched. Run per (sample, segment) so each task stages only its own shard. ch_vcf_flat = ch_final_vcf // [sample_id, segment, vcf] From f8c213516f2e5293c8899508d2ad11bb6aa35bf8 Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Tue, 29 Sep 2026 16:58:07 -0400 Subject: [PATCH 03/11] Take consensus identity QC from Nextclade output nextclade_post_process.py derives identity from the batch TSV instead of realigning every segment with Biopython. On 519 test segments 169 match exactly and the mean shift is -0.09 points; a segment Nextclade cannot align is now FAIL with identity NA. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 + bin/consensus_identity_qc.py | 147 ---------------- bin/nextclade_post_process.py | 66 ++++++- modules/nextclade/consensus_qc.nf | 30 ---- modules/nextclade/nextclade.nf | 14 +- tests/test_consensus_identity_qc.py | 246 --------------------------- tests/test_nextclade_post_process.py | 135 +++++++++++++++ workflows/nextclade.nf | 12 +- 8 files changed, 216 insertions(+), 436 deletions(-) delete mode 100755 bin/consensus_identity_qc.py delete mode 100644 modules/nextclade/consensus_qc.nf delete mode 100644 tests/test_consensus_identity_qc.py create mode 100644 tests/test_nextclade_post_process.py diff --git a/README.md b/README.md index 4446cce..8203b84 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,7 @@ The short-read workflow performs trimming, assembly, mapping, LoFreq variant cal - Per-segment batching across samples - Per-segment `nextclade run` against built-in datasets - Per-sample JSON/TSV splitting from the batch results +- Consensus/reference identity QC from the Nextclade alignment - Conversion to freebayes-style VCF, with substitutions that share a codon annotated as one codon change - Detection of influenza molecular markers with [FluMut](https://izsvenezie-virology.github.io/FluMut/docs/usage/usage-cli) - *Optional* (`--merge_flumut_mutations`, default off): reference coordinate conversion — builds a run-level, idempotent nucleotide/amino-acid coordinate map lifting flumut's fixed reference onto the selected Nextclade reference dataset coordinate system (`bin/build_reference_coordinate_map.py`) — followed by FluMutDB annotation of the VCF, which tags each fluscan amino-acid call that matches a curated FluMutDB marker with a `FLUMUT=` INFO field carrying the mutation name and its published effects, split into those directly attributable to that mutation and those associated with it through a multi-mutation marker, each qualified by the subtype it was observed in. No rows are added or removed, so a marker is only reported where fluscan called a difference from the Nextclade reference. Because FluMutDB numbers HA by its HA1/HA2 subunits, HA markers are only annotated when the Nextclade dataset also exposes that split — either as two dedicated `HA1`/`HA2` CDS features, or as `mature_protein_region_of_CDS` children of a single HA CDS (the NCBI/RefSeq model, which the GsGD dataset uses). A dataset offering only a bare `HA` CDS is flagged `stale_annotation` and passes through with its HA rows unannotated @@ -272,6 +273,7 @@ The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder `FORMAT/GQ` as |------|-------------| | `nextclade/json/*_nextclade.json` | Per-segment Nextclade JSON split out from the cross-sample batch result | | `nextclade/tsv/*_nextclade.tsv` | Per-segment Nextclade TSV split out from the cross-sample batch result | +| `qc/*_identity_qc.tsv` | Per-segment identity of the consensus to the Nextclade dataset reference, taken from Nextclade's alignment: exact matches over the reference plus inserted bases, with Ns and ambiguous bases counted as mismatches. PASS/WARN/FAIL at `--consensus_identity_pass_pct` (default 90) and `--consensus_identity_warn_pct` (default 70); a segment Nextclade could not align is FAIL with `identity_pct` `NA` | | `nextclade/*_final.vcf` | Per-segment final VCF from the Nextclade-to-VCF conversion. Rows in a codon holding more than one substitution carry the combined change, plus `MNV_CORRECTED` and `MNV_ORIGINAL_HGVS_P` where it differs from the single-base call | | `nextclade/no_mature_peptide/*_nc_nomat_final.vcf` | *Optional* (`--emit_no_mature_vcf`): the same records annotated without mature peptides, so HA carries only its HA0 `ANN=` entry and no HA1/HA2 ones | | `nextclade/flumut_annotated/*_flumut_annotated.vcf` | *Optional* (`--merge_flumut_mutations`): the same VCF with a `FLUMUT=` INFO field (`Protein\|AA_Pos\|Alt_AA\|Mutation_Name\|DirectEffect\|AssociatedEffect`) on rows whose amino-acid change matches a curated FluMutDB marker | diff --git a/bin/consensus_identity_qc.py b/bin/consensus_identity_qc.py deleted file mode 100755 index f257492..0000000 --- a/bin/consensus_identity_qc.py +++ /dev/null @@ -1,147 +0,0 @@ -#!/usr/bin/env python3 -"""Report consensus-to-reference identity QC for a single influenza segment. - -Globally aligns one cleaned consensus segment to its selected reference segment -and writes a single-row TSV with the segment's identity and PASS/WARN/FAIL -status. One invocation handles one (sample, segment); cross-segment and -cross-sample aggregation is left to the caller. -""" - -import argparse -import sys -from pathlib import Path - -import pandas as pd -from Bio import Align, SeqIO - -OUTPUT_COLUMNS = ( - "sample_id", - "segment", - "status", - "identity_pct", - "consensus_length", - "reference_length", - "reference_id", -) - - -def parse_args(): - parser = argparse.ArgumentParser( - description="Globally align one cleaned consensus segment to its selected " - "reference and write a single-row identity QC TSV." - ) - parser.add_argument("--consensus", required=True, type=Path, help="Single-segment consensus FASTA") - parser.add_argument("--reference", required=True, type=Path, help="Single-segment reference FASTA") - parser.add_argument("--sample-id", required=True, help="Sample/isolate identifier") - parser.add_argument("--segment", required=True, help="Influenza segment label (e.g. HA)") - parser.add_argument("--pass-pct", required=True, type=float, help="PASS identity threshold") - parser.add_argument("--warn-pct", required=True, type=float, help="WARN identity threshold") - parser.add_argument("--out-tsv", required=True, type=Path, help="Output single-row TSV") - return parser.parse_args() - - -def load_single_sequence(fasta_path): - """Return (record_id, sequence) of the first record, or (None, "") if none.""" - if not fasta_path.exists() or fasta_path.stat().st_size == 0: - return None, "" - for record in SeqIO.parse(fasta_path, "fasta"): - return record.id, str(record.seq).upper() - return None, "" - - -def build_aligner(): - aligner = Align.PairwiseAligner() - aligner.mode = "global" - aligner.match_score = 1 - aligner.mismatch_score = -1 - aligner.open_gap_score = -2 - aligner.extend_gap_score = -0.5 - return aligner - - -def alignment_identity_pct(query, reference): - """Return exact-match identity across global alignment columns.""" - if not query or not reference: - return 0.0 - - aligner = build_aligner() - alignment = aligner.align(reference, query)[0] - coords = alignment.coordinates - - matches = 0 - columns = 0 - for idx in range(coords.shape[1] - 1): - ref_start, ref_end = coords[0][idx], coords[0][idx + 1] - qry_start, qry_end = coords[1][idx], coords[1][idx + 1] - ref_span = ref_end - ref_start - qry_span = qry_end - qry_start - block_columns = max(ref_span, qry_span) - columns += block_columns - - if ref_span and qry_span: - for offset in range(block_columns): - if reference[ref_start + offset] == query[qry_start + offset]: - matches += 1 - - if columns == 0: - return 0.0 - return matches / columns * 100 - - -def segment_status(identity_pct, pass_pct, warn_pct): - if identity_pct >= pass_pct: - return "PASS" - if identity_pct >= warn_pct: - return "WARN" - return "FAIL" - - -def build_qc_row(sample_id, segment, consensus, reference, pass_pct, warn_pct): - consensus_id, consensus_seq = consensus - reference_id, reference_seq = reference - - if not consensus_seq or not reference_seq: - identity = None - status = "FAIL" - else: - identity = alignment_identity_pct(consensus_seq, reference_seq) - status = segment_status(identity, pass_pct, warn_pct) - - return { - "sample_id": sample_id, - "segment": segment, - "status": status, - "identity_pct": identity, # float, or None when not evaluated - "consensus_length": len(consensus_seq), - "reference_length": len(reference_seq), - "reference_id": reference_id, # str, or None when reference missing - } - - -def write_tsv(row, out_tsv): - out_tsv.parent.mkdir(parents=True, exist_ok=True) - df = pd.DataFrame([row], columns=OUTPUT_COLUMNS) - df.to_csv(out_tsv, sep="\t", index=False, float_format="%.3f", na_rep="NA") - - -def main(): - args = parse_args() - consensus = load_single_sequence(args.consensus) - reference = load_single_sequence(args.reference) - if consensus[1] == "": - print(f"WARNING: no consensus sequence for {args.sample_id} {args.segment}", file=sys.stderr) - if reference[1] == "": - print(f"WARNING: no reference sequence for {args.sample_id} {args.segment}", file=sys.stderr) - row = build_qc_row( - sample_id=args.sample_id, - segment=args.segment, - consensus=consensus, - reference=reference, - pass_pct=args.pass_pct, - warn_pct=args.warn_pct, - ) - write_tsv(row, args.out_tsv) - - -if __name__ == "__main__": - main() diff --git a/bin/nextclade_post_process.py b/bin/nextclade_post_process.py index c9e07d8..ef1e49d 100755 --- a/bin/nextclade_post_process.py +++ b/bin/nextclade_post_process.py @@ -1,5 +1,6 @@ #!/usr/bin/env python3 -"""Split Nextclade batch JSON and TSV outputs into per-sample files. +"""Split Nextclade batch JSON and TSV outputs into per-sample files, and write each +sample's consensus/reference identity QC from Nextclade's alignment. Optionally writes a per-batch status summary (one row per sample) tracking whether each sample passed or failed Nextclade alignment. @@ -9,6 +10,8 @@ --json batch_nextclade.json \\ --tsv batch_nextclade.tsv \\ --segment HA \\ + --fasta batch.fasta --reference dataset/reference.fasta \\ + --pass-pct 90 --warn-pct 70 \\ --status-summary HA_nextclade_status.tsv """ @@ -48,6 +51,61 @@ def split_tsv(batch_tsv_path: Path, segment: str) -> None: writer.writerow(row) +IDENTITY_COLUMNS = ["sample_id", "segment", "status", "identity_pct", + "consensus_length", "reference_length", "reference_id"] + + +def fasta_lengths(path: Path) -> dict[str, int]: + """Return {record id: sequence length} for a FASTA file.""" + lengths: dict[str, int] = {} + name = None + with open(path) as fh: + for line in fh: + line = line.strip() + if line.startswith(">"): + name = line[1:].split()[0] + lengths[name] = 0 + elif name is not None: + lengths[name] += len(line) + return lengths + + +def identity_pct(row: dict, reference_length: int) -> float: + """Exact-match identity across the alignment columns: the reference plus insertions. + + alignmentStart/alignmentEnd are inclusive; Ns and ambiguous bases count as mismatches. + """ + n = {k: int(row[k] or 0) for k in ("alignmentStart", "alignmentEnd", "totalSubstitutions", + "totalDeletions", "totalInsertions", "totalMissing", + "totalNonACGTNs")} + matches = (n["alignmentEnd"] - n["alignmentStart"] + 1) - n["totalSubstitutions"] \ + - n["totalDeletions"] - n["totalMissing"] - n["totalNonACGTNs"] + return matches / (reference_length + n["totalInsertions"]) * 100 + + +def write_identity_qc(batch_tsv_path: Path, fasta_path: Path, reference_path: Path, + segment: str, pass_pct: float, warn_pct: float) -> None: + """Write one {seqName}_{segment}_identity_qc.tsv per TSV row; a sample Nextclade could + not align is FAIL with identity NA.""" + consensus_lengths = fasta_lengths(fasta_path) + (reference_id, reference_length), = fasta_lengths(reference_path).items() + + with open(batch_tsv_path, newline="") as fh: + for row in csv.DictReader(fh, delimiter="\t"): + sample_id = row["seqName"] + if row.get("errors", "").strip(): + identity, status = "NA", "FAIL" + else: + pct = identity_pct(row, reference_length) + identity = f"{pct:.3f}" + status = "PASS" if pct >= pass_pct else "WARN" if pct >= warn_pct else "FAIL" + with open(f"{sample_id}_{segment}_identity_qc.tsv", "w", newline="") as out_fh: + writer = csv.writer(out_fh, delimiter="\t", lineterminator="\n") + writer.writerow(IDENTITY_COLUMNS) + writer.writerow([sample_id, segment, status, identity, + consensus_lengths.get(sample_id, 0), reference_length, reference_id]) + + def write_status_summary(batch_tsv_path: Path, segment: str, summary_path: Path) -> None: """Write a per-sample status summary using the TSV as the authoritative roster. @@ -83,11 +141,17 @@ def main() -> None: p.add_argument("--json", required=True, type=Path, dest="json_path", help="Batch Nextclade JSON") p.add_argument("--tsv", required=True, type=Path, dest="tsv_path", help="Batch Nextclade TSV") p.add_argument("--segment", required=True, dest="segment", help="Segment name (e.g. HA)") + p.add_argument("--fasta", required=True, type=Path, help="Batch FASTA Nextclade aligned") + p.add_argument("--reference", required=True, type=Path, help="Dataset reference.fasta") + p.add_argument("--pass-pct", required=True, type=float, help="PASS identity threshold") + p.add_argument("--warn-pct", required=True, type=float, help="WARN identity threshold") p.add_argument("--status-summary", type=Path, default=None, help="Path for per-sample status summary TSV") args = p.parse_args() split_json(args.json_path, args.segment) split_tsv(args.tsv_path, args.segment) + write_identity_qc(args.tsv_path, args.fasta, args.reference, args.segment, + args.pass_pct, args.warn_pct) if args.status_summary: write_status_summary(args.tsv_path, args.segment, args.status_summary) diff --git a/modules/nextclade/consensus_qc.nf b/modules/nextclade/consensus_qc.nf deleted file mode 100644 index 4831139..0000000 --- a/modules/nextclade/consensus_qc.nf +++ /dev/null @@ -1,30 +0,0 @@ -// nextclade/consensus_qc.nf - Consensus/reference identity QC for the NEXTCLADE workflow - -// Report-only consensus/reference identity QC for a single segment. -process consensus_identity_qc { - tag { "${sample_id}|${segment}" } - - errorStrategy 'ignore' - - conda "${projectDir}/environments/fluscan.yml" - - publishDir "${params.outdir}/${sample_id}/qc", mode: 'copy', pattern: "*.tsv" - - input: - tuple val(sample_id), val(segment), path(consensus_fasta), path(reference_fasta) - - output: - tuple val(sample_id), val(segment), path("*_identity_qc.tsv"), emit: tsv - - script: - """ - consensus_identity_qc.py \ - --consensus ${consensus_fasta} \ - --reference ${reference_fasta} \ - --sample-id ${sample_id} \ - --segment ${segment} \ - --pass-pct ${params.consensus_identity_pass_pct} \ - --warn-pct ${params.consensus_identity_warn_pct} \ - --out-tsv ${sample_id}_${segment}_identity_qc.tsv - """ -} diff --git a/modules/nextclade/nextclade.nf b/modules/nextclade/nextclade.nf index 21b7060..28c5c90 100644 --- a/modules/nextclade/nextclade.nf +++ b/modules/nextclade/nextclade.nf @@ -1,8 +1,9 @@ // nextclade/nextclade.nf - Per-segment Nextclade alignment and VCF conversion processes // Cross-sample per-segment batch: runs nextclade once for all samples' sequences for a -// given segment, splits the output into per-sample JSON and TSV files, and converts it to -// one final VCF per sample. A sample Nextclade cannot align gets no JSON and no VCF. +// given segment, splits the output into per-sample JSON, TSV and identity QC files, and +// converts it to one final VCF per sample. A sample Nextclade cannot align gets no JSON and +// no VCF, and an identity QC row of FAIL. process nextclade_run_batch { conda "${projectDir}/environments/nextclade.yml" @@ -17,6 +18,10 @@ process nextclade_run_batch { def subdir = filename.endsWith('_nc_nomat_final.vcf') ? 'no_mature_peptide/' : '' return "${sample_id}/${workflow}/${subdir}${filename}" } + if (filename.endsWith('_identity_qc.tsv')) { + def sample_id = (filename =~ /^[^_]+/)[0] + return "${sample_id}/qc/${filename}" + } if (filename =~ /.*_nextclade_status\.tsv$/) { def workflow = task.ext.workflow ?: 'unknown_workflow' return "${workflow}/status/${filename}" @@ -45,6 +50,7 @@ process nextclade_run_batch { tuple val(segment), path("*_nc_nomat_final.vcf"), optional: true, emit: vcf_no_mature tuple val(segment), path("*_nextclade.json"), optional: true, emit: json tuple val(segment), path("*_nextclade.tsv"), emit: tsv + tuple val(segment), path("*_identity_qc.tsv"), emit: identity_qc tuple val(segment), path("nc_output"), emit: all tuple val(segment), path("${segment}_nextclade_status.tsv"), emit: status @@ -75,6 +81,10 @@ process nextclade_run_batch { --json nc_output/batch.json \\ --tsv nc_output/batch.tsv \\ --segment ${segment} \\ + --fasta batch.fasta \\ + --reference ${dataset_dir}/reference.fasta \\ + --pass-pct ${params.consensus_identity_pass_pct} \\ + --warn-pct ${params.consensus_identity_warn_pct} \\ --status-summary ${segment}_nextclade_status.tsv REFERENCE_NAME=\$(head -n1 ${dataset_dir}/reference.fasta | sed -E 's/>(\\S+).*/\\1/g') diff --git a/tests/test_consensus_identity_qc.py b/tests/test_consensus_identity_qc.py deleted file mode 100644 index 2ab0b57..0000000 --- a/tests/test_consensus_identity_qc.py +++ /dev/null @@ -1,246 +0,0 @@ -"""Unit tests for bin/consensus_identity_qc.py (per-segment identity QC). - -Covers the QC scenarios required by the consensus-identity QC plan: PASS, WARN, -FAIL, a missing/empty consensus segment, a missing/empty reference segment, -ambiguous bases, single-record FASTA loading, and the end-to-end CLI (TSV -schema + single data row). - -Identity in the controlled tests is driven by replacing reference bases with -ambiguous ``N`` rather than by substituting other bases. ``N`` never -exact-matches a base, and the surrounding bases stay matched in place, so the -ungapped global alignment is optimal and identity is exactly ``(L - k) / L``. -Substituting real bases instead lets the global aligner recover spurious -matches via gaps, which makes identity non-deterministic. - -Run with: pytest tests/test_consensus_identity_qc.py -""" - -import csv -import importlib.util -import random -import subprocess -import sys -from pathlib import Path - -import pytest - -REPO_ROOT = Path(__file__).resolve().parents[1] -SCRIPT_PATH = REPO_ROOT / "bin" / "consensus_identity_qc.py" - -PASS_PCT = 90.0 -WARN_PCT = 70.0 - - -def _load_module(): - spec = importlib.util.spec_from_file_location("consensus_identity_qc", SCRIPT_PATH) - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -qc = _load_module() - -# A deterministic, non-repetitive 300 bp reference used across identity tests. -REF = "".join(random.Random(42).choice("ACGT") for _ in range(300)) - - -def with_ns(seq, k): - """Return seq with the first k bases replaced by ambiguous N. - - Yields identity exactly (len(seq) - k) / len(seq) against the original seq. - """ - return "N" * k + seq[k:] - - -def write_fasta(path, records): - """records: list of (header, sequence). Empty list -> empty file.""" - with open(path, "w") as handle: - for header, seq in records: - handle.write(f">{header}\n{seq}\n") - return path - - -# -------------------------------------------------------------------------- -# alignment_identity_pct: core scoring -# -------------------------------------------------------------------------- - -def test_identity_identical_is_100(): - assert qc.alignment_identity_pct(REF, REF) == pytest.approx(100.0) - - -def test_identity_80pct(): - # 60 ambiguous bases over 300 columns -> exactly 80% identity. - assert qc.alignment_identity_pct(with_ns(REF, 60), REF) == pytest.approx(80.0) - - -def test_identity_50pct(): - assert qc.alignment_identity_pct(with_ns(REF, 150), REF) == pytest.approx(50.0) - - -def test_identity_empty_sequence_is_zero(): - assert qc.alignment_identity_pct("", REF) == 0.0 - assert qc.alignment_identity_pct(REF, "") == 0.0 - - -def test_ambiguous_bases_count_as_mismatch(): - # 75 N substitutions -> 75% identity (N never exact-matches a base) -> WARN. - identity = qc.alignment_identity_pct(with_ns(REF, 75), REF) - assert identity == pytest.approx(75.0) - assert qc.segment_status(identity, PASS_PCT, WARN_PCT) == "WARN" - - -# -------------------------------------------------------------------------- -# segment_status thresholds -# -------------------------------------------------------------------------- - -@pytest.mark.parametrize( - "identity,expected", - [(100.0, "PASS"), (90.0, "PASS"), (89.999, "WARN"), (70.0, "WARN"), (69.999, "FAIL"), (0.0, "FAIL")], -) -def test_segment_status_thresholds(identity, expected): - assert qc.segment_status(identity, PASS_PCT, WARN_PCT) == expected - - -# -------------------------------------------------------------------------- -# load_single_sequence -# -------------------------------------------------------------------------- - -def test_load_single_sequence_reads_first_record(tmp_path): - path = write_fasta(tmp_path / "c.fasta", [("SAMPLE_HA", REF)]) - rec_id, seq = qc.load_single_sequence(path) - assert rec_id == "SAMPLE_HA" - assert seq == REF - - -def test_load_single_sequence_empty_file(tmp_path): - path = write_fasta(tmp_path / "empty.fasta", []) - rec_id, seq = qc.load_single_sequence(path) - assert rec_id is None - assert seq == "" - - -def test_load_single_sequence_missing_file(tmp_path): - rec_id, seq = qc.load_single_sequence(tmp_path / "does_not_exist.fasta") - assert rec_id is None - assert seq == "" - - -# -------------------------------------------------------------------------- -# build_qc_row: PASS / WARN / FAIL / missing -# -------------------------------------------------------------------------- - -def build_row(consensus, reference, segment="HA"): - return qc.build_qc_row( - sample_id="SAMPLE", - segment=segment, - consensus=consensus, - reference=reference, - pass_pct=PASS_PCT, - warn_pct=WARN_PCT, - ) - - -def test_row_pass(): - row = build_row(("SAMPLE_HA", REF), ("REF_HA", REF)) - assert row["status"] == "PASS" - assert row["identity_pct"] == pytest.approx(100.0) - assert row["segment"] == "HA" - assert row["consensus_length"] == 300 - assert row["reference_length"] == 300 - assert row["reference_id"] == "REF_HA" - - -def test_row_warn(): - row = build_row(("SAMPLE_HA", with_ns(REF, 60)), ("REF_HA", REF)) - assert row["status"] == "WARN" - assert row["identity_pct"] == pytest.approx(80.0) - - -def test_row_fail_low_identity(): - row = build_row(("SAMPLE_HA", with_ns(REF, 150)), ("REF_HA", REF)) - assert row["status"] == "FAIL" - assert row["identity_pct"] == pytest.approx(50.0) - - -def test_row_missing_consensus(): - row = build_row((None, ""), ("REF_HA", REF)) - assert row["status"] == "FAIL" - assert row["identity_pct"] is None - assert row["consensus_length"] == 0 - assert row["reference_length"] == 300 - - -def test_row_missing_reference(): - row = build_row(("SAMPLE_HA", REF), (None, "")) - assert row["status"] == "FAIL" - assert row["identity_pct"] is None - assert row["consensus_length"] == 300 - assert row["reference_length"] == 0 - assert row["reference_id"] is None - - -# -------------------------------------------------------------------------- -# End-to-end CLI: schema + single data row via subprocess -# -------------------------------------------------------------------------- - -def run_cli(tmp_path, consensus_records, reference_records, segment="HA"): - consensus = write_fasta(tmp_path / "consensus.fasta", consensus_records) - reference = write_fasta(tmp_path / "reference.fasta", reference_records) - out_tsv = tmp_path / "out.tsv" - subprocess.run( - [ - sys.executable, str(SCRIPT_PATH), - "--consensus", str(consensus), - "--reference", str(reference), - "--sample-id", "SAMPLE", - "--segment", segment, - "--pass-pct", str(PASS_PCT), - "--warn-pct", str(WARN_PCT), - "--out-tsv", str(out_tsv), - ], - check=True, - ) - with open(out_tsv) as handle: - rows = list(csv.DictReader(handle, delimiter="\t")) - return rows - - -def test_cli_schema_and_single_row(tmp_path): - rows = run_cli(tmp_path, [("SAMPLE_HA", REF)], [("REF_HA", REF)]) - assert len(rows) == 1 - row = rows[0] - assert list(row.keys()) == list(qc.OUTPUT_COLUMNS) - assert row["sample_id"] == "SAMPLE" - assert row["segment"] == "HA" - assert row["status"] == "PASS" - assert row["identity_pct"] == "100.000" - assert row["reference_id"] == "REF_HA" - - -@pytest.mark.parametrize( - "n_ns,expected_status,expected_pct", - [(0, "PASS", "100.000"), (60, "WARN", "80.000"), (150, "FAIL", "50.000")], -) -def test_cli_pass_warn_fail(tmp_path, n_ns, expected_status, expected_pct): - rows = run_cli(tmp_path, [("SAMPLE_NA", with_ns(REF, n_ns))], [("REF_NA", REF)], segment="NA") - assert rows[0]["status"] == expected_status - assert rows[0]["identity_pct"] == expected_pct - - -def test_cli_missing_consensus_segment(tmp_path): - # Empty consensus FASTA -> still writes a row, FAIL/NA (report-only). - rows = run_cli(tmp_path, [], [("REF_PB2", REF)], segment="PB2") - assert rows[0]["status"] == "FAIL" - assert rows[0]["identity_pct"] == "NA" - assert rows[0]["consensus_length"] == "0" - - -def test_cli_missing_reference_segment(tmp_path): - rows = run_cli(tmp_path, [("SAMPLE_NS", REF)], [], segment="NS") - assert rows[0]["status"] == "FAIL" - assert rows[0]["identity_pct"] == "NA" - assert rows[0]["reference_length"] == "0" - - -if __name__ == "__main__": - sys.exit(pytest.main([__file__, "-v"])) diff --git a/tests/test_nextclade_post_process.py b/tests/test_nextclade_post_process.py new file mode 100644 index 0000000..c09925d --- /dev/null +++ b/tests/test_nextclade_post_process.py @@ -0,0 +1,135 @@ +"""Unit tests for bin/nextclade_post_process.py: per-sample splits and identity QC.""" + +import csv +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +import nextclade_post_process as npp + +SCRIPT_PATH = Path(__file__).resolve().parents[1] / "bin" / "nextclade_post_process.py" + +TSV_FIELDS = ["index", "seqName", "alignmentStart", "alignmentEnd", "totalSubstitutions", + "totalDeletions", "totalInsertions", "totalMissing", "totalNonACGTNs", "errors"] + + +def _row(name, start=1, end=100, subs=0, dels=0, ins=0, missing=0, non_acgtn=0, errors=""): + return {"index": 0, "seqName": name, "alignmentStart": start, "alignmentEnd": end, + "totalSubstitutions": subs, "totalDeletions": dels, "totalInsertions": ins, + "totalMissing": missing, "totalNonACGTNs": non_acgtn, "errors": errors} + + +def _as_tsv_row(row): + return {k: str(v) for k, v in row.items()} + + +# --------------------------------------------------------------------------- +# identity_pct +# --------------------------------------------------------------------------- + +def test_full_length_identical_is_100(): + assert npp.identity_pct(_as_tsv_row(_row("S")), 100) == 100.0 + + +def test_substitutions_missing_and_ambiguous_bases_are_mismatches(): + row = _as_tsv_row(_row("S", subs=5, missing=3, non_acgtn=2)) + assert npp.identity_pct(row, 100) == 90.0 + + +def test_insertions_widen_the_alignment(): + row = _as_tsv_row(_row("S", ins=25)) + assert npp.identity_pct(row, 100) == 80.0 + + +def test_deletions_and_unaligned_ends_are_mismatches(): + # 10 reference bases before the query starts and 10 deleted inside it. + row = _as_tsv_row(_row("S", start=11, end=100, dels=10)) + assert npp.identity_pct(row, 100) == 80.0 + + +def test_blank_counts_read_as_zero(): + row = _as_tsv_row(_row("S")) + row["totalMissing"] = "" + assert npp.identity_pct(row, 100) == 100.0 + + +# --------------------------------------------------------------------------- +# write_identity_qc +# --------------------------------------------------------------------------- + +@pytest.fixture +def batch(tmp_path): + rows = [ + _row("PASSER"), + _row("WARNER", subs=20), + _row("FAILER", subs=40), + _row("BOUNDARY", subs=10), + {**_row("UNALIGNED"), "alignmentStart": "", "alignmentEnd": "", + "errors": "Unable to align"}, + ] + tsv = tmp_path / "batch.tsv" + with open(tsv, "w", newline="") as fh: + writer = csv.DictWriter(fh, fieldnames=TSV_FIELDS, delimiter="\t") + writer.writeheader() + writer.writerows(rows) + fasta = tmp_path / "batch.fasta" + fasta.write_text("".join(f">{r['seqName']}\n{'A' * 60}\n{'A' * 38}\n" for r in rows)) + reference = tmp_path / "reference.fasta" + reference.write_text(">REF1 some description\n" + "A" * 100 + "\n") + return tsv, fasta, reference + + +def _identity_rows(tmp_path, monkeypatch, batch): + monkeypatch.chdir(tmp_path) + npp.write_identity_qc(*batch, "HA", pass_pct=90, warn_pct=70) + return {p.name.split("_")[0]: next(csv.DictReader(open(p), delimiter="\t")) + for p in tmp_path.glob("*_HA_identity_qc.tsv")} + + +def test_statuses_follow_the_thresholds(tmp_path, monkeypatch, batch): + rows = _identity_rows(tmp_path, monkeypatch, batch) + assert {name: row["status"] for name, row in rows.items()} == { + "PASSER": "PASS", "WARNER": "WARN", "FAILER": "FAIL", "BOUNDARY": "PASS", "UNALIGNED": "FAIL", + } + + +def test_row_schema_and_values(tmp_path, monkeypatch, batch): + row = _identity_rows(tmp_path, monkeypatch, batch)["WARNER"] + assert list(row) == npp.IDENTITY_COLUMNS + assert row == {"sample_id": "WARNER", "segment": "HA", "status": "WARN", + "identity_pct": "80.000", "consensus_length": "98", + "reference_length": "100", "reference_id": "REF1"} + + +def test_unaligned_sample_has_no_identity(tmp_path, monkeypatch, batch): + assert _identity_rows(tmp_path, monkeypatch, batch)["UNALIGNED"]["identity_pct"] == "NA" + + +# --------------------------------------------------------------------------- +# CLI: splits, status summary and identity QC together +# --------------------------------------------------------------------------- + +def test_cli_writes_every_per_sample_output(tmp_path, batch): + tsv, fasta, reference = batch + results = [{"seqName": name} for name in ("PASSER", "WARNER", "FAILER", "BOUNDARY")] + (tmp_path / "batch.json").write_text(json.dumps({ + "results": results, "errors": [{"seqName": "UNALIGNED", "errors": ["Unable to align"]}], + })) + out = tmp_path / "out" + out.mkdir() + subprocess.run( + [sys.executable, str(SCRIPT_PATH), "--json", str(tmp_path / "batch.json"), "--tsv", str(tsv), + "--segment", "HA", "--fasta", str(fasta), "--reference", str(reference), + "--pass-pct", "90", "--warn-pct", "70", "--status-summary", "HA_nextclade_status.tsv"], + check=True, capture_output=True, text=True, cwd=out, + ) + names = {p.name for p in out.iterdir()} + assert "UNALIGNED_HA_nextclade.json" not in names + assert "UNALIGNED_HA_nextclade.tsv" in names + assert "UNALIGNED_HA_identity_qc.tsv" in names + assert {f"{s}_HA_nextclade.json" for s in ("PASSER", "WARNER", "FAILER", "BOUNDARY")} <= names + status = {r["sample"]: r["status"] for r in csv.DictReader(open(out / "HA_nextclade_status.tsv"), delimiter="\t")} + assert status["UNALIGNED"] == "fail" and status["PASSER"] == "pass" diff --git a/workflows/nextclade.nf b/workflows/nextclade.nf index fc46537..7f805e4 100644 --- a/workflows/nextclade.nf +++ b/workflows/nextclade.nf @@ -18,7 +18,6 @@ include { hash_files } from '../modules/shared/hash_files.nf' include { prepare_consensus } from '../modules/shared/consensus_qc.nf' include { split_fasta as split_fasta_consensus } from '../modules/shared/consensus_qc.nf' -include { consensus_identity_qc } from '../modules/nextclade/consensus_qc.nf' include { flumut } from '../modules/shared/flumut.nf' include { nextclade_run_batch } from '../modules/nextclade/nextclade.nf' include { coordinate_conversion } from '../modules/nextclade/coordinate_conversion.nf' @@ -89,19 +88,11 @@ workflow NEXTCLADE { sort: false ) - // Split once; reused by the identity QC and the nextclade batching below. + // Split into per-segment FASTAs for the nextclade batching below. ch_consensus_segments = split_fasta_consensus(prepare_consensus.out.cleaned) .transpose() .map(extract_segment) // [sample_id, segment, fa] - // --- Step 1b: Report-only consensus/reference identity QC --- - ch_identity_qc = ch_consensus_segments - .map { sample_id, segment, consensus -> - def ref_fasta = db.nextclade.resolve("${segment}/reference.fasta") - return [sample_id, segment, consensus, ref_fasta] - } - consensus_identity_qc(ch_identity_qc) - // --- Step 2: HPAI mutation screening (per sample, parallel branch) --- flumut(prepare_consensus.out.cleaned) @@ -115,6 +106,7 @@ workflow NEXTCLADE { } // --- Step 4: Run nextclade once per segment for all samples, and convert to VCF --- + // Also writes each sample's report-only consensus/reference identity QC. // With --emit_no_mature_vcf it also publishes each VCF without mature-peptide // (HA1/HA2) annotations. Nothing downstream reads that copy: mutation_watch and // merge_flumut_mutations resolve HA markers through the HA1/HA2 Feature_ID. From e43c62b2ec471ac5b087ac6f1e1bbf67ccf0900b Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Tue, 29 Sep 2026 17:01:19 -0400 Subject: [PATCH 04/11] Build Nextclade segment batches with channel operators splitFasta and collectFile write each segment's batch FASTA in the head job, replacing a split_fasta task per sample and the header rewrite loop in nextclade_run_batch. Co-Authored-By: Claude Opus 5.5 --- README.md | 6 ++---- modules/nextclade/nextclade.nf | 13 +------------ workflows/nextclade.nf | 31 ++++++++++--------------------- 3 files changed, 13 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 8203b84..3e11501 100644 --- a/README.md +++ b/README.md @@ -86,8 +86,7 @@ flowchart TD direction TB nextclade_prepare[prepare_consensus] nextclade_flumut[flumut] - split_consensus[split_fasta] - groupseg[group by segment across samples] + groupseg[collect each segment across samples: splitFasta + collectFile] nc_run[nextclade run batch] split_results[split per-sample JSON and TSV] nc2vcf[nextclade_to_freebayes_vcf.py] @@ -98,8 +97,7 @@ flowchart TD flumut_annotated[FluMutDB-annotated VCFs] nextclade_prepare --> nextclade_flumut - nextclade_prepare --> split_consensus - split_consensus --> groupseg + nextclade_prepare --> groupseg groupseg --> nc_run nc_run --> split_results nc_run --> nc2vcf diff --git a/modules/nextclade/nextclade.nf b/modules/nextclade/nextclade.nf index 28c5c90..288cdaa 100644 --- a/modules/nextclade/nextclade.nf +++ b/modules/nextclade/nextclade.nf @@ -41,9 +41,7 @@ process nextclade_run_batch { } input: - tuple val(segment), val(sample_ids), - path(consensus_fastas, stageAs: 'inputs/*'), - path(dataset_dir) + tuple val(segment), path(batch_fasta, stageAs: 'batch.fasta'), path(dataset_dir) output: tuple val(segment), path("*_nc_final.vcf"), optional: true, emit: vcf @@ -59,15 +57,6 @@ process nextclade_run_batch { def stem = "__\${REFERENCE_NAME}__\${SUBTYPE}__${segment}" def no_mature_arg = params.emit_no_mature_vcf ? "--no-mature-suffix \"${stem}_nc_nomat_final.vcf\"" : "" """ - # Rewrite each segment FASTA header to exactly the sample_id so nextclade - # seqName matches sample_id in the result JSON (enables per-sample demux). - mkdir -p prepared - for fasta_file in inputs/*.s.fasta; do - sample_id=\$(basename "\$fasta_file" .s.fasta | sed -E 's/_[^_]+\$//') - awk -v sample_id="\$sample_id" '/^>/{print ">"sample_id; next}{print}' "\$fasta_file" > "prepared/\${sample_id}.fasta" - done - cat prepared/*.fasta > batch.fasta - nextclade run \\ --input-dataset ${dataset_dir} \\ --output-all nc_output \\ diff --git a/workflows/nextclade.nf b/workflows/nextclade.nf index 7f805e4..0b99ab4 100644 --- a/workflows/nextclade.nf +++ b/workflows/nextclade.nf @@ -17,7 +17,6 @@ include { hash_files } from '../modules/shared/hash_files.nf' include { prepare_consensus } from '../modules/shared/consensus_qc.nf' -include { split_fasta as split_fasta_consensus } from '../modules/shared/consensus_qc.nf' include { flumut } from '../modules/shared/flumut.nf' include { nextclade_run_batch } from '../modules/nextclade/nextclade.nf' include { coordinate_conversion } from '../modules/nextclade/coordinate_conversion.nf' @@ -53,14 +52,6 @@ workflow NEXTCLADE { ch_pipeline_provenance = pipeline_provenance(ch_workflow_metadata) - // A name that does not match yields 'unknown_segment' rather than throwing, so the - // unknown-segment branch below can report it. - def extract_segment = { sample_id, fasta -> - def matcher = fasta.name =~ /.*_([A-Za-z0-9-]+)\.s.*/ - def segment = matcher ? matcher[0][1] : "unknown_segment" - return [sample_id, segment, fasta] - } - // Consensus FASTAs from the sample sheet or the input directory. // Each file is expected to contain 8 segments for a single sample. ch_fasta_input = params.samplesheet_input != 'NO_FILE' @@ -88,22 +79,20 @@ workflow NEXTCLADE { sort: false ) - // Split into per-segment FASTAs for the nextclade batching below. - ch_consensus_segments = split_fasta_consensus(prepare_consensus.out.cleaned) - .transpose() - .map(extract_segment) // [sample_id, segment, fa] - // --- Step 2: HPAI mutation screening (per sample, parallel branch) --- flumut(prepare_consensus.out.cleaned) - // --- Step 3: Group per-segment FASTAs across all samples for batching --- - ch_segment_batches = ch_consensus_segments - .map { sample_id, segment, fa -> [segment, sample_id, fa] } - .groupTuple() // [segment, [sample_ids], [fastas]] - .map { segment, sample_ids, fastas -> - def dataset_dir = db.nextclade.resolve(segment) - return [segment, sample_ids, fastas, dataset_dir] + // --- Step 3: Collect each segment across all samples into one FASTA --- + // Cleaned headers are {sample_id}_{SEGMENT} with a canonical SEGMENT; the batch + // header is the bare sample_id, which nextclade reports back as seqName. + ch_segment_batches = prepare_consensus.out.cleaned + .splitFasta(elem: 1, record: [id: true, seqString: true]) // [sample_id, record] + .collectFile { item -> + def (sample_id, record) = item + ["${record.id - "${sample_id}_"}.fasta", ">${sample_id}\n${record.seqString}\n"] } + .map { fasta -> [fasta.baseName, fasta, db.nextclade.resolve(fasta.baseName)] } + // -> [segment, batch.fasta, dataset_dir] // --- Step 4: Run nextclade once per segment for all samples, and convert to VCF --- // Also writes each sample's report-only consensus/reference identity QC. From c09c528dad0afc8b55b18a3aa3498118e4e600c9 Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Tue, 29 Sep 2026 17:33:25 -0400 Subject: [PATCH 05/11] Scan watchlists per sample and merge FluMut per segment mutation_watch runs once per sample over all its VCFs and watchlists, and merge_flumut_mutations.py annotates a whole segment in one call. Published outputs are unchanged. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 9 +- bin/merge_flumut_mutations.py | 135 +++++++++++--------- docs/vcf_annotation_and_flumut_insertion.md | 19 ++- modules/nextclade/merge_flumut_mutations.nf | 50 ++++---- modules/shared/database.nf | 7 - modules/shared/mutation_watch.nf | 67 +++++----- tests/test_merge_flumut_mutations.py | 24 +++- workflows/nextclade.nf | 70 ++++------ workflows/paftools.nf | 25 ++-- workflows/short-read.nf | 13 +- 10 files changed, 210 insertions(+), 209 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c544893..46cb345 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -62,7 +62,7 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and | `docs/watchlists.md` | Custom watchlist schema, matching rules, and the HA1/HA numbering trap | | `assets/watchlists/` | The shipped `*watchlist.csv` files; `--watchlist_dir` defaults here | | `modules/shared/sample_id.nf` | The only place a consensus or sample sheet `sample_id` is derived; holds the FASTA discovery channel both FASTA branches call and `samplesheet_channel`, which all three workflows call for `--samplesheet_input`. Changing `clean_sample_id` renames every output | -| `modules/shared/database.nf` | The only place that knows the `--db` bundle layout; `resolve_db` validates it up front. Also holds the canonical segment list — `@Field DB_SEGMENTS` is not importable across an `include`, so workflows reach it through the `db_segments()` accessor rather than repeating the literal | +| `modules/shared/database.nf` | The only place that knows the `--db` bundle layout; `resolve_db` validates it up front. Also holds the canonical segment list, `@Field DB_SEGMENTS`, which it uses to check each segment's dataset | | `scripts/validate_db.py` | Deeper, contents-level bundle check; run after building or moving a bundle | | `docs/database_bundles.md` | Bundle layout, `db.yml` schema, how to build and validate one | | `README.md` | User-facing inputs, commands, outputs, and architecture | @@ -284,9 +284,10 @@ published filenames/content. Use `-profile conda`; `conda.useMamba` is enabled. without the label one branch silently overwrites the other. Any new workflow needs a `withName: ':.*' { ext.workflow = … }` block or its records fall back to the unlabelled, collision-prone name. -- `mutation_watch` emits a provenance YAML that SHORT_READ does not join in, so - its watchlist scan is absent from the collected record. Both FASTA branches - join it, with `groupTuple()` and `remainder: true`. +- `mutation_watch` runs once per sample over all of its final VCFs and every + watchlist, and emits one provenance YAML per sample. SHORT_READ does not join + it in, so its watchlist scan is absent from the collected record. Both FASTA + branches join it with `remainder: true`. - The two FASTA branches are deliberate mirrors from input discovery through `prepare_consensus`, `flumut`, the consensus QC aggregate, and the shape of the provenance chain and watchlist block. Only the analysis between those diff --git a/bin/merge_flumut_mutations.py b/bin/merge_flumut_mutations.py index 8d2ac71..bee04c7 100755 --- a/bin/merge_flumut_mutations.py +++ b/bin/merge_flumut_mutations.py @@ -16,8 +16,8 @@ Nextclade dataset through the coordinate_conversion map. FluMutDB numbers HA by its HA1/HA2 subunits, so HA markers match only where the dataset exposes that split. -Pipeline position: after post_process_vcf_batch (nextclade workflow), consuming -coordinate_conversion output. See docs/vcf_annotation_and_flumut_insertion.md. +Pipeline position: after nextclade_run_batch, once per segment over every sample's VCF, +consuming coordinate_conversion output. See docs/vcf_annotation_and_flumut_insertion.md. """ import argparse @@ -362,71 +362,53 @@ def get_args() -> argparse.Namespace: p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) # The process's provenance block reads this, so tool_version cannot drift. p.add_argument("--version", action="version", version=f"%(prog)s {SCRIPT_VERSION}") - p.add_argument("--vcf", required=True, help="Existing per-sample/per-segment VCF to annotate") + p.add_argument("--vcf", required=True, nargs="+", + help="Per-sample VCFs for this segment; each is written to " + "_flumut_annotated.vcf in the working directory, the stem being " + "its filename without .vcf and a trailing _final") p.add_argument("--coord-map", required=True, help="coordinate_map..json (flumut -> Nextclade)") p.add_argument("--segment", required=True, help="Segment label, e.g. HA, M, NS") - p.add_argument("--sample", required=True, help="Sample id (used for logging only)") p.add_argument("--flumut-db", help="Path to flumut_db.sqlite (default: the installed flumutdb)") p.add_argument("--no-effects", action="store_true", help="Omit both effect fields, emitting only Protein|AA_Pos|Alt_AA|Mutation_Name") - p.add_argument("-o", "--output", required=True, help="Output VCF path") return p.parse_args() -def main() -> None: - args = get_args() - segment = args.segment.upper() - want_effects = not args.no_effects - - df, header_lines = read_vcf(args.vcf) - - seg_map = load_segment_map(args.coord_map, segment) - genes_map = seg_map.get("genes", {}) - alias_index = build_alias_index(genes_map) - subunit_parent_genes = set(seg_map.get("subunit_parent_genes", [])) - if seg_map.get("stale_annotation"): - log.warning("Coordinate map flags segment %s as stale (the Nextclade dataset does not " - "expose the HA1/HA2 split FluMutDB requires); HA markers cannot be annotated", - segment) - - db_path = locate_db(args.flumut_db) - # The packaged DB never changes; immutable skips the file locking that fails on network mounts. - conn = sqlite3.connect(f"file:{db_path}?immutable=1", uri=True) - try: - version = db_version(conn) - db_ref = SEGMENT_TO_DB_REFERENCE.get(segment, segment) - gene_maps = build_gene_protein_maps(conn, db_ref, genes_map) - lookup = make_marker_lookup(conn, db_ref) - - # Annotate in place: rows, order, POS, REF/ALT and ANN are all preserved; only INFO - # grows, and only on rows that matched. - info_out: list[str] = [] - annotated = 0 - markers: set = set() - unmapped_genes: set = set() - unparsed_hgvs_p: list[str] = [] - mnv_unresolved = 0 - for info in df["INFO"].astype(str): - if has_info_flag(info, "MNV_UNRESOLVED"): - # fix_mnv_codons.py left this row's HGVS.p at a value already determined to - # be wrong; matching a curated marker against it would be a false positive. - mnv_unresolved += 1 - info_out.append(info) - continue - field = row_flumut_field(info, gene_maps, lookup, want_effects, - alias_index=alias_index, - subunit_parent_genes=subunit_parent_genes, - unmapped_genes=unmapped_genes, - unparsed_hgvs_p=unparsed_hgvs_p) - if field: - annotated += 1 - for entry in field[len("FLUMUT="):].split(","): - markers.update(entry.split("|")[3].split("&")) - info_out.append(f"{info};{field}") - else: - info_out.append(info) - finally: - conn.close() +def annotate_vcf(vcf_path: str, segment: str, seg_map: dict, gene_maps: dict, lookup, + want_effects: bool, alias_index: dict, subunit_parent_genes: set, + version: str) -> None: + """Write _flumut_annotated.vcf for one VCF, logging what it matched and skipped.""" + df, header_lines = read_vcf(vcf_path) + stem = os.path.basename(vcf_path).removesuffix(".vcf").removesuffix("_final") + output = f"{stem}_flumut_annotated.vcf" + + # Annotate in place: rows, order, POS, REF/ALT and ANN are all preserved; only INFO + # grows, and only on rows that matched. + info_out: list[str] = [] + annotated = 0 + markers: set = set() + unmapped_genes: set = set() + unparsed_hgvs_p: list[str] = [] + mnv_unresolved = 0 + for info in df["INFO"].astype(str): + if has_info_flag(info, "MNV_UNRESOLVED"): + # fix_mnv_codons.py left this row's HGVS.p at a value already determined to + # be wrong; matching a curated marker against it would be a false positive. + mnv_unresolved += 1 + info_out.append(info) + continue + field = row_flumut_field(info, gene_maps, lookup, want_effects, + alias_index=alias_index, + subunit_parent_genes=subunit_parent_genes, + unmapped_genes=unmapped_genes, + unparsed_hgvs_p=unparsed_hgvs_p) + if field: + annotated += 1 + for entry in field[len("FLUMUT="):].split(","): + markers.update(entry.split("|")[3].split("&")) + info_out.append(f"{info};{field}") + else: + info_out.append(info) if mnv_unresolved: log.warning("%d row(s) carry the MNV_UNRESOLVED flag; their HGVS.p is known to be " @@ -457,15 +439,44 @@ def main() -> None: out_df = df.copy() out_df["INFO"] = info_out out_header = header_lines[:-1] + [FLUMUT_HEADER, header_lines[-1]] - write_vcf(out_df, out_header, args.output) + write_vcf(out_df, out_header, output) log.info( "Wrote %s: %d row(s) scanned, %d annotated, %d distinct FluMutDB marker(s) matched " "across gene(s) %s for %s/%s (v%s, FluMutDB %s)", - args.output, len(df), annotated, len(markers), - ", ".join(sorted(gene_maps)) or "none", args.sample, segment, SCRIPT_VERSION, version, + output, len(df), annotated, len(markers), + ", ".join(sorted(gene_maps)) or "none", stem.split("__")[0], segment, SCRIPT_VERSION, version, ) +def main() -> None: + args = get_args() + segment = args.segment.upper() + + seg_map = load_segment_map(args.coord_map, segment) + genes_map = seg_map.get("genes", {}) + alias_index = build_alias_index(genes_map) + subunit_parent_genes = set(seg_map.get("subunit_parent_genes", [])) + if seg_map.get("stale_annotation"): + log.warning("Coordinate map flags segment %s as stale (the Nextclade dataset does not " + "expose the HA1/HA2 split FluMutDB requires); HA markers cannot be annotated", + segment) + + db_path = locate_db(args.flumut_db) + # The packaged DB never changes; immutable skips the file locking that fails on network mounts. + conn = sqlite3.connect(f"file:{db_path}?immutable=1", uri=True) + try: + version = db_version(conn) + db_ref = SEGMENT_TO_DB_REFERENCE.get(segment, segment) + gene_maps = build_gene_protein_maps(conn, db_ref, genes_map) + # Queries conn lazily and caches, so one lookup serves every VCF in the batch. + lookup = make_marker_lookup(conn, db_ref) + for vcf_path in args.vcf: + annotate_vcf(vcf_path, segment, seg_map, gene_maps, lookup, not args.no_effects, + alias_index, subunit_parent_genes, version) + finally: + conn.close() + + if __name__ == "__main__": main() diff --git a/docs/vcf_annotation_and_flumut_insertion.md b/docs/vcf_annotation_and_flumut_insertion.md index 7f963af..9b470ba 100644 --- a/docs/vcf_annotation_and_flumut_insertion.md +++ b/docs/vcf_annotation_and_flumut_insertion.md @@ -237,11 +237,14 @@ deliberately does not constrain. `Mutation_Name` and both effect fields union ov ### CLI ``` -merge_flumut_mutations.py --vcf --coord-map - --segment --sample -o +merge_flumut_mutations.py --vcf [ …] + --coord-map --segment [--flumut-db ] [--no-effects] ``` +Each `--vcf` is written to `_flumut_annotated.vcf` in the working directory, where +`` is its filename less `.vcf` and a trailing `_final`. + ### Inputs and where they come from | Input | Producer | Used for | @@ -250,23 +253,25 @@ merge_flumut_mutations.py --vcf --coord-map .json`) | `coordinate_conversion` | flumut→Nextclade **AA position** mapping, per gene | | FluMutDB sqlite | `flumutdb` conda package | the marker catalogue; located via `flumut.DbReader.get_db_file()`, overridable with `--flumut-db` | -Wired in `workflows/nextclade.nf` per `(sample, segment)`: the per-segment VCF batches are -flattened and combined with the matching coordinate-map shard. The per-sample `flumut` TSV is +Wired in `workflows/nextclade.nf` once per segment: each segment's VCFs from +`nextclade_run_batch` are combined with the matching coordinate-map shard. The per-sample `flumut` TSV is **not** an input — the marker set comes from the database, so the `flumut` process and this one are independent. ### How it runs (`main`) -1. **Read the VCF** with `tools.read_vcf` → DataFrame + header lines. -2. **Load this segment's block** from the coordinate map (`load_segment_map`), warning if it +Steps 1 and 2 run once per call; `annotate_vcf` repeats steps 3 to 6 for each `--vcf`. + +1. **Load this segment's block** from the coordinate map (`load_segment_map`), warning if it carries `stale_annotation`. -3. **Pair genes with FluMutDB proteins** (`build_gene_protein_maps`): query the distinct +2. **Pair genes with FluMutDB proteins** (`build_gene_protein_maps`): query the distinct `protein_name`s the database maps onto this segment's reference, resolve each to a coordinate-map gene key (`resolve_gene`, see below), and **invert** that gene's `aa_source_to_target` once into `{target_pos: source_pos}`. Inverting is what lets the main loop start from a VCF row and query the database directly. Alignment-derived maps pair columns 1:1, so the inverse is well defined; a collision would be warned about and resolved to the lowest source position. +3. **Read the VCF** with `tools.read_vcf` → DataFrame + header lines. 4. **Skip rows flagged `MNV_UNRESOLVED`** (`has_info_flag`). `fix_mnv_codons.py` sets that flag on PAFTOOLS and SHORT_READ VCFs when it identified a multi-SNV codon but could not compute the combined change (the Nextclade converter always has the reference codon and diff --git a/modules/nextclade/merge_flumut_mutations.nf b/modules/nextclade/merge_flumut_mutations.nf index 155af37..13b3df0 100644 --- a/modules/nextclade/merge_flumut_mutations.nf +++ b/modules/nextclade/merge_flumut_mutations.nf @@ -1,13 +1,13 @@ -// nextclade/merge_flumut_mutations.nf - Annotate the Nextclade VCF with FluMutDB markers. +// nextclade/merge_flumut_mutations.nf - Annotate the Nextclade VCFs with FluMutDB markers. // -// Consumes a per-segment coordinate-map shard and a per-sample/per-segment VCF, and adds a -// FLUMUT= INFO field to rows whose ANN= AA change matches a curated FluMutDB marker. No -// rows are added, removed or reordered. Writes a NEW *_flumut_annotated.vcf, leaving the input -// *_final.vcf untouched. Markers come from the flumutdb package, not a per-sample run. +// Consumes a per-segment coordinate-map shard and every sample's VCF for that segment, and +// adds a FLUMUT= INFO field to rows whose ANN= AA change matches a curated FluMutDB marker. No +// rows are added, removed or reordered. Writes a NEW *_flumut_annotated.vcf per input, leaving +// each *_final.vcf untouched. Markers come from the flumutdb package, not a per-sample run. process merge_flumut_mutations { - tag { "${sample_id}_${segment}" } + tag { segment } conda "${projectDir}/environments/fluscan.yml" @@ -18,42 +18,38 @@ process merge_flumut_mutations { saveAs: { filename -> if (filename.endsWith('_flumut_annotated.vcf')) { def wf = task.ext.workflow ?: 'nextclade' + def sample_id = filename.split('__')[0] return "${sample_id}/${wf}/flumut_annotated/${filename}" } // Provenance is not published from here: collect_provenance concatenates it - // with the rest of the sample's records and publishes that single file. + // with the rest of each sample's records and publishes that single file. return null } input: - tuple val(sample_id), val(segment), path(vcf), path(coord_json) + tuple val(segment), path(vcfs, stageAs: 'vcf/*'), path(coord_json) output: - tuple val(sample_id), val(segment), path("*_flumut_annotated.vcf"), emit: vcf - tuple val(sample_id), path("${sample_id}_${segment}_merge_flumut_provenance.yml"), emit: provenance + tuple val(segment), path("*_flumut_annotated.vcf"), emit: vcf + tuple val(segment), path("${segment}_merge_flumut_provenance.yml"), emit: provenance script: - // Preserve the sample__ref__subtype__segment stem from the input VCF when present. - def stem = vcf.name.replaceAll(/(_final)?\.vcf$/, '') """ - printf -- "- process_name: merge_flumut_mutations\\n" > ${sample_id}_${segment}_merge_flumut_provenance.yml - printf -- " tools:\\n" >> ${sample_id}_${segment}_merge_flumut_provenance.yml - printf -- " - tool_name: python\\n tool_version: \$(python3 --version 2>&1 | cut -d' ' -f2)\\n" >> ${sample_id}_${segment}_merge_flumut_provenance.yml - printf -- " - tool_name: merge_flumut_mutations.py\\n tool_version: \$(merge_flumut_mutations.py --version | awk '{print \$NF}')\\n" >> ${sample_id}_${segment}_merge_flumut_provenance.yml - printf -- " - tool_name: flumutdb\\n tool_version: \$(python3 -c 'from flumut.DbReader import get_db_version; print("%s.%s" % get_db_version()[:2])')\\n" >> ${sample_id}_${segment}_merge_flumut_provenance.yml - - merge_flumut_mutations.py \\ - --vcf ${vcf} \\ - --coord-map ${coord_json} \\ - --segment ${segment} \\ - --sample ${sample_id} \\ - -o ${stem}_flumut_annotated.vcf + printf -- "- process_name: merge_flumut_mutations\\n" > ${segment}_merge_flumut_provenance.yml + printf -- " tools:\\n" >> ${segment}_merge_flumut_provenance.yml + printf -- " - tool_name: python\\n tool_version: \$(python3 --version 2>&1 | cut -d' ' -f2)\\n" >> ${segment}_merge_flumut_provenance.yml + printf -- " - tool_name: merge_flumut_mutations.py\\n tool_version: \$(merge_flumut_mutations.py --version | awk '{print \$NF}')\\n" >> ${segment}_merge_flumut_provenance.yml + printf -- " - tool_name: flumutdb\\n tool_version: \$(python3 -c 'from flumut.DbReader import get_db_version; print("%s.%s" % get_db_version()[:2])')\\n" >> ${segment}_merge_flumut_provenance.yml + + merge_flumut_mutations.py --vcf vcf/*.vcf --coord-map ${coord_json} --segment ${segment} """ stub: - def stem = vcf.name.replaceAll(/(_final)?\.vcf$/, '') """ - touch ${stem}_flumut_annotated.vcf - touch ${sample_id}_${segment}_merge_flumut_provenance.yml + for vcf in vcf/*.vcf; do + stem=\$(basename "\$vcf" .vcf) + touch "\${stem%_final}_flumut_annotated.vcf" + done + touch ${segment}_merge_flumut_provenance.yml """ } diff --git a/modules/shared/database.nf b/modules/shared/database.nf index e49cc53..0318e03 100644 --- a/modules/shared/database.nf +++ b/modules/shared/database.nf @@ -32,13 +32,6 @@ import org.yaml.snakeyaml.Yaml List DB_SEGMENTS = ['PB2', 'PB1', 'PA', 'HA', 'NP', 'NA', 'M', 'NS'] -// @Field variables are not importable across an include, so workflows reach the canonical -// set through this. -def db_segments() { - return DB_SEGMENTS -} - - def db_layout_help() { return """ Expected bundle layout: diff --git a/modules/shared/mutation_watch.nf b/modules/shared/mutation_watch.nf index 26357ad..cbe6464 100644 --- a/modules/shared/mutation_watch.nf +++ b/modules/shared/mutation_watch.nf @@ -1,18 +1,29 @@ -// shared/mutation_watch.nf - Scan a final annotated VCF for a custom watchlist's substitutions. +// shared/mutation_watch.nf - Scan a sample's final annotated VCFs for custom watchlist substitutions. // -// Runs in all three workflows off whatever each one calls its final VCF; the only thing it -// needs is a VCF carrying ANN=. +// Runs in all three workflows off whatever each one calls its final VCFs; the only thing it +// needs is a VCF carrying ANN=. One task per sample scans every VCF against every watchlist. // -// Publishes one file, *_watchlist_hits.tsv, with one row per matching variant. A scan that -// finds nothing writes a header-only file. +// Publishes one *_watchlist_hits.tsv per (VCF, watchlist), with one row per matching variant. +// A scan that finds nothing writes a header-only file. // // Nothing here writes to the VCF. FLUMUT= is curated and always-on and belongs in the VCF; // a watchlist is set on our own terms, and merging it in would make the same consensus // produce different VCFs depending on the contents of a CSV. +// One scan per (VCF, watchlist): [vcf, watchlist, segment, output stem]. segments[i] labels +// vcfs[i]; the watchlist name is in the stem so several lists never collide on publish. +def watch_scans(sample_id, segments, vcfs, watchlists) { + [segments, [vcfs].flatten()].transpose().collectMany { segment, vcf -> + [watchlists].flatten().collect { watchlist -> + def list_name = watchlist.baseName.replaceAll(/_watchlist$/, '') + [vcf, watchlist, segment, [sample_id, segment, list_name].findAll().join('_')] + } + } +} + process mutation_watch { - tag { segment ? "${sample_id}_${segment}_${watchlist.baseName}" : "${sample_id}_${watchlist.baseName}" } + tag { sample_id } conda "${projectDir}/environments/fluscan.yml" @@ -28,41 +39,33 @@ process mutation_watch { } input: - tuple val(sample_id), val(segment), path(vcf), path(watchlist) + tuple val(sample_id), val(segments), path(vcfs, stageAs: 'vcf/*') + path(watchlists, stageAs: 'watchlists/*') output: - tuple val(sample_id), val(segment), path("*_watchlist_hits.tsv"), emit: hits - // Globbed rather than named: the stem is computed in the script block from the - // watchlist's own filename, and an output declaration must not depend on that. - tuple val(sample_id), path("*_mutation_watch_provenance.yml"), emit: provenance + tuple val(sample_id), path("*_watchlist_hits.tsv"), emit: hits + tuple val(sample_id), path("${sample_id}_mutation_watch_provenance.yml"), emit: provenance script: - // The watchlist name is part of the output stem so several lists can be scanned for in - // one run without colliding in the publish directory. - def list_name = watchlist.baseName.replaceAll(/_watchlist$/, '') - def prefix = segment ? "${segment}_${list_name}" : list_name - def stem = "${sample_id}_${prefix}" + def scans = watch_scans(sample_id, segments, vcfs, watchlists).collect { vcf, watchlist, segment, stem -> + "mutation_watcher.py --vcf ${vcf} --watchlist ${watchlist} --sample ${sample_id} --segment '${segment}' -o ${stem}_watchlist_hits.tsv" + } """ - printf -- "- process_name: mutation_watch\\n" > ${stem}_mutation_watch_provenance.yml - printf -- " tools:\\n" >> ${stem}_mutation_watch_provenance.yml - printf -- " - tool_name: python\\n tool_version: \$(python3 --version 2>&1 | cut -d' ' -f2)\\n" >> ${stem}_mutation_watch_provenance.yml - printf -- " - tool_name: mutation_watcher.py\\n tool_version: \$(mutation_watcher.py --version | awk '{print \$NF}')\\n" >> ${stem}_mutation_watch_provenance.yml - printf -- " watchlist: ${watchlist.name}\\n" >> ${stem}_mutation_watch_provenance.yml + for watchlist in watchlists/*; do + printf -- "- process_name: mutation_watch\\n" >> ${sample_id}_mutation_watch_provenance.yml + printf -- " tools:\\n" >> ${sample_id}_mutation_watch_provenance.yml + printf -- " - tool_name: python\\n tool_version: \$(python3 --version 2>&1 | cut -d' ' -f2)\\n" >> ${sample_id}_mutation_watch_provenance.yml + printf -- " - tool_name: mutation_watcher.py\\n tool_version: \$(mutation_watcher.py --version | awk '{print \$NF}')\\n" >> ${sample_id}_mutation_watch_provenance.yml + printf -- " watchlist: \$(basename "\$watchlist")\\n" >> ${sample_id}_mutation_watch_provenance.yml + done - mutation_watcher.py \\ - --vcf ${vcf} \\ - --watchlist ${watchlist} \\ - --sample ${sample_id} \\ - --segment '${segment}' \\ - -o ${stem}_watchlist_hits.tsv + ${scans.join('\n ')} """ stub: - def list_name = watchlist.baseName.replaceAll(/_watchlist$/, '') - def prefix = segment ? "${segment}_${list_name}" : list_name - def stem = "${sample_id}_${prefix}" + def hits = watch_scans(sample_id, segments, vcfs, watchlists).collect { it[3] + '_watchlist_hits.tsv' } """ - touch ${stem}_watchlist_hits.tsv - touch ${stem}_mutation_watch_provenance.yml + touch ${hits.join(' ')} + touch ${sample_id}_mutation_watch_provenance.yml """ } diff --git a/tests/test_merge_flumut_mutations.py b/tests/test_merge_flumut_mutations.py index a2b5e29..7b5a14c 100644 --- a/tests/test_merge_flumut_mutations.py +++ b/tests/test_merge_flumut_mutations.py @@ -147,12 +147,11 @@ def identity_map(n: int = 800) -> dict: def run_script(tmp_path: Path, vcf: Path, coord_map: Path, segment: str, extra: list[str] | None = None) -> tuple[subprocess.CompletedProcess, Path]: - out = tmp_path / "out.vcf" + out = tmp_path / f"{vcf.stem}_flumut_annotated.vcf" proc = subprocess.run( [sys.executable, str(SCRIPT_PATH), "--vcf", str(vcf), "--coord-map", str(coord_map), - "--segment", segment, "--sample", "SAMPLE_1", "-o", str(out), - "--flumut-db", FLUMUT_DB] + (extra or []), - capture_output=True, text=True, + "--segment", segment, "--flumut-db", FLUMUT_DB] + (extra or []), + capture_output=True, text=True, cwd=tmp_path, ) return proc, out @@ -187,6 +186,23 @@ def test_matching_marker_is_annotated_with_name_and_effects(tmp_path): assert " " not in field and "," not in field # legal in a VCF INFO value +def test_each_vcf_in_a_batch_gets_its_own_output(tmp_path): + s1 = write_vcf(tmp_path / "S1__REF__H5N1__PB2_nc_final.vcf", [vcf_row(1881, [ann("PB2", "p.E627K")])]) + s2 = write_vcf(tmp_path / "S2__REF__H5N1__PB2_nc_final.vcf", [vcf_row(2159, [ann("PB2", "p.S715N")])]) + cmap = write_coord_map(tmp_path / "map.json", "PB2", {"PB2": identity_map()}) + + proc = subprocess.run( + [sys.executable, str(SCRIPT_PATH), "--vcf", str(s1), str(s2), "--coord-map", str(cmap), + "--segment", "PB2", "--flumut-db", FLUMUT_DB], + capture_output=True, text=True, cwd=tmp_path, + ) + assert proc.returncode == 0, proc.stderr + + for sample, name in [("S1", "PB2:E627K"), ("S2", "PB2:S715N")]: + (field,) = flumut_fields(tmp_path / f"{sample}__REF__H5N1__PB2_nc_flumut_annotated.vcf") + assert field.split("|")[3] == name + + def test_effects_are_split_into_direct_and_associated(tmp_path): """PB2:E627K is carried by both single- and multi-mutation markers. diff --git a/workflows/nextclade.nf b/workflows/nextclade.nf index 0b99ab4..d7f60b9 100644 --- a/workflows/nextclade.nf +++ b/workflows/nextclade.nf @@ -25,15 +25,11 @@ include { mutation_watch } from '../modules/shared/mutatio include { pipeline_provenance } from '../modules/shared/provenance.nf' include { collect_provenance } from '../modules/shared/provenance.nf' include { resolve_db } from '../modules/shared/database.nf' -include { db_segments } from '../modules/shared/database.nf' include { consensus_fasta_channel } from '../modules/shared/sample_id.nf' include { samplesheet_channel } from '../modules/shared/sample_id.nf' workflow NEXTCLADE { - // The canonical segment set, and the only labels coordinate_conversion shards the map by. - def SEGMENTS = db_segments() - // Only the Nextclade datasets are read here; this branch never BLASTs or runs SnpEff. def db = resolve_db(params.db, ['nextclade']) @@ -101,16 +97,7 @@ workflow NEXTCLADE { // merge_flumut_mutations resolve HA markers through the HA1/HA2 Feature_ID. nextclade_run_batch(ch_segment_batches) - // [segment, [vcfs]] -> [sample_id, segment, vcf]. Shared by the watchlist scan below - // and, when enabled, by the flumut merge further down. - ch_final_vcf = nextclade_run_batch.out.vcf - .transpose() - .map { segment, vcf -> - def sample_id = vcf.name.split('__')[0] - [sample_id, segment, vcf] - } - - // --- Step 5: Scan the final VCF for a custom watchlist (optional) --- + // --- Step 5: Scan the final VCFs for custom watchlists (optional) --- // Independent of --merge_flumut_mutations: FLUMUT= is a curated annotation written // into the VCF, this is a custom list scanned out of it into a separate TSV. if (params.watchlist_dir != 'NO_FILE') { @@ -119,7 +106,12 @@ workflow NEXTCLADE { ch_watchlists = Channel.fromPath( params.watchlist_dir + "/*watchlist.csv", checkIfExists: true ) - mutation_watch(ch_final_vcf.combine(ch_watchlists)) + // [segment, [vcfs]] -> [sample_id, [segments], [vcfs]]: one scan task per sample. + ch_sample_vcfs = nextclade_run_batch.out.vcf + .transpose() + .map { segment, vcf -> [vcf.name.split('__')[0], segment, vcf] } + .groupTuple() + mutation_watch(ch_sample_vcfs, ch_watchlists.collect()) } // --- Steps 6-7: Coordinate conversion + flumut mutation merging (optional) --- @@ -133,24 +125,9 @@ workflow NEXTCLADE { // published map names the reference rather than where the bundle sits. coordinate_conversion(Channel.value([db.nextclade_ref_name, db.nextclade])) - // --- Step 7: Annotate each per-sample/per-segment VCF with FluMutDB markers --- + // --- Step 7: Annotate every sample's VCFs with FluMutDB markers, one task per segment --- // Additive: writes a separate *_flumut_annotated.vcf and leaves *_final.vcf - // untouched. Run per (sample, segment) so each task stages only its own shard. - ch_vcf_flat = ch_final_vcf // [sample_id, segment, vcf] - - // The combine below matches on the segment label and ch_shards only carries the - // eight canonical ones. Split first so a VCF labelled anything else is reported - // rather than vanishing with no merged counterpart and no explanation. - ch_vcf_by_segment = ch_vcf_flat.branch { sample_id, segment, vcf -> - known: SEGMENTS.contains(segment) - unknown: true - } - - ch_vcf_by_segment.unknown.subscribe { sample_id, segment, vcf -> - log.warn "merge_flumut_mutations: ${sample_id} has no coordinate map for segment " + - "'${segment}' (not one of ${SEGMENTS.join(', ')}); ${vcf.name} is left " + - "unannotated and produces no *_flumut_annotated.vcf" - } + // untouched. Each task stages only its own segment's shard. // Key each per-segment shard by its segment label parsed from the filename. ch_shards = coordinate_conversion.out.json_shards @@ -164,13 +141,8 @@ workflow NEXTCLADE { [matcher[0][1], f] } - ch_merge_in = ch_vcf_by_segment.known - .map { sample_id, segment, vcf -> [segment, sample_id, vcf] } - .combine(ch_shards, by: 0) // + coord shard (by segment) - .map { segment, sample_id, vcf, shard -> [sample_id, segment, vcf, shard] } - // -> [sample_id, segment, vcf, coord_shard] - - merge_flumut_mutations(ch_merge_in) + // [segment, [vcfs]] + shard -> [segment, [vcfs], coord_shard] + merge_flumut_mutations(nextclade_run_batch.out.vcf.combine(ch_shards, by: 0)) } // Per-sample provenance. The Nextclade steps are keyed by segment rather than by @@ -182,19 +154,23 @@ workflow NEXTCLADE { ch_provenance = ch_provenance.join(flumut.out.provenance).map{ it -> [it[0], it[1] << it[2]] } if (params.watchlist_dir != 'NO_FILE') { - // Group one task per (sample, segment, watchlist) back to one entry per sample. - // remainder: true keeps samples whose scans all failed. + // One scan task per sample. remainder: true keeps samples whose scan failed. ch_provenance = ch_provenance - .join(mutation_watch.out.provenance.groupTuple(), remainder: true) - .map{ it -> [it[0], (it[1] ?: []) + (it[2] ?: [])] } + .join(mutation_watch.out.provenance, remainder: true) + .map{ sample_id, records, watch -> [sample_id, (records ?: []) + (watch ? [watch] : [])] } } if (params.merge_flumut_mutations) { - // Group one merge task per (sample, segment) back to one entry per sample. - // remainder: true keeps samples whose merges all failed. + // One merge task per segment, so every segment's record is combined onto every + // sample. ifEmpty keeps the chain alive when no segment produced a VCF. + ch_merge_provenance = merge_flumut_mutations.out.provenance + .map { segment, record -> record } + .collect() + .ifEmpty([]) + .map { records -> [records] } ch_provenance = ch_provenance - .join(merge_flumut_mutations.out.provenance.groupTuple(), remainder: true) - .map{ it -> [it[0], (it[1] ?: []) + (it[2] ?: [])] } + .combine(ch_merge_provenance) + .map{ sample_id, records, merge -> [sample_id, records + merge] } // coordinate_conversion runs once per pipeline invocation, not per sample, so // its single provenance file is combined onto every sample rather than joined. diff --git a/workflows/paftools.nf b/workflows/paftools.nf index 4e95df3..fad6b2d 100644 --- a/workflows/paftools.nf +++ b/workflows/paftools.nf @@ -148,24 +148,24 @@ workflow PAFTOOLS { ) - // [sample_id, [final VCFs]] -> [sample_id, segment, vcf], the segment read back off - // the {sample_id}__{ref}__{subtype}__{segment}_paf_final.vcf name post_process_vcf + // [sample_id, [final VCFs]] -> [sample_id, [segments], [vcfs]], each segment read back + // off the {sample_id}__{ref}__{subtype}__{segment}_paf_final.vcf name post_process_vcf // writes. baseName, not simpleName: reference accessions carry dots. - ch_final_vcf = post_process_vcf.out.vcf - .transpose() - .map { sample_id, vcf -> - def segment = vcf.baseName.split('__')[-1].replaceFirst(/_paf_final$/, '') - [sample_id, segment, vcf] + ch_final_vcfs = post_process_vcf.out.vcf + .map { sample_id, vcfs -> + vcfs = [vcfs].flatten() + def segments = vcfs.collect { it.baseName.split('__')[-1].replaceFirst(/_paf_final$/, '') } + [sample_id, segments, vcfs] } - // --- Step 9: Scan the final VCF for a custom watchlist (optional) --- + // --- Step 9: Scan the final VCFs for custom watchlists (optional) --- if (params.watchlist_dir != 'NO_FILE') { // checkIfExists so a mistyped --watchlist_dir fails loudly rather than // silently never scanning. ch_watchlists = Channel.fromPath( params.watchlist_dir + "/*watchlist.csv", checkIfExists: true ) - mutation_watch(ch_final_vcf.combine(ch_watchlists)) + mutation_watch(ch_final_vcfs, ch_watchlists.collect()) } // Per-sample provenance. Inner joins: prepare_consensus, blastn and paftools_align @@ -184,11 +184,10 @@ workflow PAFTOOLS { ch_provenance = ch_provenance.join(post_process_vcf.out.provenance).map{ it -> [it[0], it[1] << it[2]] } if (params.watchlist_dir != 'NO_FILE') { - // Group one task per (sample, segment, watchlist) back to one entry per sample. - // remainder: true keeps samples whose scans all failed. + // One scan task per sample. remainder: true keeps samples whose scan failed. ch_provenance = ch_provenance - .join(mutation_watch.out.provenance.groupTuple(), remainder: true) - .map{ it -> [it[0], (it[1] ?: []) + (it[2] ?: [])] } + .join(mutation_watch.out.provenance, remainder: true) + .map{ sample_id, records, watch -> [sample_id, (records ?: []) + (watch ? [watch] : [])] } } collect_provenance(ch_provenance) diff --git a/workflows/short-read.nf b/workflows/short-read.nf index 8ac0c71..c5e6453 100644 --- a/workflows/short-read.nf +++ b/workflows/short-read.nf @@ -134,14 +134,15 @@ workflow SHORT_READ { ch_watchlists = Channel.fromPath( params.watchlist_dir + "/*watchlist.csv", checkIfExists: true ) - // [sample_id, vcf] -> [sample_id, segment, vcf]; the segment is carried as - // '' because short-read post_process_vcf emits one VCF per sample, not per + // [sample_id, vcfs] -> [sample_id, [segments], [vcfs]]; each segment is carried + // as '' because short-read post_process_vcf emits one VCF per sample, not per // segment, and an invented label would be reported in the output. ch_watch_in = post_process_vcf.out.vcf - .transpose() - .map { sample_id, vcf -> [sample_id, '', vcf] } - .combine(ch_watchlists) - mutation_watch(ch_watch_in) + .map { sample_id, vcfs -> + vcfs = [vcfs].flatten() + [sample_id, vcfs.collect { '' }, vcfs] + } + mutation_watch(ch_watch_in, ch_watchlists.collect()) } } From 99b741fb2d87ac50c46a01eaf2530e3bef58e45f Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Tue, 29 Sep 2026 17:38:17 -0400 Subject: [PATCH 06/11] Prepare consensus FASTAs once for both FASTA branches FASTA_PREP runs hash_files, prepare_consensus and flumut once and feeds PAFTOOLS and NEXTCLADE. The two branch-named consensus QC aggregates become aggregate/consensus_qc_summary.tsv. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 22 ++++----- README.md | 26 +++++----- main.nf | 28 +++++------ nextflow.config | 5 +- workflows/fasta-prep.nf | 64 ++++++++++++++++++++++++ workflows/nextclade.nf | 106 +++++++++++++--------------------------- workflows/paftools.nf | 103 ++++++++++++-------------------------- 7 files changed, 170 insertions(+), 184 deletions(-) create mode 100644 workflows/fasta-prep.nf diff --git a/AGENTS.md b/AGENTS.md index 46cb345..874b1ab 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,8 +25,9 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and - `SHORT_READ` performs read QC/primer trimming, downsampling, assembly, BLAST-based reference selection, mapping, LoFreq calling, SnpEff annotation, MNV correction, consensus generation, and FluMut screening. -- In FASTA mode, both branches independently prepare the same consensus input - and run FluMut. +- In FASTA mode, `FASTA_PREP` (`workflows/fasta-prep.nf`) hashes, cleans and + QC-aggregates the consensus input and runs FluMut once, then hands the cleaned + FASTAs and their provenance records to both branches. - `PAFTOOLS` selects references with BLAST, aligns each segment with minimap2, calls differences with `paftools.js`, sorts and annotates VCFs, converts them to the common VCF shape, and corrects MNV codons. @@ -51,7 +52,7 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and | --- | --- | | `main.nf` | Entry point and workflow dispatch | | `nextflow.config` | Parameters, Conda profile, resources, reports | -| `workflows/` | Channel wiring for the three top-level workflows | +| `workflows/` | Channel wiring for the three top-level workflows and the `FASTA_PREP` subworkflow both FASTA branches take their input from | | `modules/shared/` | Reused BLAST, consensus, FluMut, post-processing, and provenance processes | | `modules/{short_read,paftools,nextclade}/` | Branch-specific processes | | `bin/` | Python CLI entry points *only* — Nextflow puts this directory on `PATH` for every task, so nothing that is merely importable belongs here | @@ -61,7 +62,7 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and | `docs/vcf_annotation_and_flumut_insertion.md` | Detailed Nextclade VCF and FluMutDB semantics | | `docs/watchlists.md` | Custom watchlist schema, matching rules, and the HA1/HA numbering trap | | `assets/watchlists/` | The shipped `*watchlist.csv` files; `--watchlist_dir` defaults here | -| `modules/shared/sample_id.nf` | The only place a consensus or sample sheet `sample_id` is derived; holds the FASTA discovery channel both FASTA branches call and `samplesheet_channel`, which all three workflows call for `--samplesheet_input`. Changing `clean_sample_id` renames every output | +| `modules/shared/sample_id.nf` | The only place a consensus or sample sheet `sample_id` is derived; holds the FASTA discovery channel `FASTA_PREP` calls and `samplesheet_channel`, which all three workflows call for `--samplesheet_input`. Changing `clean_sample_id` renames every output | | `modules/shared/database.nf` | The only place that knows the `--db` bundle layout; `resolve_db` validates it up front. Also holds the canonical segment list, `@Field DB_SEGMENTS`, which it uses to check each segment's dataset | | `scripts/validate_db.py` | Deeper, contents-level bundle check; run after building or moving a bundle | | `docs/database_bundles.md` | Bundle layout, `db.yml` schema, how to build and validate one | @@ -127,8 +128,7 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and explicitly unless a local config supplies it. - `--samplesheet_input` is parsed eagerly by `samplesheet_channel`, so column, blank-cell, ID and file-existence errors all fail before any task is submitted. - Both FASTA branches choose between it and `consensus_fasta_channel` with the same - ternary; keep them mirrored. `run_name` is the sheet's filename without `.csv`. + `FASTA_PREP` chooses between it and `consensus_fasta_channel`. `run_name` is the sheet's filename without `.csv`. - Channel tuple keys are part of the design. Preserve `sample_id` and `segment` through `join`, `combine`, `transpose`, and batching operations, and comment non-obvious tuple shapes next to the transformation. @@ -282,16 +282,16 @@ published filenames/content. Use `-profile conda`; `conda.useMamba` is enabled. FASTA dispatch runs PAFTOOLS and NEXTCLADE over the same samples into the same `${outdir}/${sample_id}/`, and their timestamps collide at second resolution; without the label one branch silently overwrites the other. Any new workflow - needs a `withName: ':.*' { ext.workflow = … }` block or its records fall + that collects provenance needs a `withName: ':.*' { ext.workflow = … }` block or its records fall back to the unlabelled, collision-prone name. - `mutation_watch` runs once per sample over all of its final VCFs and every watchlist, and emits one provenance YAML per sample. SHORT_READ does not join it in, so its watchlist scan is absent from the collected record. Both FASTA branches join it with `remainder: true`. -- The two FASTA branches are deliberate mirrors from input discovery through - `prepare_consensus`, `flumut`, the consensus QC aggregate, and the shape of - the provenance chain and watchlist block. Only the analysis between those - differs. A change to one of the mirrored parts belongs in both. +- The two FASTA branches share their input through `FASTA_PREP`, and each + opens its provenance chain with its own `pipeline_provenance` record ahead of + the prep records. Their watchlist blocks and provenance joins are still + mirrored; a change to one belongs in both. - `tests/conftest.py` puts `bin/` on `sys.path` so the entry points can resolve their `from lib.…` imports under pytest. Fix an import failure there, not by adding path manipulation to a production script. diff --git a/README.md b/README.md index 3e11501..8df36f8 100644 --- a/README.md +++ b/README.md @@ -47,13 +47,18 @@ flowchart TD dispatch[main.nf FASTA dispatch] input --> dispatch - dispatch --> paftools_prepare - dispatch --> nextclade_prepare + dispatch --> prepare + + subgraph FASTA_PREP[FASTA_PREP workflow] + direction TB + prepare[prepare_consensus] + prep_flumut[flumut] + + prepare --> prep_flumut + end subgraph PAFTOOLS[PAFTOOLS workflow] direction TB - paftools_prepare[prepare_consensus] - paftools_flumut[flumut] blastn[blastn reference selection] split_ref[split_fasta] minimap[minimap2 asm20] @@ -66,9 +71,8 @@ flowchart TD fixmnv_paf[fix_mnv_codons.py] paf_final[Final PAFTOOLS VCFs] - paftools_prepare --> blastn - paftools_prepare --> paftools_flumut - paftools_prepare --> split_ref + prepare --> blastn + prepare --> split_ref blastn --> split_ref split_ref --> minimap minimap --> covqc @@ -84,8 +88,6 @@ flowchart TD subgraph NEXTCLADE[NEXTCLADE workflow] direction TB - nextclade_prepare[prepare_consensus] - nextclade_flumut[flumut] groupseg[collect each segment across samples: splitFasta + collectFile] nc_run[nextclade run batch] split_results[split per-sample JSON and TSV] @@ -96,8 +98,7 @@ flowchart TD flumut_merge[merge_flumut_mutations: merge_flumut_mutations.py] flumut_annotated[FluMutDB-annotated VCFs] - nextclade_prepare --> nextclade_flumut - nextclade_prepare --> groupseg + prepare --> groupseg groupseg --> nc_run nc_run --> split_results nc_run --> nc2vcf @@ -288,8 +289,7 @@ The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder `FORMAT/GQ` as |------|-------------| | `${run_name}_fluscan-nf_multiqc_report.html` | MultiQC report summarising fastp, cutadapt, and FastQC results for all samples (short-read workflow only) | | `aggregate/paftools_qc_summary.tsv` | Run-level tabular summary of PAFTOOLS alignment coverage across all FASTA samples and segments | -| `aggregate/paftools_consensus_qc_summary.tsv` | Every `*_qc_report.tsv` row from consensus preparation, concatenated across the run (PAFTOOLS). A sample rejected at preparation — duplicate segment, or headers naming more than one sample — contributes no rows and is reported only in the Nextflow log | -| `aggregate/nextclade_consensus_qc_summary.tsv` | The same rows collected by the NEXTCLADE branch; both branches prepare the same inputs, so the two files agree | +| `aggregate/consensus_qc_summary.tsv` | Every `*_qc_report.tsv` row from consensus preparation, concatenated across the run. A sample rejected at preparation — duplicate segment, or headers naming more than one sample — contributes no rows and is reported only in the Nextflow log | | `aggregate/reference_remap/coordinate_map.json` | Primary machine-readable nucleotide/amino-acid coordinate map lifting flumut's reference (source) onto the selected Nextclade reference dataset (target) (NEXTCLADE workflow only) | | `aggregate/reference_remap/coordinate_map.nt.tsv` | Flat per-position nucleotide coordinate map (source/flumut position/base -> target/nextclade position/base, per segment) | | `aggregate/reference_remap/coordinate_map.aa.tsv` | Flat per-residue amino-acid coordinate map (source/flumut position/residue -> target/nextclade position/residue, per segment/gene) | diff --git a/main.nf b/main.nf index 2f65159..7728de7 100644 --- a/main.nf +++ b/main.nf @@ -6,6 +6,7 @@ */ include { SHORT_READ } from './workflows/short-read.nf' +include { FASTA_PREP } from './workflows/fasta-prep.nf' include { PAFTOOLS } from './workflows/paftools.nf' include { NEXTCLADE } from './workflows/nextclade.nf' include { resolve_db } from './modules/shared/database.nf' @@ -44,22 +45,19 @@ workflow { error "ERROR: Provide only one of --fastq_input, --fasta_input or --samplesheet_input." } - if (params.samplesheet_input != 'NO_FILE') { - // The header row picks the workflow; samplesheet_channel validates the rest. - def header = file(params.samplesheet_input, checkIfExists: true).splitCsv(strip: true)[0] ?: [] - if ('R1' in header) { - SHORT_READ() - } else if ('FASTA' in header) { - PAFTOOLS() - NEXTCLADE() - } else { - error "ERROR: --samplesheet_input needs an ID,R1,R2 header (short-read workflow) or an ID,FASTA header (PAFTOOLS/NEXTCLADE workflow)." - } - } else if (params.fastq_input != 'NO_FILE') { + // A sample sheet's header row picks the workflow; samplesheet_channel validates the rest. + def header = params.samplesheet_input != 'NO_FILE' + ? (file(params.samplesheet_input, checkIfExists: true).splitCsv(strip: true)[0] ?: []) + : [] + + if (params.fastq_input != 'NO_FILE' || 'R1' in header) { SHORT_READ() - } else if (params.fasta_input != 'NO_FILE') { - PAFTOOLS() - NEXTCLADE() + } else if (params.fasta_input != 'NO_FILE' || 'FASTA' in header) { + FASTA_PREP() + PAFTOOLS(FASTA_PREP.out.cleaned, FASTA_PREP.out.provenance) + NEXTCLADE(FASTA_PREP.out.cleaned, FASTA_PREP.out.provenance) + } else if (params.samplesheet_input != 'NO_FILE') { + error "ERROR: --samplesheet_input needs an ID,R1,R2 header (short-read workflow) or an ID,FASTA header (PAFTOOLS/NEXTCLADE workflow)." } else { error "ERROR: Please provide --fastq_input (short-read workflow), --fasta_input (PAFTOOLS/NEXTCLADE workflow) or --samplesheet_input (either)." } diff --git a/nextflow.config b/nextflow.config index 61cf53f..563589c 100644 --- a/nextflow.config +++ b/nextflow.config @@ -88,13 +88,14 @@ process { ext.workflow = 'short_read' ext.segment_regex = '(?P[^\\|]+)\\|[^\\|]+\\|[^\\|]+\\|(?P[^\\|]+)' } + withName: 'FASTA_PREP:.*' { + ext.segment_regex = '(.+)_(.+)' + } withName: 'PAFTOOLS:.*' { ext.workflow = 'paftools' - ext.segment_regex = '(.+)_(.+)' } withName: 'NEXTCLADE:.*' { ext.workflow = 'nextclade' - ext.segment_regex = '(.+)_(.+)' } withName: short_read_assembly { diff --git a/workflows/fasta-prep.nf b/workflows/fasta-prep.nf new file mode 100644 index 0000000..47b6166 --- /dev/null +++ b/workflows/fasta-prep.nf @@ -0,0 +1,64 @@ +#!/usr/bin/env nextflow + +/* + * Consensus FASTA preparation, shared by PAFTOOLS and NEXTCLADE + * ------------------------------------------------------------- + * Hashes each input FASTA, validates and cleans it, aggregates the consensus QC rows and + * runs FluMut, once per sample however many FASTA branches consume the result. + * + * The sample name comes from the input filename (modules/shared/sample_id.nf); input + * headers supply only the segment label. Accepted header formats: + * Underscore-delimited (legacy): >SAMPLE_ID_segmentN_SEGMENT + * e.g. >WIN-AH-2024-FAV-0415-1-OS_segment1_PB2 + * Pipe-delimited (GISAID-style): >EPI_ISL_ACCESSION|subtype|SEGMENT|...|clade_SEGMENT + * e.g. >EPI_ISL_20399898|A_/_H5N1|NA|6|EPI5261179|2.3.4.4b_NA + * Cleaned headers are normalised to {sample_id}_{SEGMENT} by prepare_consensus. + */ + +include { hash_files } from '../modules/shared/hash_files.nf' +include { prepare_consensus } from '../modules/shared/consensus_qc.nf' +include { flumut } from '../modules/shared/flumut.nf' +include { consensus_fasta_channel } from '../modules/shared/sample_id.nf' +include { samplesheet_channel } from '../modules/shared/sample_id.nf' + +workflow FASTA_PREP { + + main: + + // Consensus FASTAs from the sample sheet or the input directory. + // Each file is expected to contain 8 segments for a single sample. + ch_fasta_input = params.samplesheet_input != 'NO_FILE' + ? samplesheet_channel(params.samplesheet_input, ['FASTA']) + : consensus_fasta_channel(params.fasta_input) + + // Hash the input FASTA. One file per sample, so no list-wrapping here. + hash_files(ch_fasta_input.combine(Channel.of("fasta_input"))) + + // Validate and clean input consensus FASTA. + prepare_consensus(ch_fasta_input) + + // One run-level table of the per-segment consensus QC rows. + prepare_consensus.out.qc + .map{ it -> it[1] } + .collectFile( + name: 'consensus_qc_summary.tsv', + storeDir: "${params.outdir}/aggregate", + keepHeader: true, // keep header from first file only + skip: 1, // drop the header line from the rest + sort: false + ) + + // HPAI mutation screening (per sample). + flumut(prepare_consensus.out.cleaned) + + // Inner joins: prepare_consensus runs errorStrategy 'ignore', so a sample it rejects + // leaves every branch's provenance chain. + ch_provenance = hash_files.out.provenance + .join(prepare_consensus.out.provenance) + .join(flumut.out.provenance) + .map { sample_id, hash, prepare, mutations -> [sample_id, [hash, prepare, mutations]] } + + emit: + cleaned = prepare_consensus.out.cleaned // [sample_id, cleaned consensus FASTA] + provenance = ch_provenance // [sample_id, [hash, prepare, flumut]] +} diff --git a/workflows/nextclade.nf b/workflows/nextclade.nf index d7f60b9..c748909 100644 --- a/workflows/nextclade.nf +++ b/workflows/nextclade.nf @@ -3,21 +3,11 @@ /* * Nextclade consensus FASTA workflow * ------------------------------------ - * Takes pre-assembled consensus FASTAs through QC, per-segment Nextclade alignment - * against the --db bundle's datasets, and conversion of the JSON to a freebayes-style VCF. - * - * The sample name comes from the input filename (modules/shared/sample_id.nf); input - * headers supply only the segment label. Accepted header formats: - * Underscore-delimited (legacy): >SAMPLE_ID_segmentN_SEGMENT - * e.g. >WIN-AH-2024-FAV-0415-1-OS_segment1_PB2 - * Pipe-delimited (GISAID-style): >EPI_ISL_ACCESSION|subtype|SEGMENT|...|clade_SEGMENT - * e.g. >EPI_ISL_20399898|A_/_H5N1|NA|6|EPI5261179|2.3.4.4b_NA - * Cleaned headers are normalised to {sample_id}_{SEGMENT} by prepare_consensus. + * Takes the consensus FASTAs FASTA_PREP has cleaned through per-segment Nextclade + * alignment against the --db bundle's datasets, and conversion of the JSON to a + * freebayes-style VCF. */ -include { hash_files } from '../modules/shared/hash_files.nf' -include { prepare_consensus } from '../modules/shared/consensus_qc.nf' -include { flumut } from '../modules/shared/flumut.nf' include { nextclade_run_batch } from '../modules/nextclade/nextclade.nf' include { coordinate_conversion } from '../modules/nextclade/coordinate_conversion.nf' include { merge_flumut_mutations } from '../modules/nextclade/merge_flumut_mutations.nf' @@ -25,63 +15,37 @@ include { mutation_watch } from '../modules/shared/mutatio include { pipeline_provenance } from '../modules/shared/provenance.nf' include { collect_provenance } from '../modules/shared/provenance.nf' include { resolve_db } from '../modules/shared/database.nf' -include { consensus_fasta_channel } from '../modules/shared/sample_id.nf' -include { samplesheet_channel } from '../modules/shared/sample_id.nf' workflow NEXTCLADE { - // Only the Nextclade datasets are read here; this branch never BLASTs or runs SnpEff. - def db = resolve_db(params.db, ['nextclade']) - - ch_workflow_metadata = Channel.value([ - workflow.sessionId, - workflow.runName, - workflow.manifest.name, - workflow.manifest.version, - workflow.start, - [ - database_name: db.name, - database_version: db.version, - database_path: db.root, - ], - ]) - - ch_pipeline_provenance = pipeline_provenance(ch_workflow_metadata) - - // Consensus FASTAs from the sample sheet or the input directory. - // Each file is expected to contain 8 segments for a single sample. - ch_fasta_input = params.samplesheet_input != 'NO_FILE' - ? samplesheet_channel(params.samplesheet_input, ['FASTA']) - : consensus_fasta_channel(params.fasta_input) + take: + ch_cleaned // [sample_id, cleaned consensus FASTA] + ch_prep_provenance // [sample_id, [hash, prepare, flumut]] main: - // Hash the input FASTA. One file per sample, so no list-wrapping here. - hash_files(ch_fasta_input.combine(Channel.of("fasta_input"))) - - // --- Step 1: Validate and clean input consensus FASTA --- - prepare_consensus(ch_fasta_input) - - // One run-level table of the per-segment consensus QC rows. The name carries the - // workflow because main.nf runs both FASTA branches into the same outdir, and two - // collectFile operations must not target one path. - prepare_consensus.out.qc - .map{ it -> it[1] } - .collectFile( - name: 'nextclade_consensus_qc_summary.tsv', - storeDir: "${params.outdir}/aggregate", - keepHeader: true, // keep header from first file only - skip: 1, // drop the header line from the rest - sort: false - ) - - // --- Step 2: HPAI mutation screening (per sample, parallel branch) --- - flumut(prepare_consensus.out.cleaned) - - // --- Step 3: Collect each segment across all samples into one FASTA --- + // Only the Nextclade datasets are read here; this branch never BLASTs or runs SnpEff. + def db = resolve_db(params.db, ['nextclade']) + + ch_workflow_metadata = Channel.value([ + workflow.sessionId, + workflow.runName, + workflow.manifest.name, + workflow.manifest.version, + workflow.start, + [ + database_name: db.name, + database_version: db.version, + database_path: db.root, + ], + ]) + + ch_pipeline_provenance = pipeline_provenance(ch_workflow_metadata) + + // --- Step 1: Collect each segment across all samples into one FASTA --- // Cleaned headers are {sample_id}_{SEGMENT} with a canonical SEGMENT; the batch // header is the bare sample_id, which nextclade reports back as seqName. - ch_segment_batches = prepare_consensus.out.cleaned + ch_segment_batches = ch_cleaned .splitFasta(elem: 1, record: [id: true, seqString: true]) // [sample_id, record] .collectFile { item -> def (sample_id, record) = item @@ -90,14 +54,14 @@ workflow NEXTCLADE { .map { fasta -> [fasta.baseName, fasta, db.nextclade.resolve(fasta.baseName)] } // -> [segment, batch.fasta, dataset_dir] - // --- Step 4: Run nextclade once per segment for all samples, and convert to VCF --- + // --- Step 2: Run nextclade once per segment for all samples, and convert to VCF --- // Also writes each sample's report-only consensus/reference identity QC. // With --emit_no_mature_vcf it also publishes each VCF without mature-peptide // (HA1/HA2) annotations. Nothing downstream reads that copy: mutation_watch and // merge_flumut_mutations resolve HA markers through the HA1/HA2 Feature_ID. nextclade_run_batch(ch_segment_batches) - // --- Step 5: Scan the final VCFs for custom watchlists (optional) --- + // --- Step 3: Scan the final VCFs for custom watchlists (optional) --- // Independent of --merge_flumut_mutations: FLUMUT= is a curated annotation written // into the VCF, this is a custom list scanned out of it into a separate TSV. if (params.watchlist_dir != 'NO_FILE') { @@ -114,18 +78,18 @@ workflow NEXTCLADE { mutation_watch(ch_sample_vcfs, ch_watchlists.collect()) } - // --- Steps 6-7: Coordinate conversion + flumut mutation merging (optional) --- + // --- Steps 4-5: Coordinate conversion + flumut mutation merging (optional) --- // Gated behind --merge_flumut_mutations (default false); otherwise the workflow // stops at the *_final.vcf produced above. if (params.merge_flumut_mutations) { - // --- Step 6: Build the flumut-to-Nextclade coordinate map --- + // --- Step 4: Build the flumut-to-Nextclade coordinate map --- // source = the flumutdb package's reference (lift FROM), target = the --db // datasets (lift ONTO). The target label comes from the bundle's db.yml, so the // published map names the reference rather than where the bundle sits. coordinate_conversion(Channel.value([db.nextclade_ref_name, db.nextclade])) - // --- Step 7: Annotate every sample's VCFs with FluMutDB markers, one task per segment --- + // --- Step 5: Annotate every sample's VCFs with FluMutDB markers, one task per segment --- // Additive: writes a separate *_flumut_annotated.vcf and leaves *_final.vcf // untouched. Each task stages only its own segment's shard. @@ -147,11 +111,9 @@ workflow NEXTCLADE { // Per-sample provenance. The Nextclade steps are keyed by segment rather than by // sample, so they have no per-sample record to join here. - ch_provenance = ch_fasta_input.map{ it -> it[0] } - ch_provenance = ch_provenance.combine(ch_pipeline_provenance).map{ it -> [it[0], [it[1]]] } - ch_provenance = ch_provenance.join(hash_files.out.provenance).map{ it -> [it[0], it[1] << it[2]] } - ch_provenance = ch_provenance.join(prepare_consensus.out.provenance).map{ it -> [it[0], it[1] << it[2]] } - ch_provenance = ch_provenance.join(flumut.out.provenance).map{ it -> [it[0], it[1] << it[2]] } + ch_provenance = ch_prep_provenance + .combine(ch_pipeline_provenance) + .map{ sample_id, records, pipeline -> [sample_id, [pipeline] + records] } if (params.watchlist_dir != 'NO_FILE') { // One scan task per sample. remainder: true keeps samples whose scan failed. diff --git a/workflows/paftools.nf b/workflows/paftools.nf index fad6b2d..80a7971 100644 --- a/workflows/paftools.nf +++ b/workflows/paftools.nf @@ -3,17 +3,9 @@ /* * PAFTOOLS workflow (consensus FASTA input) * ----------------------------------------- - * Processes pre-assembled consensus FASTA files through quality control, - * subtype-aware reference selection via BLAST, minimap2/paftools variant - * calling, bcftools VCF sorting, SnpEff annotation, and summary extraction. - * - * The sample name comes from the input filename (modules/shared/sample_id.nf); input - * headers supply only the segment label. Accepted header formats: - * Underscore-delimited (legacy): >SAMPLE_ID_segmentN_SEGMENT - * e.g. >WIN-AH-2024-FAV-0415-1-OS_segment1_PB2 - * Pipe-delimited (GISAID-style): >EPI_ISL_ACCESSION|subtype|SEGMENT|...|clade_SEGMENT - * e.g. >EPI_ISL_20399898|A_/_H5N1|NA|6|EPI5261179|2.3.4.4b_NA - * Cleaned headers are normalised to {sample_id}_{SEGMENT} by prepare_consensus. + * Takes the consensus FASTAs FASTA_PREP has cleaned through subtype-aware reference + * selection via BLAST, minimap2/paftools variant calling, bcftools VCF sorting, SnpEff + * annotation, and summary extraction. * * Reference database header format (<--db>/blastn_db.fa): * >accession|subtype|segment @@ -22,17 +14,14 @@ * */ -include { hash_files } from '../modules/shared/hash_files.nf' include { pipeline_provenance } from '../modules/shared/provenance.nf' include { collect_provenance } from '../modules/shared/provenance.nf' -include { prepare_consensus } from '../modules/shared/consensus_qc.nf' include { split_fasta as split_fasta_reference } from '../modules/shared/consensus_qc.nf' include { split_fasta as split_fasta_consensus } from '../modules/shared/consensus_qc.nf' include { paftools_align } from '../modules/paftools/alignment.nf' include { consensus_coverage_qc } from '../modules/paftools/alignment.nf' include { makeblastdb } from '../modules/shared/blast.nf' include { blastn } from '../modules/shared/blast.nf' -include { flumut } from '../modules/shared/flumut.nf' include { paftools_call } from '../modules/paftools/variant_calling.nf' include { paftools_sort_vcf } from '../modules/paftools/variant_calling.nf' include { paftools_snpeff_annotation } from '../modules/paftools/variant_calling.nf' @@ -40,75 +29,49 @@ include { paftools_to_freebayes_vcf } from '../modules/paftoo include { post_process_vcf } from '../modules/shared/post_process.nf' include { mutation_watch } from '../modules/shared/mutation_watch.nf' include { resolve_db } from '../modules/shared/database.nf' -include { consensus_fasta_channel } from '../modules/shared/sample_id.nf' -include { samplesheet_channel } from '../modules/shared/sample_id.nf' workflow PAFTOOLS { - // Reference selection is BLAST-driven and every called variant is SnpEff-annotated, - // so both components are mandatory for this branch. - def db = resolve_db(params.db, ['blastn', 'snpeff']) - - ch_workflow_metadata = Channel.value([ - workflow.sessionId, - workflow.runName, - workflow.manifest.name, - workflow.manifest.version, - workflow.start, - [ - database_name: db.name, - database_version: db.version, - database_path: db.root, - ], - ]) - - ch_pipeline_provenance = pipeline_provenance(ch_workflow_metadata) - - // Consensus FASTAs from the sample sheet or the input directory. - // Each file is expected to contain 8 segments for a single sample. - ch_fasta_input = params.samplesheet_input != 'NO_FILE' - ? samplesheet_channel(params.samplesheet_input, ['FASTA']) - : consensus_fasta_channel(params.fasta_input) - - ch_blastn_db = Channel.fromPath(db.blastn) + take: + ch_cleaned // [sample_id, cleaned consensus FASTA] + ch_prep_provenance // [sample_id, [hash, prepare, flumut]] main: - // The whole snpeff/ directory is staged so the config and the database it names - // travel together; the process overrides data.dir with -dataDir. - ch_snpeff_dir = Channel.fromPath(db.snpeff_dir, checkIfExists: true) + // Reference selection is BLAST-driven and every called variant is SnpEff-annotated, + // so both components are mandatory for this branch. + def db = resolve_db(params.db, ['blastn', 'snpeff']) - // Hash the input FASTA. One file per sample, so no list-wrapping here. - hash_files(ch_fasta_input.combine(Channel.of("fasta_input"))) + ch_workflow_metadata = Channel.value([ + workflow.sessionId, + workflow.runName, + workflow.manifest.name, + workflow.manifest.version, + workflow.start, + [ + database_name: db.name, + database_version: db.version, + database_path: db.root, + ], + ]) - // --- Step 1: Validate and clean input consensus FASTA --- - prepare_consensus(ch_fasta_input) + ch_pipeline_provenance = pipeline_provenance(ch_workflow_metadata) - // One run-level table of the per-segment consensus QC rows. The name carries the - // workflow because main.nf runs both FASTA branches into the same outdir, and two - // collectFile operations must not target one path. - prepare_consensus.out.qc - .map{ it -> it[1] } - .collectFile( - name: 'paftools_consensus_qc_summary.tsv', - storeDir: "${params.outdir}/aggregate", - keepHeader: true, // keep header from first file only - skip: 1, // drop the header line from the rest - sort: false - ) + ch_blastn_db = Channel.fromPath(db.blastn) + + // The whole snpeff/ directory is staged so the config and the database it names + // travel together; the process overrides data.dir with -dataDir. + ch_snpeff_dir = Channel.fromPath(db.snpeff_dir, checkIfExists: true) // --- Step 2: Build BLAST database (once, shared across all samples) --- makeblastdb(ch_blastn_db).first().set{ch_blastdb} // --- Step 3: BLAST cleaned consensus against reference database --- - blastn(prepare_consensus.out.cleaned, ch_blastdb) - - // --- Step 3b: HPAI mutation screening (per sample, parallel branch) --- - flumut(prepare_consensus.out.cleaned) + blastn(ch_cleaned, ch_blastdb) - ch_consensus_segments = split_fasta_consensus(prepare_consensus.out.cleaned) + ch_consensus_segments = split_fasta_consensus(ch_cleaned) ch_reference_segments = split_fasta_reference(blastn.out.ref) ch_segment_sequences = ch_consensus_segments.join(ch_reference_segments) @@ -171,11 +134,9 @@ workflow PAFTOOLS { // Per-sample provenance. Inner joins: prepare_consensus, blastn and paftools_align // run errorStrategy 'ignore', so a sample failing any of them leaves the chain and // publishes no provenance file at all. - ch_provenance = ch_fasta_input.map{ it -> it[0] } - ch_provenance = ch_provenance.combine(ch_pipeline_provenance).map{ it -> [it[0], [it[1]]] } - ch_provenance = ch_provenance.join(hash_files.out.provenance).map{ it -> [it[0], it[1] << it[2]] } - ch_provenance = ch_provenance.join(prepare_consensus.out.provenance).map{ it -> [it[0], it[1] << it[2]] } - ch_provenance = ch_provenance.join(flumut.out.provenance).map{ it -> [it[0], it[1] << it[2]] } + ch_provenance = ch_prep_provenance + .combine(ch_pipeline_provenance) + .map{ sample_id, records, pipeline -> [sample_id, [pipeline] + records] } ch_provenance = ch_provenance.join(blastn.out.provenance).map{ it -> [it[0], it[1] << it[2]] } ch_provenance = ch_provenance.join(paftools_align.out.provenance).map{ it -> [it[0], it[1] << it[2]] } ch_provenance = ch_provenance.join(paftools_call.out.provenance).map{ it -> [it[0], it[1] << it[2]] } From f0003aa268c70d1c9ae031e2eddca31fda03a034 Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Tue, 29 Sep 2026 17:39:38 -0400 Subject: [PATCH 07/11] Record Nextclade in per-sample provenance Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 11 +++++++---- modules/nextclade/nextclade.nf | 10 ++++++++++ workflows/nextclade.nf | 11 ++++++++--- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 874b1ab..9d35d7d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -272,7 +272,8 @@ published filenames/content. Use `-profile conda`; `conda.useMamba` is enabled. complete without those annotated final VCF steps. - Sample-level provenance collection is wired in all three workflows, and each starts with `hash_files` (`modules/shared/hash_files.nf`, aliased to - `short_read_hash_files` in SHORT_READ). The collection chains are inner joins, + `short_read_hash_files` in SHORT_READ, run by `FASTA_PREP` for the FASTA + branches). The collection chains are inner joins, and `prepare_consensus`, `blastn` and `paftools_align` all run `errorStrategy 'ignore'` — so a sample failing any of them publishes no provenance file, losing its input hash along with everything else. Use @@ -282,8 +283,8 @@ published filenames/content. Use `-profile conda`; `conda.useMamba` is enabled. FASTA dispatch runs PAFTOOLS and NEXTCLADE over the same samples into the same `${outdir}/${sample_id}/`, and their timestamps collide at second resolution; without the label one branch silently overwrites the other. Any new workflow - that collects provenance needs a `withName: ':.*' { ext.workflow = … }` block or its records fall - back to the unlabelled, collision-prone name. + that collects provenance needs a `withName: ':.*' { ext.workflow = … }` + block or its records fall back to the unlabelled, collision-prone name. - `mutation_watch` runs once per sample over all of its final VCFs and every watchlist, and emits one provenance YAML per sample. SHORT_READ does not join it in, so its watchlist scan is absent from the collected record. Both FASTA @@ -291,7 +292,9 @@ published filenames/content. Use `-profile conda`; `conda.useMamba` is enabled. - The two FASTA branches share their input through `FASTA_PREP`, and each opens its provenance chain with its own `pipeline_provenance` record ahead of the prep records. Their watchlist blocks and provenance joins are still - mirrored; a change to one belongs in both. + mirrored; a change to one belongs in both. NEXTCLADE also combines each + per-segment record (`nextclade_run_batch`, `merge_flumut_mutations`) onto + every sample. - `tests/conftest.py` puts `bin/` on `sys.path` so the entry points can resolve their `from lib.…` imports under pytest. Fix an import failure there, not by adding path manipulation to a production script. diff --git a/modules/nextclade/nextclade.nf b/modules/nextclade/nextclade.nf index 288cdaa..a5ae091 100644 --- a/modules/nextclade/nextclade.nf +++ b/modules/nextclade/nextclade.nf @@ -51,6 +51,7 @@ process nextclade_run_batch { tuple val(segment), path("*_identity_qc.tsv"), emit: identity_qc tuple val(segment), path("nc_output"), emit: all tuple val(segment), path("${segment}_nextclade_status.tsv"), emit: status + tuple val(segment), path("${segment}_nextclade_provenance.yml"), emit: provenance script: def threads = task.cpus ?: 1 @@ -80,6 +81,15 @@ process nextclade_run_batch { SUBTYPE=\$(head -n1 ${dataset_dir}/reference.fasta | sed -E 's/.+\\(\\S+\\((\\S+)\\)\\S*\\).+/\\1/g') if echo "\$SUBTYPE" | grep -q '^>'; then SUBTYPE="unknown"; fi + printf -- "- process_name: nextclade_run_batch\\n" > ${segment}_nextclade_provenance.yml + printf -- " tools:\\n" >> ${segment}_nextclade_provenance.yml + printf -- " - tool_name: python\\n tool_version: \$(python3 --version 2>&1 | cut -d' ' -f2)\\n" >> ${segment}_nextclade_provenance.yml + printf -- " - tool_name: nextclade\\n tool_version: \$(nextclade --version | awk '{print \$NF}')\\n" >> ${segment}_nextclade_provenance.yml + printf -- " parameters:\\n" >> ${segment}_nextclade_provenance.yml + printf -- " - parameter: segment\\n value: ${segment}\\n" >> ${segment}_nextclade_provenance.yml + printf -- " - parameter: reference\\n value: \$REFERENCE_NAME\\n" >> ${segment}_nextclade_provenance.yml + printf -- " - parameter: alignment_preset\\n value: ${params.nextclade_alignment_preset}\\n" >> ${segment}_nextclade_provenance.yml + nextclade_to_freebayes_vcf.py \\ -i nc_output/batch.json \\ -d ${dataset_dir} \\ diff --git a/workflows/nextclade.nf b/workflows/nextclade.nf index c748909..4a88f07 100644 --- a/workflows/nextclade.nf +++ b/workflows/nextclade.nf @@ -109,11 +109,16 @@ workflow NEXTCLADE { merge_flumut_mutations(nextclade_run_batch.out.vcf.combine(ch_shards, by: 0)) } - // Per-sample provenance. The Nextclade steps are keyed by segment rather than by - // sample, so they have no per-sample record to join here. + // Per-sample provenance. Nextclade runs once per segment, so every segment's record + // is combined onto every sample. + ch_nextclade_provenance = nextclade_run_batch.out.provenance + .map { segment, record -> record } + .collect() + .map { records -> [records] } ch_provenance = ch_prep_provenance .combine(ch_pipeline_provenance) - .map{ sample_id, records, pipeline -> [sample_id, [pipeline] + records] } + .combine(ch_nextclade_provenance) + .map{ sample_id, records, pipeline, nextclade -> [sample_id, [pipeline] + records + nextclade] } if (params.watchlist_dir != 'NO_FILE') { // One scan task per sample. remainder: true keeps samples whose scan failed. From c319f34dafc6e384e5cba75da676d544fbb15f2e Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Thu, 1 Oct 2026 15:01:45 -0400 Subject: [PATCH 08/11] Simplify shared-codon bookkeeping in the Nextclade converter Output is byte-identical to the previous commit on the 40-sample batch. Co-Authored-By: Claude Opus 5.5 --- bin/nextclade_to_freebayes_vcf.py | 123 ++++++++++++++---------------- 1 file changed, 56 insertions(+), 67 deletions(-) diff --git a/bin/nextclade_to_freebayes_vcf.py b/bin/nextclade_to_freebayes_vcf.py index 8cd1bed..6b25715 100755 --- a/bin/nextclade_to_freebayes_vcf.py +++ b/bin/nextclade_to_freebayes_vcf.py @@ -143,11 +143,12 @@ def load_gff3(gff_path: str) -> tuple[list, list]: """Parse GFF3; return (genes, transcripts). genes: [{gene_name, gene_id, begin_0, end_0}] - transcripts: [{transcript_kind, gene_name, gene_id, feature_id, protein_id, + transcripts: [{transcript_kind, gene_name, gene_id, feature_id, protein_id, nextclade_cds, strand, segments, is_mature, warnings, cds_length, aa_length}] segments is a sorted list of (begin_0, end_0) half-open intervals. protein_id matches Nextclade's cdsName key in nucToAaMuts/aaSubstitutions. + nextclade_cds is the cdsName whose nucToAaMuts call annotates the transcript ("" for none). For main CDS transcripts, feature_id == gene_id (locus_tag). For mature peptide transcripts, feature_id == protein_id. aa_length excludes the stop codon (cds_length // 3 - 1). @@ -230,6 +231,7 @@ def load_gff3(gff_path: str) -> tuple[list, list]: "gene_id": meta["gene_id"], "feature_id": meta["gene_id"], # Feature_ID = locus_tag, matching NCBI VCF shape "protein_id": meta["protein_id"], # matches Nextclade cdsName + "nextclade_cds": meta["protein_id"], "strand": meta["strand"], "segments": segs_s, "is_mature": False, @@ -247,6 +249,7 @@ def load_gff3(gff_path: str) -> tuple[list, list]: "gene_id": meta["gene_id"], "feature_id": me["protein_id"], # mature peptide uses its own protein_id "protein_id": me["protein_id"], + "nextclade_cds": "", "strand": me["strand"], "segments": [(me["begin_0"], me["end_0"])], "is_mature": True, @@ -563,15 +566,20 @@ def build_header(chrom: str, ref_len: int, ref_path: str, sample: str) -> str: # Per-variant ANN field builders # --------------------------------------------------------------------------- +def _hgvs_p(ref_aa: str, alt_aa: str, aa_pos_1: int) -> str: + annotation, _ = _classify_aa_consequence(ref_aa, alt_aa, aa_pos_1) + if annotation == "synonymous_variant": + return _format_hgvs_p_syn(ref_aa, aa_pos_1) + return _format_hgvs_p_sub(ref_aa, aa_pos_1, alt_aa) + + def _coding_entry( - tx: dict, ref_nt: str, alt_nt: str, cds_local_1: int, rank_str: str, aa_change: tuple, + tx: dict, ref_nt: str, alt_nt: str, cds_local_1: int, rank_str: str, + ref_aa: str, alt_aa: str, aa_pos_1: int, ) -> tuple[str, str]: """Return (ANN entry, HGVS.p) for a substitution inside a coding transcript.""" - ref_aa, alt_aa, aa_pos_1 = aa_change[:3] annotation, impact = _classify_aa_consequence(ref_aa, alt_aa, aa_pos_1) - hgvs_p = (_format_hgvs_p_syn(ref_aa, aa_pos_1) - if annotation == "synonymous_variant" - else _format_hgvs_p_sub(ref_aa, aa_pos_1, alt_aa)) + hgvs_p = _hgvs_p(ref_aa, alt_aa, aa_pos_1) entry = _build_ann_entry( allele=alt_nt, annotation=annotation, impact=impact, gene_name=tx["gene_name"], gene_id=tx["gene_id"], @@ -599,69 +607,51 @@ def _ann_field_for_sub( """Return the 'ANN=...' INFO sub-field for a nucleotide substitution, followed by MNV_CORRECTED/MNV_ORIGINAL_HGVS_P when a shared codon changes its annotation.""" entries: list[str] = [] - # (entry, single-base entry, single-base HGVS.p, feature) for codons holding 2+ substitutions - mnv: list[tuple[str, str, str, str]] = [] - - def add_coding(tx, cds_local_1, rank_str, aa_change, single_base): - entry, _ = _coding_entry(tx, ref_nt, alt_nt, cds_local_1, rank_str, aa_change) - entries.append(entry) - if aa_change[3] > 1: - mnv.append((entry, *_coding_entry(tx, ref_nt, alt_nt, cds_local_1, rank_str, single_base), - tx["feature_id"])) + originals: list[str] = [] # feature:HGVS.p before codon correction, per codon with 2+ substitutions + corrected = False - # --- 1. Main CDS transcripts: in-CDS or intronic --- + # --- 1. Coding, intronic, or upstream of a mature peptide --- + # load_gff3 lists every CDS before any mature peptide, which keeps the ANN order. for tx in transcripts: - if tx["is_mature"]: - continue cds_local_1, exon_rank, exon_count = _cds_pos_from_genomic(tx, pos_0) if cds_local_1 is not None: - rank_str = f"{exon_rank}/{exon_count}" - codon = _codon_aa_change(ref_seq, tx, pos_0, qry_subs) - # Nextclade's own call translates the query codon, so it needs no correction. - aa_changes = [c for c in nuc_to_aa.get(str(pos_0), []) - if c["cdsName"] == tx["protein_id"]] - if aa_changes: - ch = aa_changes[0] - codon = (ch["refAa"], ch["qryAa"], ch["pos"] + 1, codon[3]) - single_base = codon - else: - single_base = _codon_aa_change(ref_seq, tx, pos_0, {pos_0: alt_nt}) - add_coding(tx, cds_local_1, rank_str, codon, single_base) + ref_aa, alt_aa, aa_pos_1, n_subs = _codon_aa_change(ref_seq, tx, pos_0, qry_subs) + call = next((c for c in nuc_to_aa.get(str(pos_0), []) + if c["cdsName"] == tx["nextclade_cds"]), None) + if call: + ref_aa, alt_aa = call["refAa"], call["qryAa"] + entry, hgvs_p = _coding_entry(tx, ref_nt, alt_nt, cds_local_1, + f"{exon_rank}/{exon_count}", ref_aa, alt_aa, aa_pos_1) + entries.append(entry) + if n_subs > 1: + if call: + # Nextclade translated the whole query codon, so its call is its own original. + before = hgvs_p + else: + s_ref, s_alt, s_pos, _ = _codon_aa_change(ref_seq, tx, pos_0, {pos_0: alt_nt}) + before = _hgvs_p(s_ref, s_alt, s_pos) + originals.append(f"{tx['feature_id']}:{before}") + corrected = corrected or before != hgvs_p continue - # Check for intron position in spliced transcripts - intron = _intron_info(tx, pos_0) - if intron is not None: - cds_end_1, offset, intron_rank, n_introns = intron - rank_str = f"{intron_rank}/{n_introns}" - hgvs_c = _format_hgvs_c_intron(cds_end_1, offset, ref_nt, alt_nt) - entries.append(_build_ann_entry( - allele=alt_nt, annotation="intron_variant", impact="MODIFIER", - gene_name=tx["gene_name"], gene_id=tx["gene_id"], - feature_id=tx["feature_id"], - transcript_biotype="protein_coding", - hgvs_c=hgvs_c, hgvs_p="", - cds_pos=0, cds_len=0, aa_pos=0, aa_len=0, - rank_str=rank_str, - )) - - # --- 2. Mature peptide transcripts: in-region or upstream --- - for tx in transcripts: if not tx["is_mature"]: - continue - seg_begin = tx["segments"][0][0] - seg_end = tx["segments"][-1][1] - - if seg_begin <= pos_0 < seg_end: - cds_local_1, exon_rank, exon_count = _cds_pos_from_genomic(tx, pos_0) - if cds_local_1 is None: - continue - rank_str = f"{exon_rank}/{exon_count}" - add_coding(tx, cds_local_1, rank_str, - _codon_aa_change(ref_seq, tx, pos_0, qry_subs), - _codon_aa_change(ref_seq, tx, pos_0, {pos_0: alt_nt})) - elif pos_0 < seg_begin: - distance = seg_begin - pos_0 + # Check for intron position in spliced transcripts + intron = _intron_info(tx, pos_0) + if intron is not None: + cds_end_1, offset, intron_rank, n_introns = intron + rank_str = f"{intron_rank}/{n_introns}" + hgvs_c = _format_hgvs_c_intron(cds_end_1, offset, ref_nt, alt_nt) + entries.append(_build_ann_entry( + allele=alt_nt, annotation="intron_variant", impact="MODIFIER", + gene_name=tx["gene_name"], gene_id=tx["gene_id"], + feature_id=tx["feature_id"], + transcript_biotype="protein_coding", + hgvs_c=hgvs_c, hgvs_p="", + cds_pos=0, cds_len=0, aa_pos=0, aa_len=0, + rank_str=rank_str, + )) + elif pos_0 < tx["segments"][0][0]: + distance = tx["segments"][0][0] - pos_0 hgvs_c = f"c.-{distance}{ref_nt}>{alt_nt}" entries.append(_build_ann_entry( allele=alt_nt, annotation="upstream_gene_variant", impact="MODIFIER", @@ -673,7 +663,7 @@ def add_coding(tx, cds_local_1, rank_str, aa_change, single_base): distance=str(distance), warnings=tx["warnings"], )) - # --- 3. Upstream entries for main CDS transcripts --- + # --- 2. Upstream entries for main CDS transcripts --- for tx in transcripts: if tx["is_mature"]: continue @@ -691,7 +681,7 @@ def add_coding(tx, cds_local_1, rank_str, aa_change, single_base): distance=str(distance), )) - # --- 4. Intergenic (when outside all gene boundaries) --- + # --- 3. Intergenic (when outside all gene boundaries) --- if not _is_in_gene(genes, pos_0): gname, gid, fid = _intergenic_name(genes, pos_0) entries.append(_build_ann_entry( @@ -706,9 +696,8 @@ def add_coding(tx, cds_local_1, rank_str, aa_change, single_base): if not entries: return "" ann = "ANN=" + ",".join(entries) - if any(entry != single for entry, single, _, _ in mnv): - original = "|".join(f"{feature}:{hgvs_p}" for _, _, hgvs_p, feature in mnv) - ann += f";MNV_CORRECTED;MNV_ORIGINAL_HGVS_P={original}" + if corrected: + ann += ";MNV_CORRECTED;MNV_ORIGINAL_HGVS_P=" + "|".join(originals) return ann From d0a89c36c8977ffb086c1cb19baa7cf6536f9118 Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Thu, 1 Oct 2026 17:09:53 -0400 Subject: [PATCH 09/11] Translate shared codons from the nucleotide alignment Nextclade's peptide alignment can gap a codon its nucleotide alignment keeps: at the LPAI H5 cleavage site it called K345- for an ACA (Thr) codon. Output now matches main except two NA deletion rows. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Rrtp3W2XVRynKEUHkxs6e9 --- AGENTS.md | 8 +++-- README.md | 2 +- bin/nextclade_to_freebayes_vcf.py | 10 +++--- docs/vcf_annotation_and_flumut_insertion.md | 23 +++++++++----- tests/test_nextclade_to_freebayes_vcf.py | 34 +++++++++++++++++++-- 5 files changed, 58 insertions(+), 19 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9d35d7d..30d4810 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -148,9 +148,11 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and - In the SnpEff-annotated PAFTOOLS and SHORT_READ VCFs, individual SNVs in one codon can have incorrect amino-acid annotations until `bin/fix_mnv_codons.py` runs; their final VCFs must remain downstream of that step. The Nextclade - converter translates each codon with all its substitutions applied (and keeps - Nextclade's own call where `nucToAaMuts` has one), writing the same - `MNV_CORRECTED`/`MNV_ORIGINAL_HGVS_P` fields itself. + converter translates each codon with all its substitutions applied, keeping + Nextclade's own `nucToAaMuts` call only for a codon with one substitution, and + writes the same `MNV_CORRECTED`/`MNV_ORIGINAL_HGVS_P` fields itself. Nextclade's + peptide alignment can gap a codon its nucleotide alignment keeps, so its call + must not override a shared codon. - `bin/merge_flumut_mutations.py` is the only implementation. It may add `FLUMUT=` to matching rows, but must not add, remove, reorder, or otherwise rewrite variant rows. The earlier implementation that inserted synthetic rows diff --git a/README.md b/README.md index 8df36f8..927f068 100644 --- a/README.md +++ b/README.md @@ -273,7 +273,7 @@ The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder `FORMAT/GQ` as | `nextclade/json/*_nextclade.json` | Per-segment Nextclade JSON split out from the cross-sample batch result | | `nextclade/tsv/*_nextclade.tsv` | Per-segment Nextclade TSV split out from the cross-sample batch result | | `qc/*_identity_qc.tsv` | Per-segment identity of the consensus to the Nextclade dataset reference, taken from Nextclade's alignment: exact matches over the reference plus inserted bases, with Ns and ambiguous bases counted as mismatches. PASS/WARN/FAIL at `--consensus_identity_pass_pct` (default 90) and `--consensus_identity_warn_pct` (default 70); a segment Nextclade could not align is FAIL with `identity_pct` `NA` | -| `nextclade/*_final.vcf` | Per-segment final VCF from the Nextclade-to-VCF conversion. Rows in a codon holding more than one substitution carry the combined change, plus `MNV_CORRECTED` and `MNV_ORIGINAL_HGVS_P` where it differs from the single-base call | +| `nextclade/*_final.vcf` | Per-segment final VCF from the Nextclade-to-VCF conversion. Rows in a codon holding more than one substitution carry the combined change, plus `MNV_CORRECTED` and `MNV_ORIGINAL_HGVS_P` where it differs from the call before codon correction (Nextclade's, otherwise the single-base change) | | `nextclade/no_mature_peptide/*_nc_nomat_final.vcf` | *Optional* (`--emit_no_mature_vcf`): the same records annotated without mature peptides, so HA carries only its HA0 `ANN=` entry and no HA1/HA2 ones | | `nextclade/flumut_annotated/*_flumut_annotated.vcf` | *Optional* (`--merge_flumut_mutations`): the same VCF with a `FLUMUT=` INFO field (`Protein\|AA_Pos\|Alt_AA\|Mutation_Name\|DirectEffect\|AssociatedEffect`) on rows whose amino-acid change matches a curated FluMutDB marker | | `nextclade/*_watchlist_hits.tsv` | *Optional* (`--watchlist_dir`): watchlist substitutions found in the final VCF. Independent of `--merge_flumut_mutations`: `FLUMUT=` is a curated annotation written *into* the VCF, this is a custom list scanned *out* of it | diff --git a/bin/nextclade_to_freebayes_vcf.py b/bin/nextclade_to_freebayes_vcf.py index 6b25715..5fc60a8 100755 --- a/bin/nextclade_to_freebayes_vcf.py +++ b/bin/nextclade_to_freebayes_vcf.py @@ -553,8 +553,8 @@ def build_header(chrom: str, ref_len: int, ref_path: str, sample: str) -> str: 'another substitution; ANN HGVS.p, annotation and impact describe the combined codon">' ) lines.append( - '##INFO=' ) for fid, number, typ, desc in _FREEBAYES_FORMAT: @@ -618,15 +618,15 @@ def _ann_field_for_sub( ref_aa, alt_aa, aa_pos_1, n_subs = _codon_aa_change(ref_seq, tx, pos_0, qry_subs) call = next((c for c in nuc_to_aa.get(str(pos_0), []) if c["cdsName"] == tx["nextclade_cds"]), None) - if call: + # Nextclade's peptide alignment can gap a codon its nucleotide alignment keeps + if call and n_subs == 1: ref_aa, alt_aa = call["refAa"], call["qryAa"] entry, hgvs_p = _coding_entry(tx, ref_nt, alt_nt, cds_local_1, f"{exon_rank}/{exon_count}", ref_aa, alt_aa, aa_pos_1) entries.append(entry) if n_subs > 1: if call: - # Nextclade translated the whole query codon, so its call is its own original. - before = hgvs_p + before = _hgvs_p(call["refAa"], call["qryAa"], aa_pos_1) else: s_ref, s_alt, s_pos, _ = _codon_aa_change(ref_seq, tx, pos_0, {pos_0: alt_nt}) before = _hgvs_p(s_ref, s_alt, s_pos) diff --git a/docs/vcf_annotation_and_flumut_insertion.md b/docs/vcf_annotation_and_flumut_insertion.md index 9b470ba..0d5bbea 100644 --- a/docs/vcf_annotation_and_flumut_insertion.md +++ b/docs/vcf_annotation_and_flumut_insertion.md @@ -118,8 +118,9 @@ Rows are sorted by `POS` before return. For each substitution the script walks every transcript and appends an `ANN` entry per feature it touches (so a base in an overlap region yields several entries): -1. **Inside a main CDS** — take the AA change from `nucToAaMuts` when present, otherwise - translate the codon with every substitution in it applied (`_codon_aa_change`). Classify the +1. **Inside a main CDS** — take the AA change from `nucToAaMuts` when the codon holds this one + substitution, otherwise translate the codon with every substitution in it applied + (`_codon_aa_change`); see *Substitutions sharing a codon* below. Classify the consequence (`_classify_aa_consequence`: `synonymous_variant` / `missense_variant` / `stop_gained` / `stop_lost` / `start_lost`, with LOW/MODERATE/HIGH impact) and format `HGVS.c` (`c.>`) and `HGVS.p`. `Rank` = exon/exon-count. @@ -144,12 +145,18 @@ and the `ANN`, `MNV_CORRECTED` and `MNV_ORIGINAL_HGVS_P` `##INFO` definitions. ### Substitutions sharing a codon -`nucToAaMuts` and `_codon_aa_change` both translate the whole query codon, so every -substitution in a codon carries the combined change. Where an entry differs from the change -its substitution would make alone, the row also carries `MNV_CORRECTED` and -`MNV_ORIGINAL_HGVS_P` (`featureID:HGVS.p` for every feature whose codon holds more than one -substitution, pipe-separated): the same fields `fix_mnv_codons.py` writes on the PAFTOOLS -and SHORT_READ VCFs. +Every substitution in a codon holding more than one carries the combined change, translated +by `_codon_aa_change` from the nucleotide alignment. Nextclade's own call is not used there: +its peptide alignment is computed separately and can place a gap on a codon the nucleotide +alignment keeps (at the H5 cleavage site of North American LPAI viruses it reports `K345-` +where the sample codon is ACA, Thr). + +Where an entry differs from the call made before codon correction — Nextclade's where it +made one, otherwise the change the substitution would make alone — the row also carries +`MNV_CORRECTED` and `MNV_ORIGINAL_HGVS_P` (that call as `featureID:HGVS.p`, for every +feature whose codon holds more than one substitution, pipe-separated). These are the same +fields, with the same meaning, that `fix_mnv_codons.py` writes on the PAFTOOLS and SHORT_READ +VCFs. --- diff --git a/tests/test_nextclade_to_freebayes_vcf.py b/tests/test_nextclade_to_freebayes_vcf.py index ab558da..80d2403 100644 --- a/tests/test_nextclade_to_freebayes_vcf.py +++ b/tests/test_nextclade_to_freebayes_vcf.py @@ -67,7 +67,7 @@ def _write_dataset(directory: Path, gff3: str) -> Path: def _write_json(path: Path, positions=SUB_POSITIONS, sample: str = "SAMPLE01", - qry: dict[int, str] | None = None) -> Path: + qry: dict[int, str] | None = None, nuc_to_aa: dict | None = None) -> Path: """A minimal nextclade v3 result carrying one substitution per position (qry sets the base).""" qry = qry or {} substitutions = [ @@ -81,7 +81,7 @@ def _write_json(path: Path, positions=SUB_POSITIONS, sample: str = "SAMPLE01", "substitutions": substitutions, "deletions": [], "insertions": [], - "nucToAaMuts": {}, + "nucToAaMuts": nuc_to_aa or {}, "aaDeletions": [], "aaInsertions": [], "frameShifts": [], @@ -238,6 +238,36 @@ def test_lone_substitution_carries_no_mnv_fields(mnv_records): assert "MNV_CORRECTED" not in info and "MNV_ORIGINAL_HGVS_P" not in info +def _calls(alt_aa: str) -> dict: + """nucToAaMuts giving Nextclade's TESTHA call alt_aa for the shared codon at 107/108.""" + call = [{"cdsName": "TESTHA", "refAa": "Y", "qryAa": alt_aa, "pos": 29}] + return {"107": call, "108": call} + + +def test_shared_codon_overrides_a_disagreeing_nextclade_call(tmp_path, dataset): + """Nextclade's peptide alignment can gap a codon its nucleotide alignment keeps (HA K345).""" + lone = {"400": [{"cdsName": "TESTHA", "refAa": "V", "qryAa": "-", "pos": 126}]} + json_path = _write_json(tmp_path / "mnv.json", positions=MNV_POSITIONS, qry=MNV_QRY, + nuc_to_aa=_calls("-") | lone) + first, second, third = _records(_run(dataset, json_path, tmp_path / "out")[0]) + for record in (first, second): + assert _hgvs_p(record) == {"TESTLOCUS": "p.Y30A", "HA1": "p.Y29A"} + assert "MNV_CORRECTED" in _info(record) + assert _info(first)["MNV_ORIGINAL_HGVS_P"] == "TESTLOCUS:p.Y30-|HA1:p.Y29D" + # A lone substitution keeps Nextclade's call. + assert _hgvs_p(third)["TESTLOCUS"] == "p.V127-" + + +def test_shared_codon_matching_nextclade_is_not_flagged(tmp_path): + """MNV_CORRECTED compares against Nextclade's call, not the single-base change.""" + dataset = _write_dataset(tmp_path / "dataset", GFF3_NO_MATURE) + json_path = _write_json(tmp_path / "mnv.json", positions=MNV_POSITIONS[:2], qry=MNV_QRY, + nuc_to_aa=_calls("A")) + for record in _records(_run(dataset, json_path, tmp_path / "out")[0]): + assert _hgvs_p(record) == {"TESTLOCUS": "p.Y30A"} + assert "MNV_CORRECTED" not in _info(record) and "MNV_ORIGINAL_HGVS_P" not in _info(record) + + def test_mnv_fields_are_declared(tmp_path, dataset, nextclade_json): out, _ = _run(dataset, nextclade_json, tmp_path / "out") header = out.read_text() From 6becf092e6a0dcfad1575f8c8688b61639cb4ee9 Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Thu, 1 Oct 2026 17:11:51 -0400 Subject: [PATCH 10/11] Drop the FluMut merge's MNV_UNRESOLVED skip The merge reads only Nextclade VCFs, which never carry the flag. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Rrtp3W2XVRynKEUHkxs6e9 --- AGENTS.md | 7 +++-- bin/merge_flumut_mutations.py | 13 --------- docs/vcf_annotation_and_flumut_insertion.md | 15 ++++------- docs/watchlists.md | 4 +-- tests/test_merge_flumut_mutations.py | 30 ++------------------- tests/test_mutation_watcher.py | 11 ++++++++ 6 files changed, 23 insertions(+), 57 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 30d4810..e0b2730 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -160,9 +160,8 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and `merge_flumut_mutations_v2.py`); no `_v2` suffix should reappear. The coordinate map's `target_cds` block existed only for the old script and is gone too. -- Rows flagged `MNV_UNRESOLVED` by `bin/fix_mnv_codons.py` are skipped, not - matched: their HGVS.p is known to be wrong, so a marker hit there would be a - false positive. Every row of a codon holding more than one substitution +- The merge reads only Nextclade VCFs, which never carry `MNV_UNRESOLVED`, so it + has no skip for that flag. Every row of a codon holding more than one substitution carries the same combined HGVS.p and so the same `FLUMUT=` entry — downstream marker counts must de-duplicate. - **Watchlist hits never enter the VCF.** `FLUMUT=` comes from a versioned @@ -170,7 +169,7 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and is a CSV we edit, and merging its hits in would make the same consensus produce different VCFs depending on that file. Hits go to `*_watchlist_hits.tsv`. The shared part — `ANN=` parsing, both HGVS.p - spellings, gene canonicalisation, the `MNV_UNRESOLVED` skip — lives once in + spellings, gene canonicalisation, the watchlist's `MNV_UNRESOLVED` skip — lives once in `bin/lib/aa_markers.py`; do not let a consumer grow its own copy. - A watchlist `gene` is a **protein**, not a segment, and gene resolution falls back from ANN `Gene_Name` to `Feature_ID` but **never** from a parent gene to diff --git a/bin/merge_flumut_mutations.py b/bin/merge_flumut_mutations.py index bee04c7..1de5c4d 100755 --- a/bin/merge_flumut_mutations.py +++ b/bin/merge_flumut_mutations.py @@ -38,7 +38,6 @@ HGVS_P_RE_3LETTER, aa_changes_from_info, ann_entries, - has_info_flag, parse_hgvs_p, ) from lib.flu_gene_names import canonical_gene # noqa: E402 @@ -389,14 +388,7 @@ def annotate_vcf(vcf_path: str, segment: str, seg_map: dict, gene_maps: dict, lo markers: set = set() unmapped_genes: set = set() unparsed_hgvs_p: list[str] = [] - mnv_unresolved = 0 for info in df["INFO"].astype(str): - if has_info_flag(info, "MNV_UNRESOLVED"): - # fix_mnv_codons.py left this row's HGVS.p at a value already determined to - # be wrong; matching a curated marker against it would be a false positive. - mnv_unresolved += 1 - info_out.append(info) - continue field = row_flumut_field(info, gene_maps, lookup, want_effects, alias_index=alias_index, subunit_parent_genes=subunit_parent_genes, @@ -410,11 +402,6 @@ def annotate_vcf(vcf_path: str, segment: str, seg_map: dict, gene_maps: dict, lo else: info_out.append(info) - if mnv_unresolved: - log.warning("%d row(s) carry the MNV_UNRESOLVED flag; their HGVS.p is known to be " - "wrong, so they were skipped rather than matched against FluMutDB", - mnv_unresolved) - if unparsed_hgvs_p: # Without this an unhandled HGVS.p spelling is indistinguishable from "this VCF has # no coding changes": silence rather than an error. diff --git a/docs/vcf_annotation_and_flumut_insertion.md b/docs/vcf_annotation_and_flumut_insertion.md index 0d5bbea..5b5c600 100644 --- a/docs/vcf_annotation_and_flumut_insertion.md +++ b/docs/vcf_annotation_and_flumut_insertion.md @@ -8,7 +8,7 @@ freebayes-style VCF and tag the curated FluMutDB markers among them: | [`bin/nextclade_to_freebayes_vcf.py`](../bin/nextclade_to_freebayes_vcf.py) | Convert a Nextclade JSON into one freebayes-format VCF per sample with snpEff-style `ANN=` annotations. | | [`bin/merge_flumut_mutations.py`](../bin/merge_flumut_mutations.py) | Add a `FLUMUT=` INFO field to rows whose AA change matches a FluMutDB marker. Run by the `merge_flumut_mutations` process. | | [`bin/lib/flu_gene_names.py`](../bin/lib/flu_gene_names.py) | The canonical gene-key aliases (`NS-1`→`NS1`, `NS2`/`NS-2`→`NEP`), shared by the coordinate map builder and the merge so the two cannot drift. | -| [`bin/lib/aa_markers.py`](../bin/lib/aa_markers.py) | `ANN=` extraction, both HGVS.p spellings, and the `MNV_UNRESOLVED` flag test, shared by the merge and the watchlist scanner for the same reason. | +| [`bin/lib/aa_markers.py`](../bin/lib/aa_markers.py) | `ANN=` extraction and both HGVS.p spellings, shared by the merge and the watchlist scanner for the same reason, plus the `MNV_UNRESOLVED` flag test the scanner uses. | Both VCF-writing scripts share the I/O helpers in [`bin/lib/tools.py`](../bin/lib/tools.py), which round-trip a VCF byte-for-byte. `bin/` holds only entry points — Nextflow puts it on @@ -267,7 +267,7 @@ one are independent. ### How it runs (`main`) -Steps 1 and 2 run once per call; `annotate_vcf` repeats steps 3 to 6 for each `--vcf`. +Steps 1 and 2 run once per call; `annotate_vcf` repeats steps 3 to 5 for each `--vcf`. 1. **Load this segment's block** from the coordinate map (`load_segment_map`), warning if it carries `stale_annotation`. @@ -279,12 +279,7 @@ Steps 1 and 2 run once per call; `annotate_vcf` repeats steps 3 to 6 for each `- pair columns 1:1, so the inverse is well defined; a collision would be warned about and resolved to the lowest source position. 3. **Read the VCF** with `tools.read_vcf` → DataFrame + header lines. -4. **Skip rows flagged `MNV_UNRESOLVED`** (`has_info_flag`). `fix_mnv_codons.py` sets that - flag on PAFTOOLS and SHORT_READ VCFs when it identified a multi-SNV codon but could not - compute the combined change (the Nextclade converter always has the reference codon and - never sets it), leaving the row's `HGVS.p` at a value the pipeline has already determined to be wrong; - matching a curated marker against it would manufacture a false positive. -5. **Per row, per `ANN` entry** (`row_flumut_field` → `aa_changes_from_info`): take +4. **Per row, per `ANN` entry** (`row_flumut_field` → `aa_changes_from_info`): take `Gene_Name` and `Feature_ID` — resolved as described under *Gene names* below — and the `HGVS.p` substitution, skipping synonymous changes and entries with no `HGVS.p`. Both the 1-letter (`p.E627K`, what the converter emits) and 3-letter (`p.Glu627Lys`, snpEff's @@ -297,7 +292,7 @@ Steps 1 and 2 run once per call; `annotate_vcf` repeats steps 3 to 6 for each `- mutation count that classifies it as direct or associated; `make_marker_lookup` folds those rows into per-mutation effect sets. Every join is a `LEFT JOIN` so that detecting a mutation never depends on it having a published effect. -6. **Write** with `tools.write_vcf`, inserting the `##INFO=` definition +5. **Write** with `tools.write_vcf`, inserting the `##INFO=` definition immediately before the `#CHROM` line. `ref_seq` is deliberately **not** part of the match: a marker is defined by its alt allele, @@ -415,7 +410,7 @@ recording which version of the list produced it. That breaks reproducibility for that is already available, per sample, in its own file. What the two legitimately share is the mechanics of reading `ANN=` — HGVS.p parsing in both -spellings, gene canonicalisation, and the `MNV_UNRESOLVED` skip. Those live once, in +spellings and gene canonicalisation. Those live once, in [`bin/lib/aa_markers.py`](../bin/lib/aa_markers.py), for the reason recorded under `flu_gene_names.py`: a second copy is free to drift, and the drift is silent. diff --git a/docs/watchlists.md b/docs/watchlists.md index aaffa12..2b81bd1 100644 --- a/docs/watchlists.md +++ b/docs/watchlists.md @@ -124,8 +124,8 @@ a file we edit. Merging watchlist hits into the VCF would make the same consensu different VCFs depending on the contents of a CSV, which breaks every downstream assumption about a VCF being reproducible from its inputs. -The two share the mechanics of reading `ANN=` — HGVS.p parsing, gene canonicalisation, the -`MNV_UNRESOLVED` skip — and those live in exactly one place, `bin/lib/aa_markers.py`. +The two share the mechanics of reading `ANN=` — HGVS.p parsing and gene canonicalisation — +and those live in exactly one place, `bin/lib/aa_markers.py`. ## Rows that are skipped diff --git a/tests/test_merge_flumut_mutations.py b/tests/test_merge_flumut_mutations.py index 7b5a14c..e0e0755 100644 --- a/tests/test_merge_flumut_mutations.py +++ b/tests/test_merge_flumut_mutations.py @@ -545,23 +545,9 @@ def test_non_substitution_hgvs_p_is_not_reported_as_a_parse_failure(tmp_path, hg # MNV provenance # --------------------------------------------------------------------------- -def test_mnv_unresolved_rows_are_skipped(tmp_path): - """fix_mnv_codons.py sets this flag on rows whose HGVS.p it knows to be wrong but could - not correct. Matching a curated marker against one manufactures a false positive.""" - row = vcf_row(1881, [ann("PB2", "p.E627K")]).replace(BASE_INFO, - BASE_INFO + ";MNV_UNRESOLVED") - vcf = write_vcf(tmp_path / "in.vcf", [row]) - cmap = write_coord_map(tmp_path / "map.json", "PB2", {"PB2": identity_map()}) - - proc, out = run_script(tmp_path, vcf, cmap, "PB2") - assert proc.returncode == 0, proc.stderr - assert flumut_fields(out) == [None] - assert "MNV_UNRESOLVED" in proc.stderr - - def test_mnv_corrected_rows_are_still_annotated(tmp_path): - """Only the *unresolved* flag suppresses a row; a corrected MNV carries a HGVS.p the - pipeline stands behind. The substring must not match the wrong key.""" + """A corrected MNV carries an HGVS.p the pipeline stands behind, so it is matched like + any other row.""" row = vcf_row(1881, [ann("PB2", "p.E627K")]).replace(BASE_INFO, BASE_INFO + ";MNV_CORRECTED") vcf = write_vcf(tmp_path / "in.vcf", [row]) @@ -728,7 +714,6 @@ def test_ann_parsing_comes_from_the_shared_module(): assert merge_mod.parse_hgvs_p is aa_markers.parse_hgvs_p assert merge_mod.ann_entries is aa_markers.ann_entries - assert merge_mod.has_info_flag is aa_markers.has_info_flag assert merge_mod.aa_changes_from_info is aa_markers.aa_changes_from_info @@ -780,14 +765,3 @@ def test_aa_changes_keeps_one_entry_per_feature_of_the_same_gene(): ]) def test_parse_hgvs_p(hgvs_p, expected): assert merge_mod.parse_hgvs_p(hgvs_p) == expected - - -@pytest.mark.parametrize("info,key,expected", [ - ("NS=1;MNV_UNRESOLVED;DP=3", "MNV_UNRESOLVED", True), - ("NS=1;MNV_UNRESOLVED=1", "MNV_UNRESOLVED", True), - ("NS=1;MNV_CORRECTED;DP=3", "MNV_UNRESOLVED", False), - # A substring test would match this; a token test must not. - ("NS=1;ANN=A|x|y|MNV_UNRESOLVED", "MNV_UNRESOLVED", False), -]) -def test_has_info_flag(info, key, expected): - assert merge_mod.has_info_flag(info, key) is expected diff --git a/tests/test_mutation_watcher.py b/tests/test_mutation_watcher.py index 51d81f8..9e9dc43 100644 --- a/tests/test_mutation_watcher.py +++ b/tests/test_mutation_watcher.py @@ -317,6 +317,17 @@ def test_the_flag_is_matched_as_a_token_not_a_substring(tmp_path): assert skipped == 0 +@pytest.mark.parametrize("info,key,expected", [ + ("NS=1;MNV_UNRESOLVED;DP=3", "MNV_UNRESOLVED", True), + ("NS=1;MNV_UNRESOLVED=1", "MNV_UNRESOLVED", True), + ("NS=1;MNV_CORRECTED;DP=3", "MNV_UNRESOLVED", False), + # A substring test would match this; a token test must not. + ("NS=1;ANN=A|x|y|MNV_UNRESOLVED", "MNV_UNRESOLVED", False), +]) +def test_has_info_flag(info, key, expected): + assert watcher.has_info_flag(info, key) is expected + + # --------------------------------------------------------------------------- # Loading the watchlist # --------------------------------------------------------------------------- From d0a26776562d03bbdeefea61706f8543c0ae6544 Mon Sep 17 00:00:00 2001 From: jpalmer37 Date: Thu, 1 Oct 2026 17:31:23 -0400 Subject: [PATCH 11/11] Name short-read watchlist hits by segment Every short-read segment wrote {sample}_{list}_watchlist_hits.tsv, so only the last segment's hits survived. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Rrtp3W2XVRynKEUHkxs6e9 --- AGENTS.md | 5 +++-- docs/watchlists.md | 2 +- modules/shared/mutation_watch.nf | 2 +- workflows/short-read.nf | 6 ++---- 4 files changed, 7 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e0b2730..1331563 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,8 +42,9 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and existing VCF rows. The markers and the source coordinates come from the same package, so they cannot drift apart. - `mutation_watch` scans each final VCF against the custom watchlists in - `--watchlist_dir` (default `assets/watchlists/`) and publishes exactly one - file, `*_watchlist_hits.tsv` (header-only when nothing matches). It runs in all + `--watchlist_dir` (default `assets/watchlists/`) and publishes one + `*_watchlist_hits.tsv` per final VCF and watchlist (header-only when nothing + matches), named by segment from the VCF filename. It runs in all three workflows and is independent of `--merge_flumut_mutations`. ## Repository map diff --git a/docs/watchlists.md b/docs/watchlists.md index 2b81bd1..9957582 100644 --- a/docs/watchlists.md +++ b/docs/watchlists.md @@ -103,7 +103,7 @@ would be a wrong answer with nothing to show it was wrong. ## Output -`${outdir}/${sample_id}/${workflow}/${sample_id}_[segment_]${list}_watchlist_hits.tsv`: +`${outdir}/${sample_id}/${workflow}/${sample_id}_${segment}_${list}_watchlist_hits.tsv`: ``` sample_id segment gene position ref_aa alt_aa mutation chrom pos hgvs_p note diff --git a/modules/shared/mutation_watch.nf b/modules/shared/mutation_watch.nf index cbe6464..9d84dc1 100644 --- a/modules/shared/mutation_watch.nf +++ b/modules/shared/mutation_watch.nf @@ -16,7 +16,7 @@ def watch_scans(sample_id, segments, vcfs, watchlists) { [segments, [vcfs].flatten()].transpose().collectMany { segment, vcf -> [watchlists].flatten().collect { watchlist -> def list_name = watchlist.baseName.replaceAll(/_watchlist$/, '') - [vcf, watchlist, segment, [sample_id, segment, list_name].findAll().join('_')] + [vcf, watchlist, segment, "${sample_id}_${segment}_${list_name}"] } } } diff --git a/workflows/short-read.nf b/workflows/short-read.nf index c5e6453..5197de8 100644 --- a/workflows/short-read.nf +++ b/workflows/short-read.nf @@ -134,13 +134,11 @@ workflow SHORT_READ { ch_watchlists = Channel.fromPath( params.watchlist_dir + "/*watchlist.csv", checkIfExists: true ) - // [sample_id, vcfs] -> [sample_id, [segments], [vcfs]]; each segment is carried - // as '' because short-read post_process_vcf emits one VCF per sample, not per - // segment, and an invented label would be reported in the output. + // [sample_id, vcfs] -> [sample_id, [segments], [vcfs]]; final VCFs end __{SEGMENT}_final.vcf ch_watch_in = post_process_vcf.out.vcf .map { sample_id, vcfs -> vcfs = [vcfs].flatten() - [sample_id, vcfs.collect { '' }, vcfs] + [sample_id, vcfs.collect { it.baseName.split('__')[-1].replaceFirst(/_final$/, '') }, vcfs] } mutation_watch(ch_watch_in, ch_watchlists.collect()) }