Skip to content

Optimize the Nextclade workflow - #25

Merged
jpalmer37 merged 11 commits into
mainfrom
refactor/optimize-nextclade-workflow
Oct 5, 2026
Merged

jpalmer37 merged 11 commits into
mainfrom
refactor/optimize-nextclade-workflow

Conversation

@jpalmer37

@jpalmer37 jpalmer37 commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Cuts the NEXTCLADE branch's per-(sample, segment) jobs. The converter corrects MNV codons itself and converts a whole segment batch in one call. Identity QC comes from Nextclade's own alignment. Segment batches are built with splitFasta/collectFile. Watchlists are scanned once per sample and FluMut is merged once per segment. FASTA prep runs once for both FASTA branches. The last four commits come from review: they simplify the converter's codon logic, translate shared codons from the nucleotide alignment, drop an unreachable merge check, and name short-read watchlist hits by segment.

Where to look:

  • bin/nextclade_to_freebayes_vcf.py (_codon_aa_change, _ann_field_for_sub): codon-aware HGVS.p and the MNV_* fields fluscan-db ingests. A codon with 2+ substitutions is translated from the nucleotide alignment; Nextclade's nucToAaMuts call is used only for a lone substitution, because its peptide alignment can gap a codon the nucleotide alignment keeps (HA K345- for an ACA codon at the LPAI H5 cleavage site).
  • bin/nextclade_post_process.py (identity_pct): identity from the Nextclade TSV columns.
  • modules/shared/mutation_watch.nf and its three call sites.
  • workflows/fasta-prep.nf and the main.nf dispatch.

Output changes:

  • aggregate/{paftools,nextclade}_consensus_qc_summary.tsv become one aggregate/consensus_qc_summary.tsv.
  • Nextclade VCF annotations match main except two NA deletion rows in EPI-ISL-63641 (POS 152, 182), which main's fix_mnv_codons.py had rewritten from p.P3fs to missense p.H44Y/p.I54K.
  • Nextclade VCFs are no longer rewritten by cyvcf2: float INFO/FORMAT values read 1.0/60.0/-6.000 instead of 1/60/-6, and the header drops MNV_UNRESOLVED and rewords the MNV_* lines. cyvcf2 (fluscan-db) reads the same values; text or checksum comparisons with older runs will differ.
  • The FluMut merge no longer checks MNV_UNRESOLVED, which Nextclade VCFs never carry.
  • Short-read watchlist hits are one {sample}_{SEGMENT}_{list}_watchlist_hits.tsv per segment. Before, every segment wrote the same name and only one survived.
  • Identity values shift slightly (mean 0.10, max 2.6 points). Samples Nextclade cannot align now report FAIL/NA instead of WARN.
  • NEXTCLADE provenance gains one nextclade_run_batch record per segment.
  • Per-sample JSON/TSV index fields follow the new (sorted) batch order.

Testing:

  • pytest: 241 pass. The 5 failures in test_run_snpeff.py also fail on main.
  • End to end, main vs this branch on 40 GISAID consensus FASTAs (macOS): tasks 1,971 → 685, wall time 22m → 9m33s. Published file sets match apart from the rename above. PAFTOOLS VCFs and PAFTOOLS/Nextclade watchlist hits are byte-identical, and every FLUMUT= value matches. Of 63,640 rows in each Nextclade VCF kind, 2 differ in annotation (the NA deletion rows above).
  • Short read, one Illumina sample (SRR33993032): 8 per-segment hits files, and its NA N329K hit is kept; under the old shared name the NS scan, which runs last, overwrote it. MultiQC and short_read_make_pileup were set to ignore for this run: MultiQC crashes on a polars bug in the local env, and make_pileup.py has been missing from bin/ since Improve QC checking and reporting #15, so SHORT_READ fails on main too.
  • fluscan-db test_current_vcf_compatibility.py: 9 of 10 pass on both main and this branch. test_ha_mat_peptides_get_their_own_rows fails on both because the GsGD bundle names the HA mature peptides by accession, not HA1.

Not yet run on the cluster.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Rrtp3W2XVRynKEUHkxs6e9

jpalmer37 and others added 11 commits September 29, 2026 16:48
The converter now translates each codon with all of its substitutions
applied and writes MNV_CORRECTED/MNV_ORIGINAL_HGVS_P itself, so the
Nextclade branch no longer runs fix_mnv_codons.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nextclade_run_batch now writes every sample's VCF from the batch JSON,
replacing the per-sample loop in nextclade_to_vcf_batch. A segment where
no sample aligns publishes no VCFs instead of failing the task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nextclade_post_process.py derives identity from the batch TSV instead of
realigning every segment with Biopython. On 519 test segments 169 match
exactly and the mean shift is -0.09 points; a segment Nextclade cannot
align is now FAIL with identity NA.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
splitFasta and collectFile write each segment's batch FASTA in the head
job, replacing a split_fasta task per sample and the header rewrite loop
in nextclade_run_batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mutation_watch runs once per sample over all its VCFs and watchlists, and
merge_flumut_mutations.py annotates a whole segment in one call. Published
outputs are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FASTA_PREP runs hash_files, prepare_consensus and flumut once and feeds
PAFTOOLS and NEXTCLADE. The two branch-named consensus QC aggregates become
aggregate/consensus_qc_summary.tsv.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Output is byte-identical to the previous commit on the 40-sample batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nextclade's peptide alignment can gap a codon its nucleotide alignment
keeps: at the LPAI H5 cleavage site it called K345- for an ACA (Thr)
codon. Output now matches main except two NA deletion rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rrtp3W2XVRynKEUHkxs6e9
The merge reads only Nextclade VCFs, which never carry the flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rrtp3W2XVRynKEUHkxs6e9
Every short-read segment wrote {sample}_{list}_watchlist_hits.tsv, so
only the last segment's hits survived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rrtp3W2XVRynKEUHkxs6e9
@jpalmer37
jpalmer37 marked this pull request as ready for review October 5, 2026 19:09
@jpalmer37
jpalmer37 merged commit c74a3a3 into main Oct 5, 2026
1 check failed
@jpalmer37
jpalmer37 deleted the refactor/optimize-nextclade-workflow branch October 15, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant