Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 9 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,10 @@ The canonical segment set is `PB2`, `PB1`, `PA`, `HA`, `NP`, `NA`, `M`, and
- In FASTA mode, `FASTA_PREP` (`workflows/fasta-prep.nf`) hashes, cleans and
QC-aggregates the consensus input and runs FluMut once, then hands the cleaned
FASTAs and their provenance records to both branches.
- `PAFTOOLS` selects references with BLAST, aligns each segment with
minimap2, calls differences with `paftools.js`, sorts and annotates VCFs,
converts them to the common VCF shape, and corrects MNV codons.
- `PAFTOOLS` selects references with BLAST, then one `paftools_variants`
task per sample aligns each segment with minimap2, calls differences with
`paftools.js`, annotates with SnpEff, converts to the common VCF shape, and
corrects MNV codons.
- `NEXTCLADE` batches samples by segment, runs Nextclade once per segment,
splits per-sample results, and converts JSON to the common VCF shape,
annotating substitutions that share a codon with the combined change.
Expand Down Expand Up @@ -276,8 +277,8 @@ published filenames/content. Use `-profile conda`; `conda.useMamba` is enabled.
starts with `hash_files` (`modules/shared/hash_files.nf`, aliased to
`short_read_hash_files` in SHORT_READ, run by `FASTA_PREP` for the FASTA
branches). The collection chains are inner joins,
and `prepare_consensus`, `blastn` and `paftools_align` all run
`errorStrategy 'ignore'` — so a sample failing any of them publishes no
and `prepare_consensus` and `blastn` both run
`errorStrategy 'ignore'` — so a sample failing either publishes no
provenance file, losing its input hash along with everything else. Use
`join(..., remainder: true)` with a null guard
(the pattern in `workflows/nextclade.nf`) if that record must survive.
Expand Down Expand Up @@ -305,9 +306,6 @@ published filenames/content. Use `-profile conda`; `conda.useMamba` is enabled.
- The GitHub Actions test path is legacy and short-read-only. It references an
absent output-checking script, and `--db` now needs a bundle directory that
nothing in CI provisions; reconcile it with current code before relying on CI.
- The pipeline's process scripts assume GNU userland: `grep -oP`, `sed -i`
without a suffix, `sed` with `\S`, and bash 4+ `${VAR,,}`. They do not run on
a stock macOS shell; test workflow changes on Linux or in a container.
- Known documentation drift includes the PAFTOOLS aggregate filename. Executable
code writes `aggregate/paftools_qc_summary.tsv`; fix code/docs together when
touching these areas.
- The pipeline's process scripts assume GNU userland: `sed -i` without a
suffix, `sed` with `\S`, and bash 4+ `${VAR,,}`. They do not run on a stock
macOS shell; test workflow changes on Linux or in a container.
42 changes: 21 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# fluscan-nf

This is a Nextflow pipeline for Influenza A mutation analysis from either short-read FASTQ data or pre-built consensus FASTA files.
The short-read workflow performs trimming, assembly, mapping, LoFreq variant calling, SnpEff annotation, and consensus generation. In FASTA mode (`--fasta_input`), the pipeline currently launches both a PAFTOOLS branch and a NEXTCLADE branch: PAFTOOLS validates the supplied FASTA, selects a subtype-matched reference with BLAST, aligns per segment with `minimap2`, calls differences with `paftools.js`, sorts with `bcftools`, annotates with SnpEff, and fixes multi-nucleotide variant codons; NEXTCLADE validates the same input, batches sequences by segment for `nextclade run`, and converts the results to VCF, annotating substitutions that share a codon with their combined codon change.
The short-read workflow performs trimming, assembly, mapping, LoFreq variant calling, SnpEff annotation, and consensus generation. In FASTA mode (`--fasta_input`), the pipeline currently launches both a PAFTOOLS branch and a NEXTCLADE branch: PAFTOOLS validates the supplied FASTA, selects a subtype-matched reference with BLAST, aligns per segment with `minimap2`, calls differences with `paftools.js`, annotates with SnpEff, and fixes multi-nucleotide variant codons; NEXTCLADE validates the same input, batches sequences by segment for `nextclade run`, and converts the results to VCF, annotating substitutions that share a codon with their combined codon change.

## Analyses

Expand All @@ -25,7 +25,6 @@ The short-read workflow performs trimming, assembly, mapping, LoFreq variant cal
- Automated reference selection from built-in database via BLAST
- Per-segment alignment using minimap2 (`asm20` mode)
- Variant calling using `paftools.js`
- VCF sorting with `bcftools sort`
- Annotation of amino acid variant calls using SnpEff
- MNV codon correction
- Alignment coverage QC with run-level aggregation
Expand Down Expand Up @@ -60,29 +59,30 @@ flowchart TD
subgraph PAFTOOLS[PAFTOOLS workflow]
direction TB
blastn[blastn reference selection]
split_ref[split_fasta]
minimap[minimap2 asm20]
covqc[coverage QC]
covagg[aggregate coverage summary]
paftools_call[paftools.js call]
sortvcf[bcftools sort]
snpeff[SnpEff]
paf2fb[paftools_to_freebayes_vcf.py]
fixmnv_paf[fix_mnv_codons.py]
paf_final[Final PAFTOOLS VCFs]

subgraph paftools_variants[paftools_variants, one task per sample]
direction TB
minimap[minimap2 asm20]
covqc[coverage QC]
paftools_call[paftools.js call]
snpeff[SnpEff]
paf2fb[paftools_to_freebayes_vcf.py]
fixmnv_paf[fix_mnv_codons.py]

minimap --> covqc
minimap --> paftools_call
paftools_call --> snpeff
snpeff --> paf2fb
covqc --> paf2fb
paf2fb --> fixmnv_paf
end

prepare --> blastn
prepare --> split_ref
blastn --> split_ref
split_ref --> minimap
minimap --> covqc
prepare --> minimap
blastn --> minimap
covqc --> covagg
minimap --> paftools_call
paftools_call --> sortvcf
sortvcf --> snpeff
snpeff --> paf2fb
covqc --> paf2fb
paf2fb --> fixmnv_paf
fixmnv_paf --> paf_final
end

Expand Down Expand Up @@ -257,7 +257,7 @@ The PAFTOOLS and NEXTCLADE VCFs declare and write the placeholder `FORMAT/GQ` as
| `*_ref.fasta` | Multi-FASTA containing the optimal reference sequences selected for each segment |
| `*_blastn.tsv` | BLAST search results against the reference database |
| `*_blastn_filter.tsv` | Filtered BLAST search results |
| `qc/*_coverage_qc.tsv` | Per-segment PAFTOOLS consensus coverage QC with coverage percentage and PASS/WARN/FAIL status |
| `qc/*_coverage_qc.tsv` | Per-segment PAFTOOLS consensus coverage QC with coverage percentage and PASS/WARN/FAIL status. A segment with no BLAST reference, or that minimap2 cannot align, is FAIL and has no VCF |
| `paftools/*_final.vcf` | Per-segment final VCF after `paftools.js`, SnpEff, freebayes-style conversion, and MNV codon correction |
| `paftools/*_watchlist_hits.tsv` | *Optional* (`--watchlist_dir`): watchlist substitutions found in the final VCF |
| `flumut/*_flumut_markers.tsv` | FluMut table of molecular markers identified in the consensus sequences |
Expand Down
78 changes: 17 additions & 61 deletions bin/filter_alignments.py
Original file line number Diff line number Diff line change
@@ -1,110 +1,66 @@
#!/usr/bin/env python3
#%%
import os
import os
import sys
import pandas as pd
import pandas as pd
import argparse
from Bio import SeqIO
import traceback

# Must match the -outfmt the blastn process passes
BLAST_COLUMNS = 'qseqid sseqid pident length qlen slen qstart qend sstart send evalue bitscore score'.split()

def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument('blastn', help='BlastN results file in TSV format')
parser.add_argument('-m', '--metric', default="bitscore", type=str, help='Scoring metric to select best reference/contig. Either bitscore (default), rawscore, or bsr.')
parser.add_argument('-s', '--seqs', required=True, help='Sequences file in FASTA format. Either the contig file or reference BLAST database in FASTA format.')
parser.add_argument('-o','--out_tsv', default=None, help='Filtered results output')
parser.add_argument('-O','--out_fasta', default=None, help='Filtered FASTA output')
parser.add_argument('--min_cov', default=25, type=int, help='Minimum coverage of database reference sequence by contig (percentage, default = 25)')
parser.add_argument('-i','--min_id', default=90, type=int, help='Minimum nucleotide sequence identity between database reference sequence and contig (percentage, default = 90)')
parser.add_argument('-o','--out_tsv', required=True, help='Filtered results output')
parser.add_argument('-O','--out_fasta', required=True, help='Filtered FASTA output')
return parser.parse_args()

def load_blast_format(varname='BLAST_OUTFMT'):
DEFAULT_OUTFMT = 'qseqid sseqid pident length qlen slen qstart qend sstart send evalue bitscore score'
blast_outfmt = os.environ.get(varname)

if not blast_outfmt:
print(f"BLAST_OUTFMT environment variable not found. Using default column format: {DEFAULT_OUTFMT}")
blast_outfmt = DEFAULT_OUTFMT

return blast_outfmt

def parse_blast(filepath):
cols = load_blast_format().split()
blast_df = pd.read_csv(filepath, sep='\t', names=cols)
blast_df = pd.read_csv(filepath, sep='\t', names=BLAST_COLUMNS)

blast_df.insert(0, 'sample_name', os.path.basename(filepath).split("_")[0])

blast_df.insert(3, 'segment', blast_df['sseqid'].str.split("|").str[-1])
return blast_df


#%%
def filter_alignments(blast_results, score_columns, min_cov, min_id):
'''
Find best contig for each genome segment. Returns datasheet with best contigs.'''
print('Filtering alignments...')

def filter_alignments(blast_results):
'''Return the highest-bitscore hit for each reference segment, best first.'''
blast_results = blast_results.drop_duplicates()
filtered = blast_results.loc[blast_results.groupby('segment')['bitscore'].idxmax()]

# apply a min-max scaling to the scoring columns
scoring = blast_results[score_columns]
scoring = (scoring - scoring.min()) / (scoring.max() - scoring.min())
scoring['composite'] = scoring.sum(axis=1)

blast_results = pd.concat([blast_results, scoring[['composite']]], axis=1)

# group by contig and take only the best result
idxmax = blast_results.groupby(['qseqid'])['composite'].idxmax()
filtered = blast_results.loc[idxmax]
if filtered.shape[0] != 8:
print(f"WARNING: expected 8 segments, found {filtered.shape[0]}", file=sys.stderr)

# group by segment and take the top result
idxmax = blast_results.groupby(['segment'])['composite'].idxmax()
filtered = blast_results.loc[idxmax]

filtered = filtered.sort_values('composite', ascending=False).reset_index(drop=True)

if blast_results.shape[0] == 8:
print("WARNING: Did not find 8 segments. Number of alignments present: ", blast_results.shape[0] )

return filtered
return filtered.sort_values('bitscore', ascending=False).reset_index(drop=True)

def write_best_sequence(blast_results, database_path, fasta_out):
'''
Looks up the single top hit in the DB FASTA file and writes them to their own FASTA file.
'''
if blast_results.shape[0] == 0:
print("ERROR: No BLAST results found when trying to write output sequence.", file=sys.stderr)
sys.exit(1)

# Open contigs FASTA and load seqs into dict (key=seq header)
db_seqs = SeqIO.to_dict(SeqIO.parse(database_path, 'fasta'))

final_references = [db_seqs[ref_name] for ref_name in blast_results['sseqid']]

# extract the top hit from the relevant sequence file
SeqIO.write(final_references, fasta_out, 'fasta')


def main():
args = parse_args()

# parse the BLAST file
blast_df = parse_blast(args.blastn)

# exit if no BLAST results are found

if blast_df.shape[0] == 0:
print("WARNING: No data found in blast input file. Exiting.", file=sys.stderr)
sys.exit(1)

blast_df = filter_alignments(blast_df, args.metric.split(","), args.min_cov, args.min_id)

# output blast results
blast_df = filter_alignments(blast_df)

blast_df.to_csv(args.out_tsv, index=False, sep='\t')

# write the best FASTA outputs
write_best_sequence(blast_df, args.seqs, args.out_fasta)


if __name__ == '__main__':
main()
# %%
66 changes: 24 additions & 42 deletions bin/paftools_to_freebayes_vcf.py
Original file line number Diff line number Diff line change
@@ -1,36 +1,24 @@
#!/usr/bin/env python3
"""Convert a paftools.js-format VCF to a freebayes-format VCF.
"""Convert a paftools.js + SnpEff VCF to the freebayes-shaped VCF all three workflows emit.

Paftools VCFs are produced from consensus FASTA input (depth=1) and carry only
QNAME/QSTART/QSTRAND plus snpEff annotations. The downstream summarizer
(summarize_vcf_cyvcf2.py) expects the richer freebayes INFO/FORMAT schema.

This script synthesises all missing freebayes fields with plausible depth-1
defaults (DP=1, AO=1, RO=0, AF=1, MQM=60, …) while preserving every
paftools-specific field (QNAME, QSTART, QSTRAND, ANN, LOF, NMD, MNV_*).

Pipeline position: after paftools.js + snpEff (+ optional fix_mnv_codons.py),
before summarize_vcf_cyvcf2.py.
Consensus calls have depth 1, so every freebayes INFO/FORMAT field is filled with a
depth-1 placeholder (DP=1, AO=1, RO=0, AF=1, MQM=60, ...). The paftools fields (QNAME,
QSTART, QSTRAND) and SnpEff's ANN/LOF/NMD are kept. The segment's alignment coverage QC
row is recorded as a ##fluscan_alignment_coverage header line.
"""

import argparse
import csv
import logging

import numpy as np

try:
from cyvcf2 import VCF, Writer
except ImportError as exc:
raise SystemExit("cyvcf2 is required: conda install -c bioconda cyvcf2") from exc
from cyvcf2 import VCF, Writer

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
log = logging.getLogger(__name__)

# ---------------------------------------------------------------------------
# Freebayes header definitions to inject
# Each tuple: (ID, Number, Type, Description)
# ---------------------------------------------------------------------------

# (ID, Number, Type, Description)
_FREEBAYES_INFO = [
("NS", "1", "Integer", "Number of samples with data"),
("DP", "1", "Integer", "Total read depth at the locus"),
Expand Down Expand Up @@ -86,10 +74,6 @@
("QA", "A", "Integer", "Sum of quality of the alternate observations"),
]

# ---------------------------------------------------------------------------
# Variant classification
# ---------------------------------------------------------------------------

def classify(ref: str, alt: str) -> tuple[str, str, int]:
"""Return (type_str, cigar_str, len_val) for a REF/ALT pair."""
if len(ref) == 1 and len(alt) == 1:
Expand All @@ -106,10 +90,6 @@ def classify(ref: str, alt: str) -> tuple[str, str, int]:
direction = "I" if len(alt) > len(ref) else "D"
return "complex", f"{min(len(ref), len(alt))}M{diff}{direction}", max(len(ref), len(alt))

# ---------------------------------------------------------------------------
# Main conversion
# ---------------------------------------------------------------------------

def _existing_header_ids(vcf: VCF) -> set[str]:
ids = set()
for h in vcf.header_iter():
Expand All @@ -119,7 +99,17 @@ def _existing_header_ids(vcf: VCF) -> set[str]:
return ids


def convert(input_path: str, output_path: str) -> int:
def coverage_header(coverage_tsv: str) -> str:
"""Return the ##fluscan_alignment_coverage line for a one-row coverage QC TSV."""
with open(coverage_tsv) as fh:
row = next(csv.DictReader(fh, delimiter="\t"))
return (
f"##fluscan_alignment_coverage=STATUS={row['status']};"
f"coverage_pct={row['coverage_pct']};range={row['first_pos']}-{row['last_pos']}"
)


def convert(input_path: str, output_path: str, coverage_tsv: str) -> int:
vcf = VCF(input_path)
existing = _existing_header_ids(vcf)

Expand All @@ -132,6 +122,7 @@ def convert(input_path: str, output_path: str) -> int:
vcf.add_format_to_header({"ID": field_id, "Number": number, "Type": typ, "Description": desc})

vcf.add_to_header("##source=paftools_to_freebayes_vcf.py")
vcf.add_to_header(coverage_header(coverage_tsv))

writer = Writer(output_path, vcf)
n_written = 0
Expand Down Expand Up @@ -221,27 +212,18 @@ def convert(input_path: str, output_path: str) -> int:

return n_written

# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def get_args() -> argparse.Namespace:
p = argparse.ArgumentParser(
description=(
"Convert a paftools.js+snpEff VCF to a freebayes-format VCF. "
"Synthesises depth-1 freebayes INFO/FORMAT fields so that "
"summarize_vcf_cyvcf2.py can parse the output unchanged."
)
)
p.add_argument("-i", "--input", required=True, help="Input paftools VCF")
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("-i", "--input", required=True, help="Input paftools + SnpEff VCF")
p.add_argument("-c", "--coverage-tsv", required=True, help="consensus_coverage_qc.py TSV for this segment")
p.add_argument("-o", "--output", required=True, help="Output freebayes-format VCF")
return p.parse_args()


def main() -> None:
args = get_args()
log.info("Converting %s -> %s", args.input, args.output)
n = convert(args.input, args.output)
n = convert(args.input, args.output, args.coverage_tsv)
log.info("Wrote %d records to %s", n, args.output)


Expand Down
Loading
Loading