SPINE Production System

A production system for running SPINE (Scalable Particle Imaging with Neural Embeddings) reconstruction on HPC clusters with SLURM- and PBS-based batch systems.

Overview

SPINE is a deep learning-based reconstruction framework for liquid argon time projection chamber (LArTPC) detectors. This production system provides tools for running SPINE at scale on large datasets using scheduler-managed job arrays.

Quick Start

1. Environment Setup

# Clone the repository
git clone https://github.com/DeepLearnPhysics/spine-prod.git
cd spine-prod

# Configure environment
source configure.sh

SPINE Version Control: Production jobs now run entirely from a tagged SPINE container image. The repository default release is recorded in DEFAULT_SPINE_VERSION; configure.sh exports SPINE_CONTAINER_VERSION from that value and derives the registry tag and default S3DF Singularity image path. This container packages SPINE, OpT0Finder, and runtime dependencies, and jobs invoke the container-provided spine executable directly. The current default is SPINE v1.4.0. Maintained configurations require SPINE v1.4.0 or later. Launchers check the selected executable after environment setup, inside the container when applicable. See the migration notes for ordered stages, named modifiers, and compatibility changes.

Alternative Container Location: You can override the local .sif path or container release before sourcing configure.sh:

export SPINE_CONTAINER_PATH=/path/to/spine.sif
export SPINE_CONTAINER_VERSION=x.y.z
source configure.sh

Updating SPINE Version: Update the repository default in DEFAULT_SPINE_VERSION. The default S3DF image path is derived automatically as /sdf/data/neutrino/images/spine_v<version-with-dashes>.sif. Re-source configure.sh after changing the file; automatically derived version, tag, and path values refresh together, while explicit overrides remain unchanged.

echo x.y.z > DEFAULT_SPINE_VERSION
source configure.sh

2. Basic Job Submission

# Detector shorthand resolves to the latest composite config
./submit.py --config infer/icarus --source /path/to/data.root

# Run on a single file
./submit.py --config infer/icarus/latest --source /path/to/data.root

# Run data-only processing on multiple files (glob)
./submit.py --config infer/icarus/latest --apply-mods data --source /path/to/data/*.root

# Convert LArCV simulation into SPINE truth objects for spinal-tap
./submit.py --config convert/icarus --source /path/to/simulation.root

# Run from a file list (recommended)
./submit.py --config infer/2x2/latest --source-list file_list.txt

3. Advanced Usage

# Use a specific resource profile
./submit.py --config infer/icarus/latest --source data/*.root --profile s3df_turing

# Process all matched files in a single job (default behavior)
./submit.py --config infer/icarus/latest --source data/*.root

# Process multiple files per task explicitly
./submit.py --config infer/icarus/latest --source data/*.root --files-per-task 5

# Resolve the source, keep its first two files, then split those files into tasks
./submit.py --config infer/icarus/latest --source-list files.txt --num-files 2 --files-per-task 1

# Overlay independent ND-LAr primary and secondary samples during inference
./submit.py --config infer/nd-lar/latest --apply-mods joint \
  --source-list primary=primary.txt secondary=secondary.txt

# Survey entries and give each group of up to two primary files a disjoint
# range of secondary events; run at most four tasks concurrently
./submit.py --config infer/nd-lar/latest --apply-mods joint \
  --joint-file-mode balanced \
  --source-list primary=primary.txt secondary=secondary.txt \
  --files-per-task 2 --ntasks 4

# Split the file list across 50 tasks as evenly as possible
./submit.py --config infer/icarus/latest --source data/*.root --ntasks 50

# Start a persistent training run using the loader defined in the config
./submit.py --config train/generic/uresnet/train_240718.yaml \
  --stage train --run-dir /path/to/experiments/uresnet/default \
  --source-list train.txt --val-source-list validation.txt \
  --num-files 2 --val-num-files 1

# Run a multi-stage pipeline
./submit.py --pipeline pipelines/icarus_production_example.yaml

# Ask a running training job to checkpoint and finish successfully
./submit.py --graceful-stop 12345678

# Dry run (see what would be submitted)
./submit.py --config infer/icarus/latest --source test.root --dry-run

# Interactive mode (test locally without batch submission)
./submit.py --interactive --config infer/icarus/latest --source test.root
./submit.py -I --config infer/icarus/latest --source-list files.txt --task-id 2

Balanced joint sharding currently supports local LArCV ROOT files and requires uproot or PyROOT on the submission host. Install the optional dependencies with python3 -m pip install -r requirements-joint.txt; PyROOT is not needed when uproot is installed. The survey runs where submit.py runs, so worker environments need no additional survey dependency. It surveys one reference *_tree per file (the first in sorted tree-name order), without loading the other products' tree metadata. Add --joint-validate-trees to check that all trees have equal counts, or set joint_validate_trees: true in a balanced pipeline stage. A full-validation request rescans cache entries that were counted using only a reference tree; validated counts can serve either mode. It pairs entries in sorted file-path order, processes the smaller source's total, and never reuses secondary entries. If secondary runs out, the last primary group is shortened and later groups are omitted. --files-per-task limits primary files; secondary files are selected as needed and may be shared through disjoint slices. --num-files limits the primary list before surveying. Without --files-per-task, --ntasks splits primary files as usual.

Dry runs also survey inputs and write joint_shards.json in the attempt directory, including counts, ranges and unused totals. entry_counts.json in the run directory caches metadata for retries, invalidated by file size or modification time. Inputs must remain unchanged between survey and execution. Use an unfiltered joint dataset: custom dataset entry filters/selections are not supported by this mode. Balanced mode owns source ranges and uses sequential pairing with incomplete batches retained.

Joint modifiers enable per-input output splitting with base.split_output: true and io.writer.split_source: primary. Each overlay is written once, to its primary input's output file, while both sources' provenance is preserved. For example, ten primary files in one task produce ten outputs if all ten have processed entries. Sharing secondary files across tasks does not change output routing. If secondary entries run out, the last processed primary file may have a partial output and subsequent files produce no output. Use --set base.split_output=false to combine a task's events into one output instead.

The default and minimum supported runtime is SPINE 1.4.4. It includes primary-only joint output routing, training-compatible shower-primary reporting, overall semantic recall, and initial validation/checkpoint selection before training. Explicit runtime overrides must provide SPINE 1.4.4 or newer.

Full-chain evaluation now writes shower_primary_eval_nodes.csv records using the same closest-fragment and purity policy as the model's shower objective. Reports use these for shower_fragment_primary and retain the former matched group-primary comparison as shower_fragment_group_primary. Existing runs need evaluation followed by reporting again to populate the new metric; retraining is unnecessary. Old raw CSVs alone cannot supply the new shower-node records.

SPINE 1.4.4 automatically adds overall semantic recall alongside conditional recall when reducing confusion CSVs. Ghost-aware evaluation bundles retain the ghost rows needed to include deghosting losses in the overall denominator. Existing CSVs with those rows can be reduced again for overall recall, but the updated strict report also requires the new shower-node CSVs.

New training runs with validation configured evaluate and checkpoint the starting model before the first optimizer step. Resumed runs retain their validation history. Set validation.initial: false in the training configuration to opt out.

Interactive Mode

Interactive mode (--interactive or -I) runs SPINE processing directly in your current shell without submitting to the batch scheduler. This is particularly useful for:

Testing configurations before batch submission
Debugging issues with immediate feedback
Small-scale runs on login nodes (use sparingly!)
Config validation with real execution

Interactive mode performs all the same config composition, file chunking, and environment setup as batch mode, but executes locally:

# Test a config on one file
./submit.py -I --config infer/icarus/latest --source /path/to/test.root

# Force container-backed interactive execution
./submit.py -I --interactive-runtime container --config infer/generic/latest --source test.root --world-size 0

# Test with modifiers applied
./submit.py -I --config infer/icarus/latest --source test.root --apply-mods data lite

# Test a specific task from a split file list
./submit.py -I --config infer/icarus/latest --source-list files.txt --files-per-task 5 --task-id 2

Note: Interactive mode is not supported for pipelines. Use --dry-run to preview pipeline submissions.

By default, interactive mode uses the spine executable already on PATH. If spine is unavailable, it falls back to the configured container: first SPINE_CONTAINER_PATH with Singularity/Apptainer if the .sif exists, then SPINE_CONTAINER_TAG with Docker/Podman. Use --interactive-runtime local to require the local executable, or --interactive-runtime container to force container execution. Docker/Podman fallback requests linux/amd64 by default; override SPINE_CONTAINER_PLATFORM if a different platform is needed. For local debugging or batch jobs that should use an unreleased checkout, pass --spine-path /path/to/spine to run /path/to/spine/bin/run.py (or /path/to/spine/bin/spine if present) instead of the spine executable on PATH. The checkout root is added to container bind paths automatically where supported.

Batch jobs and forced interactive container runs resolve SPINE executables inside the container, without selecting installations from the submission host's PATH. This also applies to report, filter, and cache jobs. Explicit --spine-path or SPINE_LOCAL_PATH overrides still select a custom installation. Regenerate existing batch scripts to apply this behavior. Container jobs also set PYTHONNOUSERSITE=1 to ignore Python user-site packages in mounted home directories. Dependencies must be available in the image or through explicit source/PYTHONPATH overrides; local interactive execution is unchanged.

./submit.py --spine-path /path/to/spine -I --interactive-runtime local --config infer/generic/latest --source test.root --world-size 0

EAF Interactive Container Setup

On EAF, Apptainer may be provided from CVMFS rather than as apptainer on PATH. Interactive container mode can be pointed at that executable directly, and can also pass through the extra environment flags needed there:

export SPINE_CONTAINER_RUNTIME_BIN=/cvmfs/eaf.opensciencegrid.org/apptainer/bin/apptainer
export SPINE_CONTAINER_RUNTIME_ARGS="--env LD_PRELOAD= --env LC_ALL=C.UTF-8"

Directory Structure

spine-prod/
├── configure.sh             # Environment setup script
├── submit.py                # Main submission orchestrator
├── README.md                # This file
│
├── config/                  # All SPINE configs (inference & training)
│   ├── cache/               # Stage-cache materialization configs
│   ├── convert/             # Source-to-SPINE conversion bundles
│   │   ├── dune-hd-10kt-1x2x6/  # DUNE-HD truth conversion
│   │   ├── dune-vd-10kt-1x8x6/  # DUNE-VD truth conversion
│   │   └── protodune-hd/    # ProtoDUNE-HD truth conversion
│   ├── infer/               # Inference configs (referenced as infer/...)
│   │   ├── 2x2/             # 2x2 detector configs
│   │   ├── dune-hd-10kt-1x2x6/  # DUNE-HD 10 kt module 1x2x6 configs
│   │   ├── generic/         # Generic (no detector) configs
│   │   ├── icarus/          # ICARUS detector configs
│   │   ├── nd-lar/          # ND-LAr detector configs
│   │   ├── protodune-sp/    # ProtoDUNE single-phase configs
│   │   ├── protodune-vd/    # ProtoDUNE vertical-drift configs
│   │   ├── sbnd/            # SBND detector configs
│   │   └── common/          # Shared configs
│   ├── model/               # Common structures and detector revisions
│   ├── test/                # Evaluation analyzers and report recipes
│   └── train/               # Training configs (referenced as train/...)
├── templates/               # Job templates
│   ├── profiles.yaml        # Resource profiles
│   ├── job_template_s3df.sbatch
│   ├── job_template_nersc.sbatch
│   └── job_template_anl.pbs
│
├── pipelines/               # Multi-stage pipeline definitions
│   ├── generic/              # Generic detector workflow revisions
│   │   └── uresnet_ppn_to_graph_spice_240805.yaml
│   ├── nd-lar/               # ND-LAr cached full-chain training
│   ├── protodune-sp/         # ProtoDUNE-SP cached full-chain training
│   └── icarus_production_example.yaml
│
├── scripts/                 # Utility scripts
├── tests/                   # Test suite
└── runs/                    # Automatic inference run artifacts

Configuration System

SPINE uses YAML configurations throughout. User-facing configs are either versioned .yaml files such as infer/icarus/full_chain_co_260501.yaml or detector shorthands such as infer/icarus or infer/icarus/latest, which generate a composite YAML config at submission time.

Config Organization

<family>/<detector-or-scope>/
├── ...                           # Family-specific configs and components
└── modifier/                     # Optional family-local modifier YAMLs

Named modifiers are resolved within the base configuration's family and scope. For example, convert/icarus/truth_*.yaml uses modifiers under convert/icarus/modifier/, while train/generic/uresnet/train_*.yaml uses modifiers under train/generic/modifier/. Explicit modifier file paths remain available for intentional cross-family composition.

Example: ICARUS Configurations

# Latest composite request (generated at submission time)
infer/icarus/latest

# Latest data-only configuration
infer/icarus/latest --apply-mods data

# Latest NuMI configuration
infer/icarus/latest --apply-mods numi

# Specific version with cosmic overlay
infer/icarus/full_chain_co_260501.yaml

# Data with lite outputs
infer/icarus/full_chain_co_260501.yaml --apply-mods data lite

See individual config directories for detector-specific documentation.

Resource Profiles

Resource profiles define batch resource requirements for different use cases. Profiles are defined in templates/profiles.yaml.

S3DF Node Resources

Understanding the available resources on each partition helps justify the profile configurations:

Partition	GPUs/Node	GPU Type	CPUs/Node	RAM/Node	Resources per GPU
`hopper`	4	H200 (141 GB)	224	1344 GB	56 CPUs, 336 GB
`ampere`	4	A100 (40 GB)	112	952 GB	28 CPUs, 238 GB
`turing`	10	RTX 2080 Ti (11 GB)	40	160 GB	4 CPUs, 16 GB
`milano`	0	-	120	480 GB	-
`roma`	0	-	120	480 GB	-

Profile allocations are designed to:

Hopper: Request full resources per GPU (56 CPUs x 6 GB/CPU = 336 GB per GPU)
Ampere: Request full resources per GPU (28 CPUs × 8 GB/CPU = 224 GB per GPU)
Turing: Request full resources per GPU (4 CPUs × 4 GB/CPU = 16 GB per GPU)
CPU nodes: Request minimal resources (1 CPU × 4 GB = 4 GB) for flexible scheduling

All *_full GPU profiles request exclusive access to the allocated node.

Available Profiles

Profile	Partition	GPU Type	GPU Memory	GPUs	CPUs	Memory	Time	Use Case
`s3df_hopper`	hopper	H200	141 GB	1	56	6 GB/CPU	2h	Highest-performance GPU processing
`s3df_hopper_full`	hopper	H200	141 GB	4	224	6 GB/CPU	2h	Full-node distributed training
`s3df_ampere`	ampere	A100	40 GB	1	28	8 GB/CPU	2h	High-performance GPU processing (default)
`s3df_ampere_full`	ampere	A100	40 GB	4	112	8 GB/CPU	2h	Full-node distributed training
`s3df_turing`	turing	RTX 2080 Ti	11 GB	1	4	4GB/CPU	2h	Cheaper GPU inference
`s3df_milano`	milano	-	-	0	1	4 GB/CPU	2h	CPU-only analysis
`s3df_roma`	roma	-	-	0	1	4 GB/CPU	2h	CPU-only analysis

NERSC Perlmutter Node Resources

NERSC Perlmutter is a heterogeneous system with GPU nodes in two configurations:

Node Type	Count	GPUs/Node	GPU Type	CPUs/Node	RAM/Node	Resources per GPU
GPU (40GB)	1,536	4	A100 (40 GB)	64	512 GB	32 CPUs, 128 GB
GPU (80GB)	256	4	A100 (80 GB)	64	512 GB	32 CPUs, 128 GB
CPU	3,072	0	-	128	512 GB	-

Profile allocations are designed to:

GPU nodes: Request full resources per GPU (32 CPUs × 4 GB/CPU = 128 GB per GPU)
CPU nodes: Request minimal resources (1 CPU × 4 GB = 4 GB) for flexible scheduling
Shared partitions: Allow partial node allocation for cost-efficient small jobs

NERSC Available Profiles

Profile	Partition	GPU Type	GPU Memory	GPUs	CPUs	Memory	Time	Use Case
`nersc_gpu`	gpu_ss11	A100	40 GB	1	32	4 GB/CPU	2h	Standard GPU processing (default, best availability)
`nersc_gpu_80gb`	gpu_ss11	A100	80 GB	1	32	4 GB/CPU	2h	High-memory GPU processing (limited availability)
`nersc_gpu_full`	gpu	A100	40 GB	4	128	Full node	2h	Full-node training
`nersc_gpu_full_80gb`	gpu	A100	80 GB	4	128	Full node	2h	Full-node high-memory training
`nersc_cpu`	shared	-	-	0	1	4 GB/CPU	2h	CPU-only analysis

Note: The nersc_gpu profile uses 40GB A100s by default since there are 6x more nodes available (1,536 vs 256), resulting in significantly faster queue times. Use nersc_gpu_80gb only when you specifically need >40GB GPU memory. The former nersc_gpu_exclusive names remain available as backward-compatible aliases for the corresponding *_full profiles.

ANL Polaris Available Profiles

Profile	Queue	GPUs	CPUs	Time	Use Case
`anl_polaris_capacity`	capacity	1	16	2h	Standard single-GPU processing
`anl_polaris_capacity_full`	capacity	4	64	2h	Full-node distributed training
`anl_polaris_debug`	debug	1	16	1h	Short single-GPU testing
`anl_polaris_debug_full`	debug	4	64	1h	Short full-node testing

Profile Selection

Profiles are auto-detected based on detector and config, or can be specified explicitly:

For batch jobs, spine-prod derives SPINE's world size from the effective GPU allocation after applying profile overrides. For example, --gpus 4 both requests four GPUs and launches four SPINE processes. --world-size is an optional assertion and is rejected when it disagrees with the allocation.

# Auto-detect (default)
./submit.py --config infer/icarus/latest --source data.root

# Explicit profile
./submit.py --config infer/icarus/latest --source data.root --profile s3df_turing

# ANL/Polaris using SPINE_CONTAINER_PATH from configure.sh
./submit.py --config infer/icarus/latest --source data.root --profile anl_polaris_debug

# Override specific resources
./submit.py --config infer/icarus/latest --source data.root --time 2:00:00 --cpus-per-task 8

# Exclude known-bad nodes from a Slurm submission
./submit.py --config infer/icarus/latest --source data.root --exclude sdfampere014

# Override SPINE configuration values at runtime
./submit.py --config infer/generic/latest --source data.root --batch-size 1

# Preload model weights on the submit host before submitting
./submit.py --config infer/2x2/full_chain_240819.yaml --source data.root --profile anl_polaris_debug --preload

# Optional: preload only, useful for external production pipelines
./scripts/preload_downloads.py infer/2x2/full_chain_240819.yaml

Training and Validation Runs

Inference submissions are disposable, independently named productions, but a training is a persistent experiment that may span several scheduler jobs. Use an explicit --run-dir for training and validation so that checkpoints, CSV logs, TensorBoard events, and submission records remain associated.

Start a Training Run

The training configuration owns the model, data loader, optimizer, total epoch count, checkpoint interval, and optional on-the-fly validation. SPINE v1.0.0 uses a top-level train block and an optional sibling validation block; spine-prod owns artifact locations. Inputs may remain in the configuration or be supplied at submission time with --source/--source-list for training and --val-source/--val-source-list for integrated validation.

For a new production, running validation in the training process is the recommended strategy. It evaluates the live model at checkpoint boundaries, keeps training and validation metrics in one log stream, and can drive early stopping, best-checkpoint promotion, and checkpoint-bound learning-rate schedulers. A minimal configuration shape is:

base:
  epochs: 250

train:
  save_step: 10000
  optimizer:
    name: Adam
    lr: 0.001

validation:
  file_keys: /path/to/validation/*.root
  fraction: 1.0
  early_stopping:
    monitor: loss
    mode: min
    patience: 5
  best_checkpoint:
    monitor: loss
    mode: min

./submit.py \
  --config train/generic/uresnet/train_240718.yaml \
  --stage train \
  --run-dir /path/to/experiments/uresnet/default \
  --profile nersc_gpu_exclusive \
  --tensorboard \
  --source-list /path/to/train_file_list.txt \
  --val-source-list /path/to/validation_file_list.txt

A new training run refuses to use a nonempty directory. Its artifact layout is:

experiments/uresnet/default/
├── run_metadata.json
├── train_log-0000000.csv
├── weights/
│   ├── snapshot-9999.ckpt
│   ├── snapshot-9999.ckpt.sha256
│   ├── snapshot-best.ckpt
│   └── snapshot-best.ckpt.sha256
├── tensorboard/
│   ├── train/
│   └── validation/
├── latest -> attempts/TIMESTAMP
├── stdout.log -> latest/stdout.log
├── stderr.log -> latest/stderr.log
└── attempts/
    └── TIMESTAMP/
        ├── inputs.txt
        ├── validation_inputs.txt
        ├── job_metadata.json
        ├── submit.sbatch
        ├── stdout.log
        └── stderr.log

tensorboard/ is created only when TensorBoard is requested. The timestamped attempt preserves an exact submission record, while the stable links make the current scheduler logs available directly from the run root.

Use sibling run directories such as default, augment, and ablate for comparable variants of one experiment.

Resume Training

If training is interrupted, resume the same run explicitly:

./submit.py \
  --config train/generic/uresnet/train_240718.yaml \
  --stage train \
  --run-dir /path/to/experiments/uresnet/default \
  --profile nersc_gpu_exclusive \
  --resume

spine-prod's --resume selects the checkpoint with the largest numeric suffix, passes that exact path together with SPINE's strict --resume flag, reuses weights/snapshot as the checkpoint prefix, and writes a new train_log-<start>.csv segment beside the old log. SPINE v1.0.0 restores all available optimizer, scheduler, epoch, RNG, and loader continuation state; missing state required by strict resume is an error. SPINE's TrainDrawer concatenates the log segments. To roll back intentionally, use --resume-from RUN_DIR/weights/snapshot-N.ckpt. An existing run cannot be reused without one of these resume options.

New SPINE checkpoints include an adjacent SHA-256 sidecar. spine-prod verifies that sidecar before resuming or scheduling validation and rejects a mismatch. Older checkpoints without sidecars remain valid and are used without integrity verification.

The configured epoch count remains the total target, not a number of additional epochs. To extend a run without changing the stored configuration file, pass a runtime override such as --epochs 400 when resuming.

Gracefully Complete Training

SPINE versions with graceful-training control can accept the current training progress without canceling the downstream workflow. The following command asks SPINE to finish its active minibatch, run configured validation, write a complete checkpoint, update the best checkpoint when appropriate, and exit successfully:

./submit.py --graceful-stop 12345678

spine-prod detects Slurm through scancel or PBS through qsig. If both scheduler clients are available on the submit host, select one explicitly:

./submit.py --graceful-stop 12345678 --scheduler slurm
./submit.py --graceful-stop 12345678.server --scheduler pbs

On Slurm, spine-prod signals only the batch shell with SIGUSR1; on PBS it uses qsig. For training submissions, the shell translates that signal into a marker file scoped to the immutable submission attempt. SPINE rank zero polls the marker at minibatch boundaries and shares the request with other ranks. Signals therefore never cross the Singularity, Shifter, or Apptainer boundary or reach auxiliary processes. Once SPINE returns status zero, normal afterok dependencies may proceed.

This is an intentional successful completion, unlike scancel JOB_ID or a SIGTERM caused by timeout or machine failure. Do not use it with a SPINE version that lacks the graceful SIGUSR1 training handler; such a process will terminate unsuccessfully instead of checkpointing. Use --dry-run to inspect the scheduler command without signaling the job.

Run Validation Independently

Attach standalone validation to the training run:

./submit.py \
  --config /path/to/uresnet_validation.yaml \
  --stage validation \
  --run-dir /path/to/experiments/uresnet/default \
  --profile nersc_gpu \
  --tensorboard

spine-prod scans weights/snapshot-*.ckpt, identifies complete associated inference_log-*.csv files, writes only the missing checkpoint paths to the submission's weights.txt, and passes that list to SPINE. Repeating the command therefore validates only new checkpoints; if none are missing it exits without submitting a scheduler job. Empty or malformed validation logs do not count as complete. --rerun-validation deliberately selects every checkpoint again.

The unnamed validation is the run's primary validation series and writes its CSV files beside the training logs, which is the layout expected by TrainDrawer. Additional suites use an isolated name:

./submit.py \
  --config /path/to/uresnet_data_validation.yaml \
  --stage validation \
  --validation-name data \
  --run-dir /path/to/experiments/uresnet/default

Named CSV logs live under validation/<name>/, and their TensorBoard events live under tensorboard/validation/<name>/. Validation identity includes the configuration digest; changing a suite's configuration requires a new name or an explicit --rerun-validation.

With SPINE v1.0.0, the primary validation configured alongside train needs only the training command. Standalone validation remains useful for legacy runs, alternate datasets, and checkpoints produced without integrated validation or after changing the validation policy.

Compare Training Runs

The primary CSV layout can be passed directly to SPINE's TrainDrawer:

from spine.vis.drawer.train import TrainDrawer

drawer = TrainDrawer("/path/to/experiments/uresnet")
drawer.draw(
    model=["default", "augment", "ablate"],
    metric="loss",
    smoothing=100,
)

TensorBoard discovers the corresponding train and validation event streams recursively:

tensorboard --logdir /path/to/experiments/uresnet

Pipeline Mode

Pipelines allow you to chain multiple processing stages with automatic dependency management.

Pipeline Definition

Create a YAML file in pipelines/:

workspace: null

variables:
  raw_source: /path/to/raw/*.root

defaults:
  profile: s3df_ampere
  time: "08:00:00"

stages:
  - name: reconstruction
    config: infer/icarus/latest
    source: ${raw_source}
    run_dir: ${workspace}/reconstruction
    output: ${workspace}/reconstruction/output
    ntasks: 100
    # Replace with a concrete YAML config if you need specific modifiers

  - name: analysis
    depends_on: [reconstruction]  # Wait for reconstruction to complete
    config: path/to/downstream_stage.yaml
    source: ${workspace}/reconstruction/output/*.h5
    run_dir: ${workspace}/analysis
    profile: s3df_milano
    ntasks: 20

workspace defines the pipeline's shared output root and is available as the reserved ${workspace} variable. Declaring workspace: null keeps a checked-in pipeline independent of its launch location and requires an explicit --workspace at submission. A concrete YAML value remains a valid default and can also be overridden from the CLI. Pipelines that do not use a shared workspace may omit the field entirely.

Additional string values under variables can reference the workspace or one another. Expansion applies recursively to stage fields, including structured sources and module weights. Undefined variables, dependency cycles, missing required workspaces, and invalid declarations fail before any job is submitted; environment variables are not expanded implicitly.

Repeated stage variants can be declared as a flat string collection and expanded before validation:

collections:
  splits:
    - name: train
      source: /data/train.root
    - name: validation
      source: /data/test.root

stages:
  - name: cache_${split.name}
    for_each:
      collection: splits
      as: split
    config: cache/example.yaml
    source: ${split.source}

This produces the concrete cache_train and cache_validation stages. It is load-time shorthand only: dependencies still name concrete stages explicitly, and each expanded stage remains an independent scheduler job.

Submit Pipeline

./submit.py \
  --pipeline pipelines/my_pipeline.yaml \
  --workspace /path/to/production

Pipeline-wide CLI overrides take precedence over individual stage values, which in turn take precedence over defaults. This is useful for launch-specific software and scheduler settings:

./submit.py \
  --pipeline pipelines/my_pipeline.yaml \
  --workspace /path/to/production \
  --spine-path /path/to/spine \
  --profile s3df_hopper \
  --account my_account

Software paths, profiles, scheduler resources, and first-class SPINE runtime options can be overridden pipeline-wide. Inputs, outputs, run directories, dependencies, model weights, and lifecycle options remain stage-specific and must be configured in YAML. Unsupported pipeline CLI options and unknown YAML fields fail before any jobs are submitted. --spine is an explicit alias for --spine-path.

Restart a Pipeline

After canceling or confirming termination of a failed pipeline submission, restart it at the first failed stage while retaining its workspace:

./submit.py \
  --pipeline pipelines/my_pipeline.yaml \
  --workspace /path/to/production \
  --from-stage cache_train_fragment_graphs

Stages earlier in pipeline order are treated as completed. Dependencies among the selected stages are rebuilt with the new scheduler IDs, while dependencies on skipped stages are considered satisfied by their existing artifacts. Use --to-stage NAME to stop at an inclusive boundary when only a bounded range should be regenerated.

Use --skip-stage NAME [NAME ...] to reuse completed stages within that range:

./submit.py \
  --pipeline pipelines/sbnd/full_chain_261029.yaml \
  --workspace /path/to/production \
  --from-stage train_grappa_shower \
  --skip-stage train_grappa_track

Skipped stages reuse their existing artifacts; they are not submitted again. Dependencies are contracted through omitted stages, so particle caching still waits for shower training in this example. --skip-stage can also be used alone or with --to-stage, but cannot be combined with --select-stage. Unknown stage names and selections that exclude every stage are rejected. Skipping does not verify that the existing artifacts are complete or compatible.

For a sparse repair, select multiple stages in one option:

./submit.py \
  --pipeline pipelines/my_pipeline.yaml \
  --workspace /path/to/production \
  --select-stage cache_train_segmentation cache_validation_segmentation \
                 cache_train_particle_graphs cache_validation_particle_graphs

Selected stages run in pipeline order, regardless of their CLI order. Direct dependencies are contracted through omitted stages, so selected downstream stages still wait for the nearest selected ancestors. --select-stage cannot be combined with --from-stage or --to-stage.

On Slurm, dependent stages are submitted with --kill-on-invalid-dep=yes, so a stage is canceled automatically when an upstream afterok dependency can no longer succeed. PBS Professional provides the corresponding deletion behavior as part of its native dependency semantics and requires no additional directive.

Every submission, including the first, receives a new attempts/TIMESTAMP directory, preserving earlier scripts, logs, and metadata. Each stage root has a latest link to its current attempt. A retried training stage resumes its latest valid checkpoint automatically. If its previous scheduler job never started, the unchanged empty training run is reused. A run containing training logs but no checkpoint is rejected because there is no unambiguous state from which to resume. Do not use --from-stage while jobs from the previous submission are still active.

Pipeline stages accept the CLI-equivalent source, source_list, val_source, and val_source_list fields. Composite datasets use structured sources and validation_sources, while module_weight forwards named model checkpoints through SPINE's native CLI. A model-only export_weights stage composes those checkpoints into one inference artifact without initializing data I/O. An inference stage with in_place: true passes no --output, --output-dir, or --output-suffix override, leaving writer routing entirely to SPINE. This is the intended mode for extending a sharded cache repository; it cannot be combined with explicit writer output fields. See pipelines/generic/full_chain_240805.yaml for a complete staged-training workflow with centralized paths. Its materialization jobs publish successive immutable stages to one logical cache repository per data split, then compose the independently trained modules into one full-chain checkpoint. The ND-LAr and ProtoDUNE-SP pipelines apply the same production pattern with their detector-specific model and data contracts. The generic full-chain pipelines finally evaluate that assembled checkpoint with SPINE's metric analyzers and submit a CPU-only kind: report reduction. The report stage waits for all metric inference jobs, records dataset, selection, and checkpoint provenance, and writes plots plus summary.json under metrics/full_chain/report/artifacts.

Run Management

Job Artifacts

Inference creates a timestamped run directory automatically. Every submission uses the same attempt layout. The common single-task case is deliberately flat:

runs/20260810_143022_spine_icarus_latest/
├── latest -> attempts/TIMESTAMP
├── stdout.log -> latest/stdout.log
├── stderr.log -> latest/stderr.log
└── attempts/
    └── TIMESTAMP/
        ├── inputs.txt
        ├── job_metadata.json
        ├── submit.sbatch
        ├── stdout.log
        ├── stderr.log
        ├── inference_log-*.csv
        ├── outputs.txt
        └── output/

output/ is created only when spine-prod supplies the default writer output; it is omitted when --output selects an external destination or --in-place leaves the writer destination config-defined. There are no empty scheduler, task, or log directories. For a managed default output, outputs.txt is initialized at submission and populated with absolute paths after each successful task. It is also available through latest/outputs.txt and its location is recorded as output_manifest in job_metadata.json.

Only a real scheduler array creates task directories. Scheduler chunking is represented by numbered submit scripts instead of another directory layer:

attempts/TIMESTAMP/
├── job_metadata.json
├── outputs.txt
├── submit_000.sbatch
├── submit_001.sbatch
├── JOB_*.out                   # Scheduler array logs
├── JOB_*.err
└── tasks/
    ├── 000_1/
    │   ├── inputs.txt
    │   ├── inference_log-*.csv
    │   ├── outputs.txt
    │   └── output/
    └── 000_2/
        └── ...

Use an explicit --output only when a deliberately shared or externally managed output location is required. An optional --run-dir can select a specific new inference directory; it must be empty. Existing run trees are not migrated or modified; the new layout applies to new attempts.

Pipeline workspaces also expose logs/<stage>, a symlink to each stage's latest attempt. Thus the current scheduler log for a scalar stage is available as WORKSPACE/logs/STAGE/stdout.log or stderr.log without traversing the stage's durable artifact hierarchy.

Monitoring Jobs

# View job status on SLURM
squeue -u $USER

# View job details on SLURM
scontrol show job <job_id>

# View job status on PBS
qstat -u $USER

# View job details on PBS
qstat -fx <job_id>

# View the latest scalar-job log
tail -f runs/<run_dir>/stdout.log

# View one pipeline stage's latest log
tail -f <workspace>/logs/<stage>/stdout.log

# Cancel job on SLURM
scancel <job_id>

# Cancel job on PBS
qdel <job_id>

Job Metadata

Each job saves complete metadata for reproducibility:

{
  "job_name": "spine_icarus_latest",
  "detector": "icarus",
  "config": "infer/icarus/latest",
  "profile": "s3df_ampere",
  "num_files": 100,
  "job_ids": ["12345", "12346"],
  "submitted": "2026-01-01T14:30:22",
  "command": "./submit.py --config ..."
}

Automatic Cleanup of Intermediate Files

Pipelines can automatically clean up intermediate outputs once downstream stages complete. Add a cleanup field to any stage that produces temporary files:

stages:
  - name: reconstruction
    config: infer/icarus/latest
    files: /path/to/input/*.root
    output: output_reco
    # Clean up output_reco/ after all dependent stages finish
    cleanup:
      - output_reco
      - temp_files
  
  - name: analysis
    depends_on: [reconstruction]
    config: path/to/downstream_stage.yaml
    files: output_reco/*.h5
    output: output_analysis

The cleanup job:

Only runs if downstream stages complete successfully (afterok dependency)
Runs as a minimal resource job (1 CPU, 1GB RAM, 10min timeout)
Safely checks for path existence before removal
Logs all cleanup actions for auditing

This is especially useful for large-scale production to save disk space by removing intermediate reconstruction outputs after final analysis completes.

Advanced Features

Custom Software Paths

# Use custom LArCV installation
./submit.py --config infer/icarus/latest --source data.root --larcv-path /path/to/larcv

# Use custom flash-matching setup
./submit.py --config infer/icarus/latest --source data.root --flashmatch-path /path/to/flashmatch

# Expose CVMFS inside the container
./submit.py --config infer/icarus/latest --source data.root --cvmfs

--expandable-segments exports PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before starting SPINE. It is an explicit opt-in for batch, interactive, and pipeline submissions. Maintained pipelines leave this option disabled by default.

There is no need to pass --flashmatch. The flag is accepted only for backward compatibility and is ignored. Use --flashmatch-path to source a custom flash-matching setup instead.

For sites without CVMFS, point detector configs at local copies of their data release directories before sourcing configure.sh:

export ICARUS_DATA_DIR=/path/to/icarus_data
export SBND_DATA_DIR=/path/to/sbnd_data
source configure.sh

Job Dependencies

# Submit with dependency on another job
./submit.py --config path/to/downstream_stage.yaml --source output/*.h5 --dependency afterok:12345

Array Job Optimization

# Process 5 files per job (reduces overhead)
./submit.py --config infer/icarus/latest --source data/*.root --files-per-task 5

# Limit concurrent tasks to 50
./submit.py --config infer/icarus/latest --source data/*.root --ntasks 50

Detector-Specific Guides

ICARUS

ICARUS uses split cryostat processing with cosmic overlay:

# Standard cosmic overlay processing
./submit.py --config infer/icarus/latest --source data.root

# Data-only mode (no truth labels)
./submit.py --config infer/icarus/latest --apply-mods data --source data.root

# NuMI beam configuration
./submit.py --config infer/icarus/latest --apply-mods numi --source data.root

# Lite output (reduced file size)
./submit.py --config infer/icarus/latest --apply-mods data lite --source data.root

SBND

./submit.py --config infer/sbnd/latest --source data.root

2x2

2x2 uses higher resource requirements:

./submit.py --config infer/2x2/latest --source data.root --profile s3df_ampere

ND-LAr

./submit.py --config infer/nd-lar/latest --source data.root

Troubleshooting

Environment Not Set

WARNING: SPINE_PROD_BASEDIR not set. Did you source configure.sh?

Solution: Source the environment:

source configure.sh

Missing Dependencies

ERROR: jinja2 is required. Install with: pip install jinja2

Solution: Install Python dependencies:

pip install jinja2 pyyaml

Job Failures

Check the latest batch log at runs/<run_dir>/stdout.log and runs/<run_dir>/stderr.log (or inspect latest/ for an array)
Review run metadata in runs/<run_dir>/job_metadata.json
Test configuration on a single file with --dry-run
Verify input files exist and are accessible

Out of Memory

Solution: Use a profile with more memory or override memory:

./submit.py --config infer/icarus/latest --source data.root --profile s3df_ampere

Or override memory:

./submit.py --config infer/icarus/latest --source data.root --mem-per-cpu 16g

Job Time Limit

Solution: Request more time:

./submit.py --config infer/icarus/latest --source data.root --time 4:00:00

Best Practices

1. Test Before Production

Always test configurations on a small sample:

# Test with dry run
./submit.py --config infer/icarus/latest --source test.root --dry-run

# Test with single file
./submit.py --config infer/icarus/latest --source test.root

2. Use Appropriate Profiles

Use s3df_ampere for high-performance GPU processing (default)
Use s3df_turing for cheaper GPU inference
Use s3df_milano or s3df_roma for CPU-only analysis

3. Optimize File Batching

# For many small files, batch them
./submit.py --config infer/icarus/latest --source small_files/*.root --files-per-task 10

# For large files, process individually
./submit.py --config infer/icarus/latest --source large_files/*.root --files-per-task 1

4. Monitor Resource Usage

Check actual resource usage to optimize future jobs. On SLURM systems:

seff <job_id>

5. Track Your Work

Job metadata is automatically saved. Keep important job directories:

# Inference runs are in timestamped directories
ls -lt runs/

Environment Variables

Set by configure.sh:

SPINE_PROD_BASEDIR - Base directory of this repository
SPINE_PROD_CONFIGURED - Marker that the shell environment was configured
SPINE_CONFIG_PATH - Configuration search path
ICARUS_DATA_DIR - ICARUS data release path; defaults to the standard ICARUS CVMFS location
SBND_DATA_DIR - SBND data release path; defaults to /cvmfs/sbnd.opensciencegrid.org/products/sbnd/sbnd_data/
SPINE_CONTAINER_VERSION - Tagged SPINE container version
SPINE_CONTAINER_PATH - Singularity/Apptainer image path
SPINE_CONTAINER_TAG - Registry image tag for Shifter-style runtimes, including docker:
SPINE_CONTAINER_VERSION_AUTO - Tracks whether the version follows the repository default
SPINE_CONTAINER_PATH_AUTO - Tracks whether SPINE_CONTAINER_PATH was auto-derived
SPINE_CONTAINER_TAG_AUTO - Tracks whether the registry tag was auto-derived
SPINE_CONTAINER_RUNTIME_BIN - Optional full path or command name for the Singularity/Apptainer executable used by interactive SIF execution
SPINE_CONTAINER_RUNTIME_ARGS - Optional extra Singularity/Apptainer arguments for interactive SIF execution
SPINE_CONTAINER_PLATFORM - Docker/Podman platform for interactive fallback

Contributing

Adding New Profiles

Edit templates/profiles.yaml:

profiles:
  my_custom_profile:
    partition: my_partition
    gpus: 2
    cpus_per_task: 16
    mem_per_cpu: 8g
    time: "6:00:00"
    description: "My custom profile description"

Adding Detector Defaults

Edit templates/profiles.yaml:

detectors:
  my_detector:
    default_profile: s3df_ampere
    configs_dir: infer/my_detector
    account: "my:account"

Development

Installation for Development

For contributors who need to run tests and development tools:

# Create a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate  # On Linux/Mac

# Install SPINE (required for running parsing tests)
pip install "spine-ml @ git+https://github.com/DeepLearnPhysics/SPINE.git@main"

# Install development dependencies
pip install -r requirements-dev.txt

# Install pre-commit hooks
pre-commit install

Note: SPINE installation is only required for development if you want to run the configuration parsing tests. Production users only need to point to an existing SPINE installation via configure.sh.

Running Tests

# Run all tests
pytest

# Run config validation tests only
pytest tests/test_config_validation.py

# Run with coverage
pytest --cov=. --cov-report=html

Pre-commit Hooks

This repository uses pre-commit hooks for code quality:

check-yaml: Validates YAML syntax
yamllint: Lints YAML files for style
prettier: Formats YAML files
trailing-whitespace: Removes trailing whitespace
end-of-file-fixer: Ensures files end with newline

# Run hooks manually
pre-commit run --all-files

Config Validation

All configuration files are automatically validated in CI to ensure they parse correctly:

from spine.config import load_config_file

config = load_config_file("config/infer/icarus/full_chain_co_260501.yaml")

Related Tools

Production Database (spine-db)

A companion tool for indexing and browsing SPINE production metadata. See spine-db for documentation.

Support

Issues: https://github.com/DeepLearnPhysics/spine-prod/issues
SPINE Documentation: https://github.com/DeepLearnPhysics/spine
Contact: SPINE development team

License

This software is provided under the same license as SPINE.

Citation

If you use SPINE in your research, please cite the relevant SPINE publications.

Acknowledgments

Development supported by the DOE Office of High Energy Physics and the National Science Foundation.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
config		config
pipelines		pipelines
scripts		scripts
src		src
templates		templates
tests		tests
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
.yamllint.yaml		.yamllint.yaml
CHANGELOG.md		CHANGELOG.md
DEFAULT_SPINE_VERSION		DEFAULT_SPINE_VERSION
MIGRATION_1_4.md		MIGRATION_1_4.md
QUICKREF.md		QUICKREF.md
README.md		README.md
codecov.yml		codecov.yml
configure.sh		configure.sh
pyproject.toml		pyproject.toml
requirements-dev.txt		requirements-dev.txt
requirements-joint.txt		requirements-joint.txt
requirements.txt		requirements.txt
submit.py		submit.py
version.py		version.py

Latest commit

History

Folders and files

Repository files navigation

SPINE Production System

Overview

Quick Start

1. Environment Setup

2. Basic Job Submission

3. Advanced Usage

Interactive Mode

EAF Interactive Container Setup

Directory Structure

Configuration System

Config Organization

Example: ICARUS Configurations

Resource Profiles

S3DF Node Resources

Available Profiles

NERSC Perlmutter Node Resources

NERSC Available Profiles

ANL Polaris Available Profiles

Profile Selection

Training and Validation Runs

Start a Training Run

Resume Training

Gracefully Complete Training

Run Validation Independently

Compare Training Runs

Pipeline Mode

Pipeline Definition

Submit Pipeline

Restart a Pipeline

Run Management

Job Artifacts

Monitoring Jobs

Job Metadata

Automatic Cleanup of Intermediate Files

Advanced Features

Custom Software Paths

Job Dependencies

Array Job Optimization

Detector-Specific Guides

ICARUS

SBND

2x2

ND-LAr

Troubleshooting

Environment Not Set

Missing Dependencies

Job Failures

Out of Memory

Job Time Limit

Best Practices

1. Test Before Production

2. Use Appropriate Profiles

3. Optimize File Batching

4. Monitor Resource Usage

5. Track Your Work

Environment Variables

Contributing

Adding New Profiles

Adding Detector Defaults

Development

Installation for Development

Running Tests

Pre-commit Hooks

Config Validation

Related Tools

Production Database (spine-db)

Support

License

Citation

Acknowledgments

About

Topics

Resources

Stars

Watchers

Forks