Skip to content

development.md

Latest commit

 

History

History
506 lines (383 loc) · 17.3 KB

development.md

File metadata and controls

506 lines (383 loc) · 17.3 KB
Raw

Development Guide

This guide describes the current source-development workflow for TICO. Repository commands are routed through the ./ccex helper in the project root.

Contents

Prerequisites

  • Linux development environment
  • Python 3.10 or newer
  • git
  • A Python virtual environment is strongly recommended

Neither conversion nor the test suite requires ONE (one-compiler) or onert. Converted Circle models are executed by the built-in reference runtime (tico.circle.runtime). ONE's circle-interpreter and the onert package are optional compatibility runtimes that are used only when selected explicitly.

Create a development environment

git clone https://github.com/Samsung/TICO.git
cd TICO

python3 -m venv .venv
source .venv/bin/activate

# Install a supported Torch build and TICO in editable mode.
./ccex install

# Install formatter/static-check dependencies and test-only dependencies.
./ccex configure

The order matters. ./ccex configure test validates an already installed Torch package and installs the matching TorchVision build; it does not install TICO or Torch. Run ./ccex install first.

Set up only one part of the environment when needed:

./ccex configure format
./ccex configure test

Torch selection

The source installer accepts a stable family, an exact version, a qualification candidate, a repository-pinned nightly, or the latest published nightly pair:

./ccex install --torch_ver 2.13
./ccex install --torch_ver 2.7    # legacy best-effort
./ccex install --torch_ver 2.11
./ccex install --torch_ver 2.13.0+cu132
./ccex install --torch_ver 2.14
./ccex install --torch_ver nightly          # repository-pinned Torch/TorchVision
./ccex install --torch_ver nightly-latest   # latest published nightly pair

TICO keeps Torch 2.5 through 2.10 as legacy best-effort source-install choices, qualifies 2.11, 2.12, and 2.13, and uses 2.13 as the default. Torch 2.14 is installable as a qualification candidate but is not part of the release-support window. Family requests resolve to a project-pinned patch rather than allowing pip to select an arbitrary patch release. The package metadata itself keeps a bare torch dependency, so a normal pip install does not reject a user-managed older version. nightly uses the versions pinned under infra/dependency/; nightly-latest resolves Torch and TorchVision together from one moving nightly index.

Compute-platform options:

# Force a CPU wheel.
./ccex install --cpu_only

# Override detected host CUDA capability when selecting a compatible wheel.
./ccex install --cuda_ver 12.8

--cpu_only and --cuda_ver are mutually exclusive.

When no Torch version is explicitly requested and a configured stable Torch package is already installed, ./ccex install preserves that installation if its compute platform is compatible. This includes legacy best-effort families. ./ccex configure test then installs the matching TorchVision package and verifies the final package pair with pip check. Nightly selectors are deliberately re-resolved: nightly restores the repository pin, while nightly-latest upgrades the Torch/TorchVision pair together before test configuration validates it.

The source of truth for families, exact patches, CUDA wheel variants, and CI matrices is tico/utils/compat/torch_version_policy.py. See the PyTorch Version Policy for qualification, promotion, and release-branch rules.

Build and install a wheel

Build the distribution artifacts:

./ccex build

Install from dist/ instead of editable source:

./ccex install --dist

A clean wheel workflow similar to CI is:

./ccex build
./ccex install --dist --torch_ver 2.13
./ccex configure test --torch_ver 2.13
pt2-to-circle -h

Run tests

Default suite

./ccex test

With no filter, ccex runs unittest discovery under test/. Large model tests are selected separately with -m and are not part of the normal model-independent suite. ./ccex test supports repository paths containing spaces.

Keyword filtering

./ccex test -k add
./ccex test -k ConvertMatmulToLinear
./ccex test -k quantization

Quote keyword patterns containing spaces to pass them to unittest as one argument.

The shorthand keywords op and net are expanded to the corresponding generated module-test namespaces:

./ccex test -k op
./ccex test -k net

Use the narrowest relevant test first, then expand to the owning subsystem and finally to the complete suite for cross-cutting changes.

Other test-runner options

./ccex test --all                 # Explicit full suite
./ccex test -i                    # Include internal-only tests
./ccex test -v                    # Set TICO_LOG=4 for this run
./ccex test -r reference          # Built-in reference runtime (default)
./ccex test -r onert              # Optional: execute with the onert package
./ccex test -p                    # Run performance benchmarks

--all and --keyword cannot be combined. --model also cannot be combined with --all or --keyword.

Performance benchmarks

./ccex test -p runs the configured conversion-time and serialized-size threshold benchmark:

./ccex test -p

The full Circle O1 scheduler comparison needs a caller-provided .circle artifact and is invoked directly:

python3 -m test.performance.benchmark_circle_optimizer \
  model.circle \
  --repeat 3

It compares the legacy RESTART scheduler with O1's round-based UNTIL_NO_CHANGE scheduler, verifies both outputs, and requires byte-identical Circle binaries. It is an opt-in diagnostic benchmark rather than a repository threshold. See test/performance/README.md for all options.

Test organization

test/
├── modules/                 # Small PyTorch modules used by generated E2E tests
│   ├── op/
│   ├── net/
│   └── model/               # Opt-in dependency-isolated model tests
├── unit_test/               # Focused tests for core conversion and Circle utilities
│   ├── circle/
│   ├── ops/
│   ├── passes/
│   ├── quantization/
│   ├── serialize/
│   └── utils/
├── quantization/            # Algorithms, WrapQ, recipes, configs, export, and analysis
├── support/                 # Shared runtime and test-builder utilities
├── performance/             # Conversion/size and Circle O1 scheduler benchmarks
└── pt2_to_circle_test/      # CLI/API and end-to-end conversion tests

The module test harness normally:

  1. Creates the PyTorch reference outputs before export.
  2. Exports directly or saves and reloads a .pt2 file.
  3. Converts the ExportedProgram to Circle.
  4. Validates the serialized model: CircleDocument.verify() for structural consistency, reference-runtime preparation for operator support, and a zero-input contract probe for static models.
  5. Executes with the reference runtime (or an explicitly selected external runtime), unless inference is disabled by a test tag.
  6. Compares output count, shape, dtype, and values with explicit tolerances.

See System Test Guide for the complete test strategy.

Runtime selection

The default runtime for module parity tests, CircleModel.__call__(), and tico.interpreter.infer() is the built-in reference runtime. It executes the serialized Circle bytes with NumPy/CPU PyTorch kernels, supports dynamic shapes, and needs no external package:

./ccex test -k add
./ccex test -r reference -k add

Two optional compatibility runtimes can be selected from the command line or the environment. They are never chosen implicitly, even when installed:

# ONE luci-interpreter through /usr/share/one/lib/libcircle_interpreter_cffi.so
pip install cffi              # plus a ONE release providing the shared library
./ccex test -r circle-interpreter -k add

# onert Python package (pre-release wheels)
pip install --pre onert==0.2.0.dev250922
./ccex test -r onert -k add
CCEX_RUNTIME=onert ./ccex test -k add

circle-interpreter cannot execute dynamic-shape models; the harness runs a test tagged use_onert with onert when that legacy runtime is selected. A negative test whose expected error is runtime-specific (@test_negative(..., runtime="onert")) runs as a normal parity test under the other runtimes.

See Circle artifact tools for the supported operator set, quantized execution modes, and debugging options.

Model tests

Model tests are selected by directory name or shell-style pattern:

pip install -r test/modules/model/<model_name>/requirements.txt
./ccex test -m <model_name>

# Quote wildcard patterns so the shell does not expand them.
./ccex test -m "Llama*"

The -m option sets CCEX_TEST_MODEL and runs the model-test loader. Model tests may have additional dependencies and can be significantly more expensive than synthetic unit or module tests. Do not use a full model test when a small deterministic graph can cover the behavior.

Debug conversion

Log levels

Set TICO_LOG before the Python process imports TICO:

TICO_LOG=4 ./ccex test -k add
TICO_LOG=4 python examples/my_conversion.py
Value Level
1 fatal
2 warning
3 info
4 debug

Debug mode includes instrumented graph and constant-size diffs around passes and conversion phases.

Intermediate graph images

TICO_GRAPH_DUMP=1 ./ccex test -k add

The main conversion pipeline writes these stage snapshots when applicable:

.tico_tmp/session_<timestamp>/1_after_decompose.png
.tico_tmp/session_<timestamp>/2_after_legalize.png
.tico_tmp/session_<timestamp>/3_after_quantfold.png

Graph rendering uses pydot and Graphviz. The output directory is session-scoped; do not commit generated images.

Circle artifact inspection

tico-circle inspect model.circle --tensors --operators
tico-circle verify model.circle
tico-circle extract model.circle --ops 20-64 -o region.circle
tico-circle optimize model.circle --preset o1 -o model.o1.circle

Use tico-circle verify for static Circle consistency. Use an end-to-end runtime test for numerical parity and backend execution. CircleReferenceRuntime(...).run(inputs, trace=True) returns every intermediate tensor value for debugging.

Formatting and static checks

Before opening a pull request, apply formatting and run all static checks with one command:

./ccex format

This runs UFMT (Black formatting and import sorting), Pylint, and mypy together. To check without applying formatter patches:

./ccex format --no-apply-patches

Use Python 3.12 and the development environment described above. Pylint and mypy need the project and test dependencies: missing PyTorch or different installed library versions can change inferred types without reporting an import error.

To reproduce CI in a fresh Python 3.12 virtual environment, install the policy-selected CPU Torch build and configure the same dependencies:

TORCH_FAMILY=$(python3 tico/utils/compat/torch_version_policy.py default-family)
./ccex install --torch_ver "$TORCH_FAMILY" --cpu_only
./ccex configure test --torch_ver "$TORCH_FAMILY" --cpu_only
./ccex configure format
python3 -m pip check
python3 -m pip list --format=freeze
./ccex format --no-apply-patches

Use a separate environment when developing against another Torch version or optional model dependencies. The static-analysis CI job records installed package versions to help compare environments.

To check tracked files changed from the local main branch, use --diff-only. Add new files to Git before checking them:

./ccex format --diff-only --no-apply-patches

CI runs formatting and static analysis as separate jobs so formatting can finish without installing PyTorch. Both jobs use the same lintrunner configuration as ./ccex format; developers do not need to select individual tools.

The formatter's transitive dependencies are pinned in requirements-lintrunner.txt, including stdlibs and libcst. Keep those pins when updating formatter packages: their versions can change import classification and formatting even when Black, UFMT, and usort are unchanged.

Coverage

Run the discovery suite under coverage and print a terminal report:

./ccex coverage

Write a report under test/reports/cov/:

./ccex coverage -f txt
./ccex coverage -f xml

If coverage is not installed, the current helper requests version 7.6.1. Failed tests return a nonzero exit status while still generating the requested report; report failures also return nonzero when the tests pass.

Pull-request CI

The PR workflow targets main and rel/*. PyTorch versions are resolved from the central policy module rather than duplicated in workflow YAML.

  1. Commit-message check

    • Runs when the pull request is ready for review.

    • Requires at least one commit-body line in the form:

      TICO-DCO-1.0-Signed-off-by: <NAME> <<EMAIL>>

  2. Format check

    • Runs on Ubuntu 24.04 with Python 3.12.
    • Executes ./ccex configure format, then runs lintrunner for UFMT only without applying patches.
    • Does not depend on the package build or install PyTorch.

  3. Package build

    • Builds the TICO wheel once.
    • Uploads one short-lived artifact reused by static analysis and versioned tests.

  4. Static analysis

    • Runs on Ubuntu 24.04 with Python 3.12.
    • Reuses the test-environment setup action with the default Torch family from the central policy, including CPU Torch, matching TorchVision, TICO, and test dependencies.
    • Configures the analysis tools, checks package dependency consistency, records installed versions, and runs lintrunner for Pylint and mypy without applying patches.
    • The existing Check style status succeeds only when both formatting and static analysis succeed, preserving existing required-check settings.

  5. Versioned tests

    • Runs the complete suite on the default qualified family, currently 2.13.
    • Runs blocking export and quantization smoke tests on the oldest supported family, currently 2.11.
    • Runs the same smoke tests non-blockingly on the qualification candidate, currently 2.14.

A separate compatibility workflow runs nightly-latest smoke tests daily and the complete supported/candidate/nightly-latest matrix weekly. Official package publication runs the full suite on every qualified stable family before publishing. See the PyTorch Version Policy for the exact tiers.

The conversion/size threshold benchmark is available through ./ccex test -p. The full Circle O1 scheduler benchmark is invoked directly with a caller-provided artifact. Neither benchmark is part of the current PR matrix.

Change-specific guidance

Add or change a PyTorch-IR pass

  • Implement PassBase.call() and return PassResult(modified=...) accurately.
  • Preserve ExportedProgram graph-signature mappings while replacing nodes.
  • Update metadata for newly created or rewritten tensors.
  • Add focused coverage under test/unit_test/passes/.
  • Include both matching and structurally similar non-matching cases.
  • Add an end-to-end module test when serialization or runtime behavior changes.

Add a serialized operator

  • Add or update a NodeVisitor under tico/serialize/operators/.
  • Register every supported ATen overload with register_node_visitor.
  • Add serializer/operator unit coverage and a module conversion/parity test.
  • Do not leave an unsupported operator in the final graph and rely on a runtime to reject it; TICO validates supported targets before serialization.

Change quantization

  • Preserve the prepare -> calibration/statistics -> convert lifecycle.
  • Put generic algorithms and infrastructure in their owning quantization packages.
  • Put model-family behavior in recipe adapters rather than generic stages.
  • Prefer deterministic synthetic tests before full model evaluation.
  • Update the relevant quantization README or configuration reference when behavior is user visible.

Change Circle artifact tools

  • Keep PyTorch-IR passes in tico/passes/ and serialized Circle-to-Circle passes in tico/circle/passes/.
  • Verify index remapping, graph boundaries, buffers, signatures, multi-subgraph behavior, and cleanup contracts as applicable.
  • Follow the dedicated pass guidance in Circle artifact tools.

See also