Skip to content

Add native BF16 VGGT reconstruction - #1635

Open
for-the-juan wants to merge 3 commits into
ModelTC:mainfrom
for-the-juan:feat/vggt-bf16
Open

for-the-juan wants to merge 3 commits into
ModelTC:mainfrom
for-the-juan:feat/vggt-bf16

Conversation

@for-the-juan

@for-the-juan for-the-juan commented Oct 20, 2026 •

Copy link
Copy Markdown

Summary

Split from #1628 at reviewer request. Add native BF16 VGGT-1B inference through LightX2V's runner registry (recon). This PR targets main independently and requires no other model PR.

  • Native ViT-L/14, alternating aggregation, camera/depth/point-map heads, strict checkpoint loading and CPU geometry outputs.
  • Ordered CPU image preprocessing with eight workers; the one-worker reference uses the same pixel transform and input order.
  • Existing registered MM/attention backends, with no sparse-operator or other reconstruction-model dependency.
  • Model-specific preparation tool, pinned public revisions, byte-checked input manifest, dependency pins and reproduction guide at scripts/bench/vggt/README.md.

Model graphs, algorithms and operator selection are separate. Platform setup remains in existing lightx2v_platform; new model code has no vendor imports or branches. Shared dense adapters and measurement code are identical commits in the two split PRs, so they remain one implementation after merging. The other PR is #1636. Central runner/config registration entries may need rebasing after either PR merges; local composition was checked with both models present.

Validation

  • 8 model/shared numerical tests passed without exposing any accelerator; all 27 tests passed after composing the two branches locally.
  • pre-commit run --all-files passed.
  • Independent wheel build passed, with this model's license included and the other reconstruction model absent.
  • Post-split single-card MLU regression: two fixed requests / four A/B runs, zero failures, exact agreement of every returned tensor, and BF16 operand audits passed. Networking was disabled; the only source mount was this independent checkout. Dependency images were reused from the previously validated vendor-runtime builds.

These short runs validate the split; they are not a new complete performance measurement. The original complete frozen-cohort measurements are documented with their source revision (c20b99d) in the guide. The shared event/wall boundaries, alternating A/B order, seeds, input hashes and reductions are preserved; rerun the documented full suite for this checkout's timings.

Reproduction and scope

Run from the repository root:

python -m scripts.bench.vggt.prepare_assets weights --output weights/vggt
python -m scripts.bench.vggt.benchmark \
  --model-path weights/vggt --input-root benchmark_inputs \
  --output benchmark_outputs/vggt --audit-bf16

Prepare the frozen inputs first using the guide. Expose one idle device per process. Matrix weights, large matrix/conv operands and attention Q/K/V are BF16; normalization, phase/coordinate arithmetic, accumulation and geometry may be FP32. Coordinates remain integers.

The formal cohort is 36 requests with original ETH3D JPEGs (32/32/26 views, three scenes × 12 repetitions), excluding the first pair for 35 observations. Events time one forward; synchronized wall time covers image reading through CPU predictions. Tracking is outside this integration.

Weights, input data and the vendor runtime/driver remain external prerequisites; no private experiment checkout or patch is needed. CUDA/A100 and MetaX performance, offload, quantization and distributed inference have not been validated. Full-dataset GT metrics have not been rerun after splitting. Upstream CI status should be checked separately from local validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant