[hujc7]

Summary

Windows CI on the [self-hosted, gpu-windows] runners: Part 3 of the cross-platform CI series (Part 1 #5695 and Part 2 #5698 are merged). Two new jobs, each started by a Detect Changes job of under 10s on ubuntu-latest. They run in parallel on separate runners, and each test runs once per install method: Windows CI tests the locked uv workspace, Installation Tests (Windows) the wheel. Steps are listed in the order they run and add up to the job's total.

Windows CI / windows-ci, the counterpart of ARM CI / Build & Test:

Step	What runs	Reused runner	Fresh runner
Set up	checkout, uv	7s	19s
Sync	uv sync --frozen --extra test --extra isaacsim --extra ov, into an environment the runner keeps	0s	5m 40s
Restore caches	Warp kernel, Kit shader and OVRTX shader caches, then a check that Warp uses its cache directory	3s	11s
Marker tests	CI_MARKER=windows_ci pytest tools: 4 isaaclab/test/utils files, 62 tests	23s	29s
Timeout test	the orchestrator's Windows timeout case, 1 test	re-measuring	6s
Cartpole smoke	install_ci/misc/cartpole_training_smoke.py --run-gpu, 12 tests: state training and the camera render check on Newton, and camera training on the 10 physics × renderer pairings that can share a process	re-measuring	≈13m 06s
Wrap up	cache sizes, test reports	6s	6s
Total		re-measuring	≈19m 57s

Installation Tests (Windows), beside x86 and ARM. tools/run_install_ci.py native runs the suite they share, minus test_workspace.py, whose uv sync install and training Windows CI already runs:

Step	What runs	Reused runner	Fresh runner
Set up	checkout, uv	8s	17s
Build wheel	tools/wheel_builder/build.sh under Git's bash	25s	32s
Start pytest	collection	1s	3s
Lock resolution	uv lock --upgrade --dry-run: the dependencies resolve without the committed lock	3s	24s
Wheel install, base	uv pip install of the built wheel into a new environment, then torch from the cu130 index	42s	1m 52s
Wheel training, base	Cartpole on Newton	20s	1m 48s
Wheel install, isaacsim	the same with [isaacsim], from pypi.nvidia.com	1m 41s	5m 50s
Wrap up	JUnit report	6s	6s
Total		≈3m 26s	10m 52s

Fresh: new runners with nothing installed or cached, before main has saved a Windows cache (runs 40431035981 and 40431035912, 554ec7b). There the first OVRTX and Isaac RTX pairings take 3m 51s and 2m 57s while their shader caches fill; reused, they take 27s and 25s. Reused: the runner of the previous run, which keeps the environment and caches (runs 40288044529 and 40288044651, 223b1af, before the duplicates were removed). ≈ is a sum of steps measured when the same tests ran in separate steps; the changed rows are re-measured on the next run.

Description

• Orchestrator (tools/conftest.py): on Windows, pipes are peeked with PeekNamedPipe before a read, a hung test's process tree is killed with taskkill /T, and report names drop the drive colon. A Windows-only case in test_test_orchestrator_result_handling.py covers the first two.
• Caches: the Linux jobs' Warp kernel and Kit/OVRTX shader caches, each in a Windows collection; pull requests restore, pushes save. The Linux cache actions run bash, which is WSL on these runners, so .github/scripts/windows_ci_caches.py keys and reports them in the same key format, reusing collection_id.py and warp_cache_inventory.py unchanged, and checks that Warp uses its cache directory. A push reinstalls the environment from its own uv cache and clears the cache directories before restoring; uv, its Python and the runner's files still carry over from pull-request jobs.
• Cartpole smoke: the camera-training pairings are a gpu-marked test in the shared cartpole_training_smoke.py. ARM CI runs that file without --run-gpu, so it lists them as 9 skipped tests.
• uv: astral-sh/setup-uv v10.2.0 with uv 0.12.9. v6, which the Linux workflows use, crashes on exit on these runners (Node/libuv assertion error on Windows self-hosted GitHub Actions runner astral-sh/setup-uv#686).
• Installation tests: --basetemp under RUNNER_TEMP, because pytest's default put Kit's DLLs past the 260-character path limit; test_workspace.py is skipped on Windows (see Summary).

Test plan

• pre-commit, the changelog check and the orchestrator's unit tests pass; the new Windows case fails on Linux when only the hung process is killed.
• windows_ci_caches.py on desk against the windows_ci_caches.ps1 it replaces: the keys the .ps1 printed on run 40431035981, and the same save decisions in 6 restore/save scenarios (restore mode; miss with and without new files; hit unchanged, grown, rewritten). The Warp path check fails when Warp ignores the directory. Rerun on 67399ad with the same results.
• Every workflow green after merging main (b62ed9d): Docker + Tests (30 jobs), ARM, Multi-GPU, the x86/ARM/Windows installation tests and windows-ci; both Windows jobs again on 223b1af and, after removing the duplicate tests, on 554ec7b.
• Every workflow on 67399ad (Windows cache script in Python, Cartpole smoke in one file).
• First push to main after merge: the save run rebuilds the environment and publishes the Windows caches.

Type of change

• New feature (non-breaking change which adds functionality)

Release backport

• Backport this pull request to the active release branch after it merges into develop

Screenshots

Not applicable.

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with uv run isaaclab --format
• Documentation changes are not required because the documented install path is unchanged
• My changes generate no new warnings
• I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[isaaclab-review-bot]

Review Summary

This PR adds experimental Windows CI jobs for perception smoke testing, building on the uv-based installation path. The approach is well-structured, with incremental jobs that progressively test more functionality.

✅ Strengths

1. Incremental approach: The workflow builds from simple torch/scipy tests → IsaacLab core install → full perception smoke
2. Cross-platform fix in AssetConverterBase: Moving from hardcoded /tmp/IsaacLab to tempfile.gettempdir() is the correct POSIX/Windows-agnostic solution
3. Proper marker recognition in AppLauncher: The windows_ci and arm_ci markers are now correctly filtered from sys.argv
4. Good use of continue-on-error: true: Appropriate for experimental jobs that should not gate merges

Architecture

• ✅ Clean separation of reusable GitHub Actions (windows-instance-state, windows-sim-paths) from the main workflow
• ✅ Consolidated single-job design prevents runner teardown between steps — smart approach for autoscaling runners
• ✅ OS-level watchdogs via Start-Process + WaitForExit instead of relying on GIL-vulnerable Python thread watchdogs
• ✅ Develop-aligned Isaac Sim builds from internal Artifactory index (authenticated)
• ✅ Perception and training smoke tests enabled (WDDM mode confirmed)

⚠️ Optional Suggestions (Non-blocking)

1. run_docker_tests: 'false' in build.yaml: The comment says "TEMP: disable heavy Linux Docker + Tests" — should be reverted before merge if not intentional.
2. Hardcoded Git Bash Path: C:\Program Files\Git\bin\bash.exe assumes standard installation. Consider fallback or documentation.

---

Update (ad92f43): Reviewed incremental diff from bae6977...ad92f43 (rebase on develop + Windows CI refinements):

Windows CI Changes:

• ✅ Perception test fully re-enabled: Both the standalone test-perception step and the perception subcase in training smoke now run. Perception added to $blocking in aggregate.
• ✅ Develop-aligned Isaac Sim install: Setup step now resolves and installs the develop-branch-aligned Isaac Sim build from the internal Artifactory index (authenticated via ISAACSIM_ARTIFACTORY_READONLY_* secrets). This replaces the previous public pip install and ensures CI tests against the same Sim as the Linux/ARM develop container.
• ✅ Wheel install adjusted: The [all] extra (which pins isaacsim==5.1.0) is no longer used during wheel validation, preventing a downgrade of the develop Sim build.
• ✅ Training smoke expanded: Step renamed to "state + perception" with increased timeout (25 min); -k "not perception" filter removed.

Non-CI changes (develop rebase): The diff includes a large number of unrelated develop-branch changes (cloner refactoring, observation pipeline improvements, new tutorials, bugfixes, etc.) that are not part of this PR's Windows CI scope. No new issues found.

Approval stands.

[isaaclab-review-bot]

Updated Review: Windows CI Pipeline (Part 3)

This is a comprehensive and well-structured PR that adds Windows CI support to IsaacLab. The implementation shows strong attention to platform-specific challenges and includes robust error handling. Below is my detailed analysis:

---

✅ Strengths

1. Solid Architecture

• Clean separation of reusable GitHub Actions (windows-instance-state, windows-sim-paths) from the main workflow
• Marker-driven test discovery (windows_ci) allows adding tests without YAML edits
• Consolidated single-job design prevents runner teardown between steps — smart approach for autoscaling runners

2. Robust Process Management

• OS-level watchdogs via Start-Process + WaitForExit instead of relying on GIL-vulnerable Python thread watchdogs
• Per-job timeouts with --timeout-method=thread (correctly avoiding Unix-only SIGALRM)
• Hard timeouts with explicit kill on hung Kit/Vulkan processes

3. Cross-Platform Improvements

• AssetConverterBase now uses tempfile.gettempdir() instead of hardcoded /tmp — correct fix
• wheel_builder/build.sh PYTHON override handles Windows git-bash (which only has python, not python3)
• HDF5 fixture uses ignore_cleanup_errors=True to handle Windows file-lock races — good catch

4. Comprehensive Environment Setup

• EULA acceptance variables properly configured
• DLL search paths correctly prepended for Vulkan ICD discovery
• Instance state reporting helps diagnose runner issues (GPU presence, disk space, cache sizes)

---

⚠️ Suggestions & Minor Issues

1. Temporary Docker/Tests Disable in build.yaml

run_docker_tests: 'false'  # TEMP: disable heavy Linux Docker + Tests

The comment says "revert before final review" — this should be reverted before merge. Consider adding a # TODO: tag that linters/CI can catch.

2. Hardcoded Git Bash Path

$gitBash = "C:\Program Files\Git\bin\bash.exe"
if (-not (Test-Path $gitBash)) { throw "Git Bash not found at $gitBash" }

This assumes a standard Git for Windows installation. Consider:

• Falling back to where.exe bash or checking common alternate paths (C:\Program Files (x86)\Git\bin\bash.exe)
• Or documenting this as a runner requirement

3. Perception Test Gating
The perception test depends on WDDM mode for Vulkan. If runners could be in TCC mode, consider:

• Auto-detecting driver mode via nvidia-smi output
• Adding a conditional skip rather than a CI failure

4. pytestmark Placement in test_configclass.py

from isaaclab.utils.configclass import _field_module_dir, configclass

pytestmark = pytest.mark.windows_ci
from isaaclab.utils.dict import class_to_dict, ...

The pytestmark is sandwiched between imports — while functional, it's unconventional. Consider grouping it with other module-level declarations after all imports.

5. Magic Numbers
Several timeout values are hardcoded (300000ms, 180000ms, 600s, etc.). Consider:

• Documenting why these specific values were chosen
• Using named constants or environment variables for runner-specific tuning

---

📝 Questions

1. Checkpoint Cleanup: The cartpole training smoke tests run two iterations and create checkpoints. Are these cleaned up, or will they accumulate on runners?

2. Change Detection Logic: The changes job patterns don't include .github/actions/windows-* — is this intentional? Changes to these actions wouldn't trigger the workflow for validation.

3. Python 3.12 Assumption: The user site-packages path assumes Python 3.12:

   "user site-pkgs" = "$env:APPDATA\Python\Python312\site-packages"

   Should this be dynamic based on the installed Python version?

---

🔧 Final Notes

Overall, this is high-quality CI infrastructure work. The attention to Windows-specific quirks (DLL search order, file locking, PowerShell encoding, process watchdogs) demonstrates deep platform expertise.

Before merge:

1. Revert the run_docker_tests: 'false' workaround in build.yaml
2. Consider addressing the Git Bash path hardcoding

Other items are suggestions for future improvement rather than blockers.

---

Update (ad92f43): Reviewed the new commits since bae6977.

Changes Observed

1. Perception/Rendering Steps Re-enabled ✅

   • Commit 27def394 re-enables the perception and rendering test steps in the Windows CI workflow
   • The cartpole-camera perception smoke test now exercises the full RTX/Vulkan path

2. Develop-Aligned Isaac Sim Installation ✅

   • The workflow now installs a develop-aligned Isaac Sim build from the internal Artifactory index (instead of the public pip release)
   • Uses authenticated index URLs with ISAACSIM_ARTIFACTORY_READONLY_USERNAME/PASSWORD secrets
   • The tools/resolve_isaacsim_develop.py resolver verifies the build is on the develop branch
   • Good approach to ensure Windows CI tests the same Sim as Linux/ARM develop

3. Authentication with Read-Only Secrets ✅

   • Proper credential handling with URL-encoding for special characters
   • GitHub masks secrets and uv redacts credentials when echoing index URLs

Outstanding Items

1. build.yaml Still Has run_docker_tests: 'false' ⚠️

   • The temporary disable is still in place with the comment "revert before final review"
   • This needs to be reverted before merge

2. Hardcoded Git Bash Path (unchanged from previous review)

   • Still assumes standard Git for Windows installation at C:\Program Files\Git\bin\bash.exe

Summary

The new commits address the perception/rendering testing and add proper develop-aligned Isaac Sim installation with authentication. The Windows CI workflow is now more comprehensive and aligned with the develop branch testing strategy.

Remaining before merge:

• Revert run_docker_tests: 'false' in build.yaml

---

Update (5c04d23): Reviewed incremental changes since ad92f43.

Changes in This Push

1. Artifactory URL Migration ✅
   • All internal Artifactory URLs updated from urm.nvidia.com to artifactory.nvidia.com
   • PyPI index paths updated from sw-isaacsim-pypi/simple → sw-isaacsim-pypi-local/simple and ct-omniverse-pypi/simple → ct-omniverse-pypi-local/simple
   • This affects both the uv pip install index URLs and the resolve_isaacsim_develop.py --index-url argument
   • Standard infrastructure migration from the legacy URM hostname to the canonical Artifactory hostname with corrected local repository paths

Assessment

This is a straightforward infrastructure-only change — no logic modifications, no new features. The URL pattern change (-local/simple suffix) suggests these now point to the correct local Artifactory repositories rather than virtual aggregators.

No new concerns introduced. Previously noted items still apply:

• build.yaml still has run_docker_tests: 'false' that needs reverting before merge
• Hardcoded Git Bash path remains unchanged

---

Update (5941931): Reviewed incremental changes since 5c04d23.

Changes in This Push

1. Fork PR Secret Handling — Graceful Degradation ✅

   • Previously the workflow would hard-fail (throw) when ISAACSIM_ARTIFACTORY_READONLY_* secrets were not set
   • Now emits a GitHub Actions ::warning:: annotation and proceeds unauthenticated
   • This allows fork PRs (which don't receive repository secrets) to still exercise URL/DNS reachability
   • Auth-dependent failures will surface later in in-repo runs where secrets are present
   • Good approach — matches standard CI best practice for fork-safe workflows

2. Artifactory URL Path Correction ✅

   • Removed /api/pypi/ prefix and /simple suffix from all Artifactory index URLs
   • Before: https://...artifactory.nvidia.com/artifactory/api/pypi/sw-isaacsim-pypi-local/simple
   • After: https://...artifactory.nvidia.com/artifactory/sw-isaacsim-pypi-local/
   • Same change applied to ct-omniverse-pypi-local and the resolve_isaacsim_develop.py --index-url argument
   • This likely aligns with the actual repository layout on the new artifactory.nvidia.com host (non-virtual repos don't use the /api/pypi/.../simple PEP 503 endpoint)

Assessment

Both changes are operational improvements with no logic risk:

• The fork PR fix unblocks external contributors from running the workflow
• The URL path fix ensures correct repository resolution on the migrated Artifactory instance

No new concerns introduced. Previously noted items still apply:

• build.yaml still has run_docker_tests: 'false' that needs reverting before merge
• Hardcoded Git Bash path remains unchanged

---

Update (77685d1): Reviewed incremental changes since 5941931.

Changes in This Push

1. Network Reachability Diagnostic Step ✅
   • Added a new "Probe index reachability (diagnostic)" step in the Windows CI workflow
   • Uses if: always() and continue-on-error: true — purely diagnostic, won't block the build
   • Probes three hosts: artifactory.nvidia.com, urm.nvidia.com, pypi.nvidia.com
   • Tests both DNS resolution ([System.Net.Dns]::GetHostAddresses) and TCP 443 connectivity (Test-NetConnection)
   • Logs runner metadata: COMPUTERNAME, RUNNER_NAME, proxy environment variables, DNS server addresses
   • This decouples network/DNS diagnostics from pip failures — useful for debugging transient CI infra issues

Assessment

This is a non-blocking diagnostic addition. It helps isolate whether Windows CI failures stem from network reachability issues vs. package availability, especially given the recent Artifactory URL migrations. The step runs unconditionally (always()) so it will capture state even on prior step failures.

No new concerns introduced. Previously noted items still apply:

• build.yaml still has run_docker_tests: 'false' that needs reverting before merge
• Hardcoded Git Bash path remains unchanged

[hujc7]

Superseded by #6086 — re-opened from a branch in this repo (not a fork) so the Windows CI job receives the ISAACSIM_ARTIFACTORY_READONLY_* org secrets (fork PRs don't get workflow secrets). Closing this one.

[hujc7]

Reopened — #6086 (same-repo variant) is closed. Confirmed the fork rule was not the blocker: even on an in-repo PR the ISAACSIM_ARTIFACTORY_READONLY_* secrets resolve empty, so they aren't scoped to isaac-sim/IsaacLab. Fix is on the secret placement (scope to this repo like NGC_API_KEY) or bake the read-only creds into the Windows self-hosted runner; no change needed in this PR's wiring. This branch also now has develop merged in (no longer conflicting).

[StafaH]

do we need the test?

[hujc7]

run-ci

[hujc7]

run-ci

[hujc7]

run-ci

[hujc7]

run-ci

[hujc7]

run-ci

[hujc7]

run-ci