You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(dashboard): run Playwright tests as non-blocking outerloop - #20642
Dashboard Playwright tests run in a browser-driven, timing-sensitive environment. This PR separates them from the regular Dashboard test suite and runs them in the outerloop lane. Recognized test failures remain visible in results without failing the scheduled run, while build, setup, runner, malformed-result, and unexpected empty-run failures remain blocking.
Moves the browser-driven Dashboard tests into Aspire.Dashboard.Playwright.Tests and selects the project for outerloop runs.
Enables NonBlockingTestFailures for this project and validates Microsoft.Testing.Platform exit codes and TRX files before making recognized test outcomes non-blocking.
Filters marked non-gating test artifacts before scheduled outerloop failure classification so they are not included in gating-failure issues.
Keeps Native AOT Dashboard validation blocking.
Removes an obsolete shared-source include so the extracted Playwright project builds.
A separate project isolates the browser suite without applying continue-on-error to the entire job. No product dependencies are introduced.
Validation
Aspire.Dashboard.Playwright.Tests project build: passed with 0 warnings and 0 errors.
Ankit Jain (@radical) Playwright tests are very fragile, and there is no dev feedback until they're broken on outer loop. This is going to be an ongoing problem.
I think the only solution will be to move them to there own job, and not have the build rely on them all passing. Occassionally someone will need to take a look at fixing them and identifying any real regressions.
Ankit Jain (radical)
changed the title
test(dashboard): disable two failing outerloop tests
test(dashboard): disable three failing outerloop tests
Oct 1, 2026
Ankit Jain (radical)
changed the title
test(dashboard): disable three failing outerloop tests
test(dashboard): run Playwright tests as non-blocking outerloop
Oct 1, 2026
[automated] James Newton-King (@JamesNK) Implemented this direction. The Dashboard Playwright suite now lives in its own outerloop-only test project and therefore runs as a separate matrix job.
The project opts into NonBlockingTestFailures: recognized Microsoft.Testing.Platform test outcomes remain visible in the job summary and artifacts without failing the workflow, while missing or invalid result evidence, unknown exit codes, setup failures, and infrastructure/configuration failures still fail the job.
The three tests tracked by #20640, #20641, and #20644 are re-enabled. Native Dashboard validation still runs its Playwright test as a strict, gating check.
This specialized workflow consumes eng/testing/CITestsProperties.props through the runsheet builder, and the new nonBlockingTestFailures property directly controls whether its jobs gate. Because that input is absent from this pull_request.paths list, a props-only change can alter outerloop behavior without triggering the workflow's PR smoke run. Add it to the shared paths list.
Add CITestsProperties.props to quarantine PR paths
.github/workflows/tests-quarantine.yml:33
This specialized workflow also consumes eng/testing/CITestsProperties.props when generating its runsheet. Without that file in pull_request.paths, a props-only change can alter quarantine failure handling without triggering the quarantine workflow's PR smoke run. Keep this shared dependency synchronized with the outerloop workflow.
Resolve conflicts between the Playwright CI split and upstream
workflow changes without rewriting the PR's history.
Use the Windows NuGet cache for both Native AOT Dashboard test steps.
Keep native validation blocking and preserve the specialized runner's
advisory test-failure policy while removing the required input default.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cd22c3aa-82c7-4428-9ca4-ca9af142519c
The Playwright test project still referenced TestDashboardTelemetrySender
from tests/Shared after the telemetry migration removed that helper from
main. Remove the obsolete compile include so the project builds again.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: da2fb89c-73ca-4d5b-84a3-4745c3731257
🔴 BLOCKED — fix before merge: the moved Dashboard browser suite, representative quarantine producer integration, and scheduled artifact-filter integration lack qualifying final-tree execution. Required validation: 🔴 3 missing · 🟡 0 attention · 🟢 16 verified · ⚪ 0 not applicable
EDIT: ☑️ Validation finished:Quarantined Tests run — canceled again; package and Linux x64 archive jobs did not complete, so the quarantine test matrix did not run. Score not yet re-evaluated. EDIT: ☑️ Validation finished:Outerloop Tests run — cancelled; the CLI E2E Docker image build timed out fetching moby/buildkit:buildx-stable-1 from Docker Hub, so no outerloop test matrix ran. Score not yet re-evaluated. EDIT: ☑️ Validation finished:Quarantined Tests run — cancelled; the CLI E2E Docker image build timed out fetching moby/buildkit:buildx-stable-1 from Docker Hub, so quarantine tests did not run. Score not yet re-evaluated. EDIT: ✅ Local integration: final-tree fixture pipeline passed marker filtering → TRX failure extraction → mocked issue reporting; no GitHub writes. This did not exercise the scheduled event or artifact-download action. EDIT: ☑️ Validation finished:Outerloop Tests run — success at workflow level, but both Dashboard Playwright OS lanes reported 14 failures, 74 passed and 2 skipped out of 90; failures were configured as non-gating. Score not yet re-evaluated. EDIT: ☑️ Validation finished:Quarantined Tests run — cancelled; its Docker image build hit the same Docker Hub BuildKit timeout, so no quarantine test matrix ran. Score not yet re-evaluated. EDIT: ☑️ Validation finished:Quarantined Tests run — failure after cancellation; runsheet generation and the Docker image build succeeded, but the package build and Linux x64 archive were canceled before the quarantine test matrix ran. Score not yet re-evaluated. Independent review: completed Coverage aspects (catalog v1): 🔴 3 missing · 🟡 0 attention · 🟢 5 verified · ⚪ 4 not applicable
Material gaps
🔴 Gap: Dashboard Playwright outerloop suite — The moved project's browser fixtures, test discovery and project-level nonblocking integration were not exercised as an outerloop test job.
Gap details
Reason: The moved project's browser fixtures, test discovery and project-level nonblocking integration were not exercised as an outerloop test job.
Evidence: Outerloop runsheet job built the new project and discovered 90 tests, then selected only Acquisition for three OS variants. No Dashboard matrix job ran. Native Windows job passed one packaged-native browser test, not the remaining suite.
🔴 Gap: Quarantine result-producer compatibility — Representative real-producer integration remains missing for normalized empty-lane TRX/counters and the materially distinct package-dependent invocation. Acquisition smoke does not exercise those contracts.
Gap details
Reason: Representative real-producer integration remains missing for normalized empty-lane TRX/counters and the materially distinct package-dependent invocation. Acquisition smoke does not exercise those contracts.
Evidence: Quarantine run executed Acquisition's no-package invocation on Linux, Windows and macOS. The inspected Linux job reported exit 0 and one executed test in one valid TRX. A genuine normalized empty lane and the distinct package-dependent DLL invocation remain unobserved under the advisory verifier. Runner invocation branches, empty-lane contract, and the RequiresNugets project flag plus existing quarantined CLI E2E scenarios establish these consumers. The 37 passing verifier fixtures already prove classification and abort/missing-evidence rejection; neither deliberate crashing nor every project is required.
🔴 Gap: Scheduled outerloop artifact filtering — The download → marker filtering → failed-test extraction → reporter integration did not execute. A PR smoke or workflow_dispatch run cannot reach this schedule-only branch.
Gap details
Reason: The download → marker filtering → failed-test extraction → reporter integration did not execute. A PR smoke or workflow_dispatch run cannot reach this schedule-only branch.
Evidence: Outerloop Report failures job was skipped. The changed filtering step only runs after a scheduled failure, checks out main and uses an issues-write reporter. Four filter fixture tests passed, but neither a final-source equivalent trial nor a qualifying non-author maintainer acceptance was found.
Reviewer actions
Required before merge: Validate the affected specialized producer boundaries: Dashboard Playwright outerloop suite and Quarantine result-producer compatibility, using final definitions and the pinned tree.
Done when, evidence and safety
Done when: The affected Dashboard project executes on its emitted OS lanes with nonzero results and advisory metadata. Representative quarantine integration shows genuine normalized empty-lane TRX/counters and the distinct package-dependent DLL producer reaching the shared verifier. Compare affected runsheet/OS sets against the intended split. Existing passing classification fixtures remain reusable; deliberately aborted/crashed producers and every quarantined project are not required.
Evidence: Qualifying final-source targeted or equivalent controlled runtime, runsheets, invocation/normalization/verifier logs and TRX counters. The Acquisition smoke and one native browser test do not exercise these remaining boundaries. Full dispatch is an operational option only if the supported route cannot target them, not intrinsically required coverage.
Safety: Separate scoped authorization and an executable final-source ref are required. Both workflows expose workflow_dispatch without filter inputs; no filtered dispatch is currently supported. Dispatch skips schedule-only issue reporters. The fork-backed PR has no upstream branch at this head in the read-only branch listing; equivalent trees/tags were not searched. Establish an authorized ref or qualifying controlled equivalent and check concurrency. Do not dispatch the fork name upstream, use a base-only definition, invent inputs or launch unrelated/deployment scenarios merely to fill this gap. No run is launched by this audit.
Required before merge: Establish supported validation of Scheduled outerloop artifact filtering, preferably without issue publication, or record qualifying non-author maintainer acceptance of its remaining unrun risk.
Done when, evidence and safety
Done when: An authorized equivalent final-source trial demonstrates download → marker exclusion → failed-test extraction → reporter classification, or a non-author maintainer with write access explicitly accepts this named unsafe-to-run remainder, its constraint and alternative checks. Accepted unrun remains yellow.
Evidence: A runtime receipt identifying the tested source and permitted environment substitutions, or a linked qualifying acceptance. Four passing filter fixture tests establish the algorithm only.
Safety: The original branch requires a scheduled upstream failure, checks out main and writes issues. No currently established nonpublishing trial was found. Establish the safe trial with the workflow owner; generic testing authorization must not be used to publish upstream issues or bypass production gates.
Coverage aspects (catalog v1)
🔴 1. GitHub execution and coverage — hierarchy roll-up only; not counted.
Reason and evidence
Reason: The roll-up takes the worst applicable child disposition.
Evidence / missing boundary: Final-definition execution is verified; outerloop and quarantine scenario coverage remain missing; deployment is not applicable.
Reason: Six directly touched executable YAML definitions ran at the pinned merge tree.
Evidence / missing boundary: Run and definition receipts establish load and execution; specialized scenario and schedule-only gaps have separate homes. Requirement homes: native-dashboard-validation.yml, run-tests.yml, specialized-test-runner.yml, tests-outerloop.yml, tests-quarantine.yml, tests.yml.
Reason: Dashboard browser tests moved into a new outerloop-only, nonblocking project.
Evidence / missing boundary: The automatic run discovered 90 tests but executed only Acquisition smoke; the Windows native test is not full suite coverage. Requirement homes: Dashboard Playwright outerloop suite.
Reason: No deployment scenario, CLI deployment, Azure integration, deployment workflow, fixture or runtime package input changed.
Evidence / missing boundary: The shared Playwright target itself is unchanged; narrowed routing is validation policy, not a deployment workload change.
Reason: Changed metadata is imported by TestEnumerationRunsheetBuilder and SpecializedTestRunsheetBuilderBase, not the official managed build invocation.
Evidence / missing boundary: The pinned definition-1602 BuildAndTest invokes SkipTestProjects=true; its native/sign/pack/assemble inputs are unchanged. No template-test invocation/plumbing or Arcade target was modified. Loading azdo-internal and read-only definition/build discovery did not turn unrelated main builds into receipts. Official invocation.
🟢 3. CI selection/result gates — applicability: yes.
Reason and evidence
Reason: Selectors, metadata propagation and nonblocking result verification changed.
Evidence / missing boundary: Behavioral Infrastructure TRX and source/consumer tracing cover the contracts; source-string tests are not treated as runtime proof. Requirement homes: Selector routing and exclusion contracts, Nonblocking result classification, Nonblocking matrix metadata propagation.
Reason: Managed Dashboard test organization and packaged-native validation changed.
Evidence / missing boundary: Managed Linux/Windows jobs and six RID native jobs passed; one Windows browser test executed. Requirement homes: Managed Dashboard and packaged-native regression coverage.
Reason: Changed YAML, Bash and PowerShell require separate analyzers and embedded coverage.
Evidence / missing boundary: Paired scans and embedded diagnostic comparisons were completed; generic Write-Host warnings are explained, not silently discarded. No gh-aw generated source/lock changed. Requirement homes: actionlint paired workflow scan, ShellCheck standalone and embedded scans, PSScriptAnalyzer standalone and embedded scans, zizmor paired Actions scan.
⚪ 7. Dependency/action/task/tool version safety — applicability: no.
Reason and evidence
Reason: No action SHA, Azure task selector, central dependency version or CI tool version changed.
Evidence / missing boundary: Package references moved with the browser test project and retain the same central versions; Playwright provisioning targets are unchanged.
Reason: Result filtering and schedule-only reporting interact with workflow artifacts and issue-write permissions.
Evidence / missing boundary: Separate source/event/consumer review found no new exploitable trust regression; runtime gaps remain independently scored. Requirement homes: Targeted CI trust-boundary review.
Reason: No skill, agent prompt, gh-aw Markdown source or executable agent decision contract changed.
Evidence / missing boundary: COPILOT maintenance comments only update the shared path-list description; they do not change tool access, decisions, mutations or published agent output.
Successful requirement evidence
native-dashboard-validation.yml — Verified: Expected: final-definition execution. Observed: Six RID jobs passed; Windows x64 executed one NativeDashboard browser test. The managed NativeDashboard filter ran zero tests and is not browser coverage. Runtime receipt: run.
run-tests.yml — Verified: Expected: final-definition execution. Observed: Regular test jobs exercised the strict path; quarantine Acquisition exercised the new verifier and marker steps on three operating systems. Runtime receipt: run.
specialized-test-runner.yml — Verified: Expected: final-definition execution. Observed: Runsheet generation, properties injection, Acquisition matrix jobs and final aggregation completed in both specialized workflows. Runtime receipt: run.
tests-outerloop.yml — Verified: Expected: final-definition execution. Observed: The final pull_request definition ran; its schedule-only reporter was skipped and is separately inventoried. Runtime receipt: run.
tests-quarantine.yml — Verified: Expected: final-definition execution. Observed: The final pull_request definition ran; Acquisition smoke jobs and final aggregation passed. Runtime receipt: run.
tests.yml — Verified: Expected: final-definition execution. Observed: The final reusable definition executed the enforcing selector, regular jobs, Native AOT jobs and final result gate successfully. Runtime receipt: run.
Selector routing and exclusion contracts — Verified: Expected: native job routing and intentional outerloop-only selection without losing managed import consumers. Observed: 145 TestTriggerMapTests, 38 SelectTestsCliTests and 18 SelectTestsWorkflowTests passed in Infrastructure TRX job. The selection artifact has enforcing mode, skipLayer1=false and ALL due to run-tests.yml. ImportPaths indexing and solution membership support Playground/Hosting.Testing attribution; their omitted Layer-2 duplication is not a defect.
Nonblocking result classification — Verified: Expected: reject missing/malformed evidence, invalid counters, unknown/infrastructure exits and unintended empty runs; accept known outcomes only with executed-test evidence. Observed: all 37 VerifyNonBlockingTestResultsTests cases passed in Infrastructure job, including malformed-plus-valid TRX, all-skipped, missing executed count, artifact-only aborts and zero-test opt-in cases.
Nonblocking matrix metadata propagation — Verified: Expected: preserve nonBlockingTestFailures and false defaults. Observed: 22 BuildTestMatrixTests passed, including PreservesProjectLevelNonBlockingTestFailures; specialized runsheet emitted the new field for Acquisition as false. Shared specialized property resolution resolves the same registry dynamically.
Managed Dashboard and packaged-native regression coverage — Verified: Expected: managed tests still run strictly and native interactivity remains gating. Observed: managed Dashboard Linux/Windows and all six packaged-native RID validations passed in CI attempt 3; Windows x64 passed one NativeDashboard browser test. The separate managed NativeDashboard_* selection was empty and normalized exit 8, so it contributes no native test coverage.
Marker artifact layout and fail-closed filtering — Verified: Expected: delete only immediate marked logs-* children; preserve gating results; reject malformed layouts before any deletion. Observed: four FilterNonBlockingTestArtifactsTests passed in Infrastructure job. Filter source validates exact parent/name/testresults layout in a first pass. Quarantine job logs show the marker step ran; uploading includes testresults. This fixture receipt does not close scheduled integration.
actionlint paired workflow scan — Verified: Expected: validate handwritten workflow syntax and reusable inputs with the repository pin. Observed: actionlint 1.7.12 scanned base/head with the same head configuration, integrations disabled; zero findings on both sides. Final-head CI handwritten-workflow lint also passed. Local typed receipt: scanner details.
ShellCheck standalone and embedded scans — Verified: Expected: inspect the new artifact filter and extractable Bash/sh in all six touched workflows. Observed: ShellCheck 0.11.0 standalone base/head comparison had zero findings; 37 Bash blocks per tree were analyzed, with no new diagnostics after stable filename normalization. Local typed receipt: scanner details.
PSScriptAnalyzer standalone and embedded scans — Verified: Expected: inspect the new verifier and extractable PowerShell blocks. Observed: PSScriptAnalyzer 1.25.0 captured diagnostic objects for the standalone script and 17 base/19 head embedded blocks. Six new PSAvoidUsingWriteHost warnings concern intentional displayed GitHub job output/annotations, not returned pipeline objects. The rule remains enabled and discourages host output; this command intentionally communicates through visible annotations and process exit status, so the warnings are retained but are not functional/security defects. Embedded comparison has no new diagnostics. Local typed receipt: scanner details.
zizmor paired Actions scan — Verified: Expected: same-version baseline and head scan, with new high-confidence handwritten findings identified. Observed: zizmor 1.30.1 regular persona, no config changes, inline suppressions preserved: 288 baseline findings, 285 head findings, zero new/worsened, three native template-injection findings resolved. Local online mode was not established; it is not called an online pass. Final-head CI scan ran 1.30.1 with online-audits=true and a masked token using the exact action contract. The local baseline delta, not the green upload job alone, establishes no new findings.
Targeted CI trust-boundary review — Verified: Expected: separately inspect event/ref/permission, script-expression, artifact and mutation trust. Observed: None found for new exploitable trust-boundary regressions. PR test paths remain read-oriented; issue writes stay confined to upstream schedule/failure reporters. Native inputs move through environment variables rather than source interpolation. The filter handles NUL-delimited filenames, quotes paths and validates all markers before deletion. Markers remove downloaded copies, not original run artifacts; verifier failures still propagate through needs/results as infrastructure failures. Existing trusted runsheet-to-shell expansion remains baseline behavior, not a new permission/data source. Native source, runner, reporter gate. Analyzers are separate receipts, not this review.
Scope and method
Pinned scope:microsoft/aspire, head ea83da56eb0c386d64e7cd710a50cee8762a8e75, base 4a69a9f04f983f5a9f58a42e97ec688a70461ada; all 48 changed paths and complete paginated discussion were retained. The tested synthetic merge was b0dd9ed62e45dd562aba3e2df4f5ab464285f0e4, with direct parents equal to this base and head. Its tree was 180cff930be754492de43fc3db9a8dfe2ef1a541. Immutable reusable-workflow references identify the same merge; content-API blob IDs for both specialized root workflows equal the PR-head blobs. Pre-review and final-return refreshes confirmed unchanged head/base; the PR remains open and unmerged.
Execution classification: all six touched YAML workflows are executable and required, and all loaded final definitions ran. This does not certify their unexecuted branches. The scheduled-filter boundary is separately required under artifact integrity. The JavaScript reporter diff changes comments only and has no new executable behavior. Trigger-map changes are validation policy; they do not require running every workload mentioned. No action pin/task/tool version changed.
Consumer audit: shared loose-input routing still covers root build files, RepoTesting, Docker, TemplatesTesting, template data and certificates. Playwright target/runtimeconfig paths now have exact browser/template/native consumers; evaluated import indexing covers solution-member Playground and Hosting.Testing consumers. The Playwright provisioning target is not changed by this PR. No deployment or official-build template invocation was changed. Definition 1602 was identified live as microsoft-aspire with the expected YAML; unrelated main/personal builds are not validation receipts for this PR.
Policy observation: moving Dashboard Playwright to an outerloop-only project deliberately removes it from regular PR matrix selection, and the project explicitly makes known test outcomes non-gating. That is the intended policy, not an accidental selector defect. The native interactivity job invokes it strictly. Actual outerloop execution of the project remains missing.
Falsifiable tests: reverting executed-counter validation would fail RejectsAllSkippedTests; reverting fail-closed TRX validation would fail RejectsMalformedTrxWhenAnotherTrxIsValid; weakening marker path checks would fail RejectsMarkerOutsideTestResultsDirectory and RejectsMarkerInNestedArtifactDirectory. All these cases actually passed in final-tree TRX, not merely in source.
Independent review: one fresh read-only adversarial reviewer loaded the global skill and read its policy in full. Auto / Balance was requested through supported kickoff model/auto_tier fields. The reviewer reports Auto; its concrete model is hidden, and Balance/actual reasoning settings are not independently exposed, so exact configuration equivalence cannot be attested. It independently inspected immutable source, raw analyzer packets, Infrastructure TRX, all CI attempts, specialized/native logs, discussions, and refreshed PR/merge identity. No recursion, execution, installation or mutation occurred.
Reconciliation E1 — resolved: the reviewer found the required testing broader than the demonstrated producer boundary. Accepted after pinned-source checks of the package-dependent DLL versus no-package dotnet-test branches, specialized empty-lane normalization, RequiresNugets CLI E2E consumer and existing quarantined scenarios. Quarantine now requires representative empty-lane and package-dependent advisory integration, not every project or intentional crashes. Dashboard requires its affected project/OS lanes, not unrelated outerloop workloads. Full dispatch is only an operational option when supported targeting is unavailable. The 37 passing verifier fixtures already prove classification and fail-closed abort/missing-evidence rejection. The three narrower integration gaps remain red; no additional omitted load-bearing requirement or final-source defect was found in the bounded review.
Policy revision: the global normative policy and validator changed during the audit. The updated policy was re-read in full, and the draft was regenerated with its fixed-order catalog and mechanically validated before independent review.
Safety: global skill and normative policy used; audit only. No code/PR edits, posts, pushes, remote runs or external mutations occurred. Scratch analyzer extraction replaced expressions with inert literals and never executed embedded PR code. Read-only API queries and artifact downloads supplied primary evidence.
Local analyzer receipts
Command: python3 -B "$S/scan_ci_delta.py" --facts "$A/facts/facts.json" --repo-dir "$R" --scratch-dir "$A/scans" --tool zizmor --tool actionlint --tool shellcheck --tool psscriptanalyzer --summary. Exit 1 means six new diagnostic warnings, not scanner failure; packet complete=true, execution errors=0. The helper's coverage_complete=false initially recorded unreviewed embedded scope and unestablished local online mode; those limits were reviewed rather than treating the packet as universally clean.
Embedded supplement: python3 -B "$A/scan-embedded.py" completed; 74 Bash blocks and 36 PowerShell blocks across base/head, no new normalized diagnostics. Expressions were replaced with AUDIT_VALUE; source/workflow/job/step mappings and raw diagnostic objects were retained. This is parser/static evidence, not workflow execution or an injection-proof substitution. Semantic trust review was separate.
actionlint 1.7.12 is the head repository pin; same head config on both trees, no config changes. ShellCheck 0.11.0 and PSScriptAnalyzer 1.25.0 were latest-release resolutions. Six standalone PSSA Write-Host warnings are retained and explained; embedded Unicode/BOM and other pre-existing findings are not new regressions.
PSScriptAnalyzer rule documentation confirms the warning concerns displayed host output rather than objects sent down the pipeline. The annotation/exit-status contract, not an assumption that modern PowerShell disables the rule, supports this disposition.
zizmor 1.30.1 matched the resolved CI version; same version/persona/suppressions on both trees. No new/worsened findings or changed suppressions. Three native template-injection findings resolved. Local online audit mode remains unestablished; the CI run explicitly enabled online audits with a masked token using the exact action input contract. Existing baseline findings are not new PR defects.
Generated consistency: no gh-aw source, lock file or compiler pin changed, so no compiler regeneration/trial requirement was manufactured.
Runs and links consulted
CI attempt 1: final head/merge, failed. One Windows Hosting test, WithUrlsTests.WithUrlForEndpointCanLinkToAnotherResourcesEndpoint, timed out in Host.StopAsync; the CLI E2E Docker-image dependency was canceled. Final Results and Final Test Results reflected those dependencies. The shared runner is in scope, so this failure was not silently waived as unrelated.
CI attempt 2: same tree, failed aggregates. Hosting-5 Windows passed; the Docker-image dependency remained canceled. Successful affected jobs can count individually, but canceled consumers did not.
CI attempt 3: same tree, success. The Docker image and consumers ran successfully; Hosting-5 Windows passed; both aggregate checks succeeded. Across all attempts there were five failed job records: one test job plus four dependency-result gates, and two canceled image records. No unresolved failed final-attempt job remains.
Outerloop PR run: final head/merge, success, but smoke only. Built/discovered the new browser project (90 tests), then selected Acquisition on Linux/Windows/macOS. Package/native/image prerequisites and schedule-only Report failures were skipped; no moved Dashboard suite job executed.
Quarantined PR run: final head/merge, success, smoke only. Acquisition ran on Linux/Windows/macOS. The inspected Linux verifier classified exit 0 and 1 executed test in valid TRX. It is not evidence of full quarantine or a real empty lane.
Infrastructure test job: 1,536 succeeded, zero failed/skipped; downloaded TRX independently identified 37 verifier, four artifact-filter, 22 matrix, 145 trigger-map, 38 selector-CLI and 18 selector-workflow cases as Passed. Selection artifacts show enforcement and ALL selection for the current PR, not a single-input proof for every future mapping.
Windows native job: managed NativeDashboard selection had zero tests and exit 8 normalized; the browser selection succeeded with one test. Six RID validation jobs passed, but browser interactivity ran only on Windows x64.
Final-head zizmor and handwritten lint: scanner configuration/steps inspected. A SARIF upload's success is not zero findings; the paired finding delta is the comparison receipt.
Complete PR discussion was read. Maintainer proposal supports nonblocking browser policy but does not accept the named unrun boundaries. Author implementation response is not non-author acceptance.
Prior confidence reports first, second, third, and fourth refer to older heads and are discovery context only, not current execution or acceptance. Their cited stale runs were not fetched as current receipts.
The run-discovery helper inventories older branch runs as different_subject_revision. These are not counted. Linked retry comments and earlier review receipts identify history only; no valid acceptance for the current three missing boundaries was found.
MTP exit-code primary contract confirms known test outcomes and distinguishes empty execution from infrastructure/configuration errors.
Authorized testing handoff
Use the pr-testing skill to test PR #20642. Reuse successful final-tree regular CI, native Windows browser coverage, Infrastructure classification fixtures and Acquisition specialized smoke. With separate scoped authorization, establish qualifying targeted or equivalent final-source execution of the affected Dashboard Playwright project/OS lanes and representative quarantine normalized-empty/package-dependent advisory producers. Compare affected runsheet coverage against the intended split. No deliberate crashes or exhaustive unrelated project set are required. Both workflows lack dispatch filter inputs, and no executable upstream final-source ref is established; determine authorized supported targeting/ref setup before prescribing commands. A full dispatch may be an operational option if no narrower qualifying route exists. Separately establish nonpublishing final-source scheduled-filter/extraction/reporter integration, or qualifying non-author acceptance of its unsafe unrun boundary. Do not publish upstream issues, alter production gates, deploy, push or start runs based on this audit.
EDIT: ☑️ Validation finished:Quarantined Tests run — cancelled; package and Linux x64 archive jobs remained queued with no assigned runner for 8-core-ubuntu-latest; no quarantine test matrix ran. Score not yet re-evaluated.
[automated] Updated the CI confidence report with the trial outcomes. The targeted Dashboard Playwright run completed, but both Ubuntu and Windows lanes recorded 14 failed, 74 passed, and 2 skipped tests; the workflow was green only because test failures were configured as non-gating. Quarantine producer retries never reached the test matrix because Docker Hub timed out fetching moby/buildkit:buildx-stable-1. The report remains blocked, including the unexecuted scheduled artifact-download path.
[automated] Full PR-tree validation completed: Dashboard Playwright and Quarantine coverage gaps are closed.
CI-confidence gap
Result
Dashboard Playwright outerloop
Ubuntu and Windows each ran 90 tests: 74 passed, 14 failed, 2 skipped. Both jobs succeeded—the test failures were correctly nonblocking.
Quarantine producer compatibility
Full workflow passed, including package-dependent tests with failures and a genuine zero-test lane accepted after exit-code normalization.
Scheduled artifact filtering/reporting
Still unverified end-to-end: manual dispatch skips this schedule-only job. Earlier local integration covers filtering → extraction → mocked reporting, but not artifact download or scheduled execution.
Outerloop's overall result was red from CLI E2E Ubuntu and Templates Ubuntu—not Dashboard Playwright. Both runs used the PR merge tree. The original report's remaining gap is the scheduled reporting path; these runs do not establish an all-green CI-confidence verdict.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Description
Dashboard Playwright tests run in a browser-driven, timing-sensitive environment. This PR separates them from the regular Dashboard test suite and runs them in the outerloop lane. Recognized test failures remain visible in results without failing the scheduled run, while build, setup, runner, malformed-result, and unexpected empty-run failures remain blocking.
Aspire.Dashboard.Playwright.Testsand selects the project for outerloop runs.NonBlockingTestFailuresfor this project and validates Microsoft.Testing.Platform exit codes and TRX files before making recognized test outcomes non-blocking.A separate project isolates the browser suite without applying
continue-on-errorto the entire job. No product dependencies are introduced.Validation
Aspire.Dashboard.Playwright.Testsproject build: passed with 0 warnings and 0 errors.Addresses #20640
Addresses #20641
Addresses #20644
Checklist
<remarks />and<code />elements on your triple slash comments?