[CI] Detect regressions in native execution and fallback coverage #13504

Description

Problem

Correctness-only unit tests can remain green when Gluten stops offloading work and falls back to Spark. The results are still being checked, but the native implementation may no longer be exercised. Plugin registration, enabled configuration, and even a passing native smoke test are insufficient to detect a drop in native execution across the real test workload.

Add CI reporting and regression detection for native versus non-native execution coverage, starting with Delta Spark UT and designing the evidence so it can also support other Gluten test pipelines.

Concrete example: Delta 4.4 partitioned writes

As investigated in #13502, Delta 4.4's writer defaults to physically writing partition columns into Parquet when delta.writePartitionColumnsToParquet is unset. This does not require the separate materializePartitionColumns protocol feature.

Before that PR, Gluten's gate rejected partitioned writes requiring materialization. Under the default settings, those file writes used Delta's JVM writer even with native writes enabled. Reads and other operators could still execute in Velox. Consequently, passing partitioned-table tests and a GlutenDelta command in the plan did not establish native file-write coverage. Enabling native materialization exposed existing native limitations again.

The current Delta known-failures gate compares test outcomes, not how much work ran natively. Removing now-passing failures after fallback can make the baseline look better while native coverage has decreased.

Requested validation

• Establish minimum native-execution proof from at least one actual workload test in each Spark test JVM/fork expected to exercise Gluten. A missing proof must fail independently of aggregate ratios or correctness baselines. Explicitly document exemptions for forks containing only Spark-free, vanilla-only, or no-work tests; do not infer an exemption merely because no native execution was observed.

• Record execution evidence with stable test/suite identities and the backend, Spark/Delta versions, and relevant configuration. Report counts and ratios for native, partially native, and fully fallback executions, with explicit handling of Spark-free/no-work tests and intentional disabling.

• Break coverage down by operation category, especially native scans versus native file writes, rather than relying on one global percentage. Capture fallback reasons and identify which tests or operations lost native coverage.

• Use final executed plans/runtime evidence where appropriate, including AQE. File writes need evidence of the actual writer used; neither a GlutenDelta wrapper nor native upstream input operators prove native file writing. Reuse existing metrics/reporting where possible.

• Compare comparable workloads against a base revision or an explicitly reviewed coverage baseline. Account for added/removed tests, matrix/version differences, retries, and duplicate events. Report the numerator and denominator; do not treat a change in raw counts or one aggregate ratio as sufficient evidence by itself.

• Fail CI for unexpected loss of native coverage according to documented, reviewed expectations/tolerances. Intentional fallbacks remain valid, but coverage reductions require an explicit explanation. Missing or incomplete instrumentation must not silently look like zero applicable work or successful coverage.

• Keep this gate independent of correctness known-failure lists and flaky quarantine. Refreshing known-failures.txt must not automatically accept an offload regression. Canary queries should not count as evidence that the actual workload retained native coverage.

Acceptance criteria

• Every non-exempt Spark test JVM/fork expected to exercise Gluten provides native execution evidence from the actual workload. A negative control disabling Gluten in one such fork, or dropping its evidence, fails the gate even when other forks still execute natively.

• CI publishes native/fallback counts, ratios, operation-level breakdowns, and changed test identities alongside correctness results.

• A negative control that forces previously native partitioned file writes to fall back fails the coverage gate even when results remain correct and native scans/smoke tests still pass.

• A broad or complete loss of native execution cannot produce a green coverage result.

• Intentional disabling, unsupported operations, and no-work tests have explicit treatment without requiring every test query to offload.

• Missing/truncated evidence and duplicate events are detected or handled without inflating coverage; aggregation verifies required shard evidence is present.

• Baseline comparison and approved coverage changes are documented and covered by positive and negative tests.

Related work and scope

This issue is the single follow-up for native-execution evidence and fallback-regression detection, superseding #13496 and #13499. The earlier PR remains available as an implementation reference; reuse its evidence recording and missing-evidence enforcement where useful rather than maintain two independent validation pipelines.

A real test executing natively proves that Gluten is functioning for that observed execution. One native test somewhere in the overall run does not establish activation in every fork or Spark session. Require native workload evidence per non-exempt Spark test JVM/fork expected to exercise Gluten, and document the coverage boundary; this does not claim that every auxiliary session is correctly configured or that every query must offload. This consolidation does not require porting all of the earlier fixture-level checks. Legitimate fallback and explicitly justified vanilla-only/no-work cases remain supported.

Drafted with assistance from GitHub Copilot (GPT-6 Astra).

Gluten version

main branch