[CI][Delta] Validate default Gluten activation in Delta Spark unit tests #13496

Description

Problem

The Delta Spark UT workflow (.github/workflows/delta_spark_ut.yml), introduced in #12388, runs upstream Delta tests against a Gluten Velox bundle.

The pipeline now supplies JVM-wide Gluten defaults through java-test-args.sh, including spark.plugins=org.apache.gluten.GlutenPlugin (#13131). However, supplying defaults is not the same as verifying the actual test environment. A regression in JVM options, plugin registration, or suite/session setup could cause tests to run with vanilla Spark and pass without validating the intended Gluten-enabled environment.

This is a missing safeguard, not a claim that the current tests are running without Gluten. The known-failures gate may detect such a change indirectly through now-passing tests, but it should not be the mechanism that proves Gluten is active.

Coverage snapshot

For the successful October 28 run, matching the eight shards' executed-test manifests to transitive fixture inheritance in Delta 4.4.1 / Spark 4.1 gives:

This excludes 5,160 skipped tests. At least 518 tests across 43 of the suites outside DeltaSQLCommandTest use other Spark-backed fixtures or direct context/session creation. These include plain SharedSparkSession, Hive, Java/JUnit, and the newer session-test fixture. The remaining tests include helpers that may not require Spark.

Therefore, adding an assertion only to DeltaSQLCommandTest would leave a meaningful coverage gap. These counts describe one run's fixture reach, not guaranteed runtime coverage or fixed thresholds for future runs.

Proposed approach

Start with CI-only fixture validation and a native-execution smoke test, without introducing a Java agent or changing production Gluten behavior:

1. Add runtime checks to DeltaSQLCommandTest and the identified alternative Spark fixture/session-creation paths. Validate the actual context's plugin configuration, installed Gluten SQL rules, and effective enablement, including the thread-local disable setting. Check the default state before test execution and its restoration after scoped overrides.
2. Execute a small, supported query in every forked JVM that runs Spark-backed tests. Verify both a Gluten/native execution plan and successful execution. Use the inherited Gluten configuration rather than explicitly re-enabling Gluten inside the smoke test.
3. Record validation evidence and enforce it independently of the known-failures comparison. Missing required evidence or a validation violation must fail the shard, including during baseline refresh and when a test otherwise matches a known-failure or flaky entry.
4. Report coverage and exceptions explicitly. Spark-free tests can be classified as not applicable. Intentional disabling is allowed, but a disabled setting must not automatically exempt a test from validation.

Acceptance criteria

• Removing Gluten from the inherited plugin configuration, preventing its SQL-rule installation, or disabling it by default causes CI to fail with an actionable diagnostic.
• Missing required guard or per-JVM smoke-test evidence fails CI rather than silently reducing coverage.
• Validation failures cannot be accepted by known-failures.txt, flaky quarantine, update_baseline=true, or fail_on_fixed=false.
• Scoped intentional disabling, such as withSQLConf("spark.gluten.enabled" -> "false"), remains supported when the default is restored. Tests intentionally starting with Gluten disabled have explicit, documented exemptions.
• Coverage includes the identified alternative Spark fixture paths; any remaining uncovered Spark-backed suites are reported explicitly.
• Legitimate Spark fallback and tests that execute no native query remain valid. Individual test queries are not required to contain Gluten operators.
• Positive and negative checks exercise the guard and independent gate, including missing instrumentation and intentional opt-outs.

Scope and remaining limitation

Fixture-level validation does not prove that every additional session created inside a test is correctly configured. For example, the session-isolation test in DeltaSuite.scala creates another session with newSession().

The initial implementation should document this limitation and any uncovered session paths. Runtime-level validation of every additional session can be a follow-up; this issue should not be described as providing that stronger guarantee.

Drafted with assistance from GitHub Copilot (GPT-6 Astra).

Gluten version

main branch