[felipepessoto]

What changes are proposed in this pull request?

Fixes #12538.

The Delta statistics tracker assumes that its local statistics plan always becomes a native whole stage. A rejected projection instead leaves a Spark ProjectExec, causing the unchecked cast to fail. Checking only the root is insufficient: disabling aggregation offload leaves a native projection above a Spark aggregate, and the local evaluator then receives raw input batches where it expects aggregate output.

For both Delta 3.3 and Delta 4, select the statistics implementation before constructing the native task. Use native aggregation only when the complete stage is transformable, with the expected statistics-input adapters. Otherwise log the validation reasons and reuse GlutenDeltaJobStatsFallbackTracker, which converts the incoming batches to rows for Delta's original tracker. Native file writing and supported native statistics remain enabled.

Add a shared backend regression suite covering supported native statistics, rejected projections, partially offloaded aggregation, and rejected NTZ statistics. Extend the gluten-ut partitioned-write matrix to cover statistics fallback across file rotation and batch splitting, and update the writer documentation. Tests check actual writer provenance, file contents, row/null counts, nested min/max, all-null columns, and native statistics execution rather than only absence of exceptions.

The existing Delta 4 NTZ file-writer gate is unchanged; this does not enable native NTZ file writing. No dependency pins, DATE-reader patches, CI workflows, or upstream-test baseline entries are changed.

How was this patch tested?

Built fresh native libraries from main commit cce7716962d3152b46fc22882bed4a3887a792ef and its unchanged IBM Velox pin bd5d1828bc100d7c3d83c0fb77ed15a1c2e8b8c4, before building the JVM modules. Native build and runtime used the same apache/gluten:centos-9-jdk17 builder filesystem, GCC toolset 13 and JDK 17.0.20. No historical or reader-patched native binaries were reused.

Red/green: with only the new tests overlaid on unmodified main, GlutenDeltaStatsFallbackSuite ran four tests on Spark 4.1.1 / Delta 4.4.1: the supported-native control passed, while the other three failed. Projection and NTZ rejection reproduced ProjectExec cannot be cast to WholeStageTransformer; aggregation-only rejection reproduced (3 vs. 10) Value stream batch has fewer columns than the declared output type. All four pass with this fix.

Runtime	Suite / selection	Result
Spark 4.1.1 / Delta 4.4.1 / Scala 2.13.17	GlutenDeltaStatsFallbackSuite	4/4 passed
Spark 4.1.1 / Delta 4.4.1 / Scala 2.13.17	GlutenDeltaNativePartitionWriteSuite	5/5 passed
Spark 4.1.1 / Delta 4.4.1 / Scala 2.13.17	Existing VeloxDelta4WriteSuite: TIMESTAMP_NTZ falls back, at any depth	1/1 passed
Spark 3.5.5 / Delta 3.3.2 / Scala 2.12.18	GlutenDeltaStatsFallbackSuite	4/4 passed
Spark 3.5.5 / Delta 3.3.2 / Scala 2.12.18	Existing GlutenDeltaStatsSuite native NTZ statistics test	1/1 passed
Spark 3.5.5 / Delta 3.3.2 / Scala 2.12.18	GlutenDeltaNativePartitionWriteSuite	5/5 passed

The JVM modules were built with ./build/mvn, using the corresponding Spark/Scala profiles plus backends-velox,delta,spark-ut,fast-build; Spark 4.1 also used java-17 and -Dmaven.compiler.release=17. A clean reactor build was used when switching to Spark 3.5. Actual Spark/Delta/Scala runtime versions were checked in the JUnit classpaths. The tested Linux source tree matches the committed tree (21bfaef702a524bca532c37f969a6e922e86ddc0).

Also passed ./dev/format-scala-code.sh, the license-header check (check.py header main --fix), and git diff --cached --check.

Local validation did not rerun the historical Spark 4.1.0 / Delta 4.2.0 DATE-to-NTZ ALTER setup or the entire upstream/Spark-version matrix. The NTZ fallback regression exercises the statistics stream without bypassing Delta 4's file-writer gate.

Completed CI for head 8863b207d27a1128065429edbbef4bed7eea30a6: Velox x86 and ARM, Velox Kafka, ClickHouse, Scala formatting and license checks passed. The Delta 4.4.1 workflow completed all eight shards: 22,100 passed, 471 expected baseline failures, 5,160 skipped, zero regressions, and 14 now-passing baseline entries. The workflow is red only because strict fail-on-fixed enforcement requires removing those 14 stale nested-map MERGE entries; their separate cleanup is already in #13508, and is not duplicated here. These counts were independently reproduced from all downloaded JUnit XML using the exact CI comparator and baseline/quarantine blobs. CI checkout d30bfe2ad14752b499fa6af27eb01b77e922b0a7 has parents cce7716 and this PR head. No reader override was used, and this result is not a claim that the DATE-widening baseline failures were fixed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot (GPT-6 Astra, model ID: gpt-6-astra)

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

🟢 Approval recommended

No active review issues remain for the statistics-fallback change.

No open issues 🎉

What changed in this PR

Addresses #12538 by falling back to Delta’s row-based statistics tracker when the complete statistics plan cannot run natively. This preserves native file writing and supported native statistics without changing Delta 4’s TIMESTAMP_NTZ writer restriction.

Changes:

• Validate the full statistics stage before constructing native tasks in both Delta variants, and log fallback reasons.
• Add regression coverage for supported and rejected statistics plans, plus fallback during partitioned file rotation and batch splitting.
• Update the writer documentation to describe statistics fallback.

File	Description
gluten-ut/​spark35/​src-delta/​test/​scala/​org/​apache/​spark/​sql/​execution/​datasources/​GlutenDeltaNativePartitionWriteSuite.scala	Adds partitioned-write cases with statistics offload disabled.
docs/​backends/​velox/​get-started/​VeloxDelta.md	Documents statistics fallback and removes the resolved crash limitation.
backends-velox/​src-delta4/​main/​scala/​org/​apache/​spark/​sql/​delta/​stats/​GlutenDeltaJobStatsTracker.scala	Selects native or fallback statistics after full-stage validation for Delta 4.
backends-velox/​src-delta33/​test/​scala/​org/​apache/​spark/​sql/​delta/​GlutenDeltaStatsFallbackSuite.scala	Checks native writer provenance, statistics values, and rejected-plan fallback.
backends-velox/​src-delta33/​main/​scala/​org/​apache/​spark/​sql/​delta/​stats/​GlutenDeltaJobStatsTracker.scala	Applies the same statistics-stage validation and fallback for Delta 3.3.

_{Review effort: Balanced}