[felipepessoto]

What changes are proposed in this pull request?

Fixes #13500.

Add Velox native-write support for Delta's Materialize Partition Columns writer feature in the Spark 4.x / Delta 4 integration. This PR remains a draft pending refreshed CI; the focused native integration validation on Spark 4.1 passes.

Scope: default Delta 4.4 partitioned writes, not only feature-enabled tables. Delta 4.4's writer defaults to physically writing partition columns into Parquet when delta.writePartitionColumnsToParquet is unset (fromMetaData(metadata).getOrElse(true)). This behavior does not require the materializePartitionColumns protocol feature. That separate writer feature makes materialization mandatory even when the property is explicitly false.

Before this PR, the native-write gate rejected every write reaching it with both partition columns and the resolved writePartitionColumns=true. Under the default settings, those partitioned file writes therefore used Delta's JVM writer, even with spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true. Reads and other query operators could still run in Velox; this was not necessarily whole-query fallback. Explicitly setting the property to false could avoid this gate when no table feature required materialization.

The full Delta CI configuration enables native writes but does not globally override that default to false. Consequently, passing partitioned-table tests after the Delta 4.4 upgrade did not, by themselves, demonstrate native file-write coverage. A GlutenDelta command in the plan is also insufficient proof. This PR enables eligible default partitioned writes to execute natively again, which explains why existing native limitations reappear and why the added coverage verifies physical Parquet schemas, values, and writer identity.

Delta already resolves feature/property precedence and includes partition columns in the requested file schema. The native splitter previously discarded those columns regardless of that decision.

• Forward the existing reservePartitionColumns argument through the Scala/Java/JNI boundary. Retain partition columns in the native output batch after the regular data columns, matching Delta's file schema even when input columns or partition keys have a different order.

• Give statistics trackers a separate data-only batch, retaining their existing non-partition schema and excluding partition columns from Delta file statistics. Preserve batch ownership and per-file slicing.

• Replace the blanket materialization fallback with targeted partition-type restrictions. Keep column mapping, Iceberg compatibility, and the other existing safety fallbacks. FLOAT/DOUBLE partition columns fall back explicitly; materialized TIMESTAMP partition columns remain on Delta's writer because the native splitter cannot represent sub-millisecond values.

• Cover feature/property precedence, native Parquet schemas and values, null/empty-string and multiple partition keys, file splitting and statistics, mixed old/new files, and UPDATE/DELETE parity with vanilla Delta. Let the existing Delta 4 native-write suite exercise Delta's materialization default and update the support documentation.

• Restore ten previously documented Delta native-write failures removed by e920e70e4 when partitioned writes fell back to Delta. Materialization re-exposes existing JNI exception-loss and custom-filesystem limitations; this restoration does not change production behavior or weaken test assertions.

This does not enable full UniForm support or change the Delta 3.3 writer's materialization behavior.

How was this patch tested?

Validated commit 1ef879fd7885b6733fe0ec88cc26b20b993eaec1 in a local WSL Docker DevContainer using apache/gluten:centos-9-jdk17, a Linux-backed workspace volume, GCC 12, JDK 17, Spark 4.1, and Scala 2.13. The container follows the repository's dynamic-link DevContainer setup (VELOX_BUILD_SHARED=ON) with isolated Maven and ccache volumes.

• Built the pinned Velox dependency and this branch's libgluten.so and libvelox.so, including the modified JNI entry point.
• Built and installed main/test classes for backends-velox and gluten-ut/spark41 against those native libraries.
• 54 tests passed; no failures, aborted suites, canceled, ignored, or pending tests.

Suite	Passed
org.apache.gluten.delta.GlutenDeltaNativeWriteGateSuite	19
org.apache.gluten.execution.VeloxDelta4WriteSuite	17
org.apache.spark.sql.delta.DeltaNativeWriteSuite	12
org.apache.spark.sql.execution.datasources.GlutenDeltaNativePartitionWriteSuite	6

The integration cases check physical Parquet schemas/values and writer identity, not just correctness through fallback. They cover feature/property precedence, null/empty-string and reordered multiple partition keys, exact file splitting and statistics, mixed old/new files, UPDATE/DELETE parity with vanilla Delta, and retained column-mapping fallback.

Commands run inside the container:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
export PATH="/opt/rh/gcc-toolset-12/root/usr/bin:$JAVA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="/opt/rh/gcc-toolset-12/root/usr/lib64:/opt/rh/gcc-toolset-12/root/usr/lib:${LD_LIBRARY_PATH:-}"
export VELOX_BUILD_SHARED=ON

./dev/builddeps-veloxbe.sh --run_setup_script=OFF --build_arrow=OFF \
  --num_threads=8 --enable_s3=OFF --enable_gcs=OFF --enable_hdfs=OFF --enable_abfs=OFF

PROFILES="-Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta -Pspark-ut -Pfast-build -Dmaven.compiler.release=17"
./build/mvn -B -ntp $PROFILES -DskipTests \
  -pl backends-velox,gluten-ut/spark41 -am install
./build/mvn -B -ntp $PROFILES -pl backends-velox \
  -DargLine=-Dspark.test.home=/opt/shims/spark41/spark_home/ \
  -Dsuites=org.apache.gluten.delta.GlutenDeltaNativeWriteGateSuite,org.apache.gluten.execution.VeloxDelta4WriteSuite,org.apache.spark.sql.delta.DeltaNativeWriteSuite \
  scalatest:test
./build/mvn -B -ntp $PROFILES -pl gluten-ut/spark41 \
  -DargLine=-Dspark.test.home=/opt/shims/spark41/spark_home/ \
  -Dsuites=org.apache.spark.sql.execution.datasources.GlutenDeltaNativePartitionWriteSuite \
  scalatest:test

dev/format-scala-code.sh and its check mode passed with an affected-file selector; dev/format-cpp-code.sh ran with clang-format 15.0.7; the repository license-header check and git diff --check passed before committing. Other Spark versions and the full repository test suites were not run in this validation.

CI triage and known-failure restoration

Compared this PR's eight-shard Delta run against the successful nightly on its exact parent commit. The only ten newly failing tests are exact entries removed by e920e70e453fe9d035e84fd75b635ba03324f387:

• Eight invalid replaceWhere / replaceArbitrary should enforce proper usage of backtick cases across DeltaSuite and the three catalog-owned variants. JNI converts the original DeltaInvariantViolationException to a message-only GlutenException, so Delta cannot restore the expected AnalysisException. Original Java throwable preservation remains tracked separately in [CORE] Preserve original Java exceptions across JNI #13133.
• Two optimize - multiple jobs start executing at once cases in OptimizeCompactionSQLSuite and OptimizeCompactionScalaSuite. Their custom block: Hadoop filesystem is not supported by the native writer; see Hadoop file system options doesn't work #4963.

Restored only those ten exact entries in known-failures.txt. The provenance and reasons are documented here; remove the entries when the underlying JNI and custom-filesystem limitations are resolved. No production code, test assertions, flaky quarantines, or gate settings changed. The underlying failures are not fixed by this baseline update.

Replayed the unchanged compare-test-results.py enforce gate against the downloaded JUnit XML from all eight shards, keeping --fail-on-fixed true: 22,571 tests ran, 22,076 passed, 495 expected failures, 0 unexpected failures, and 0 now-passing entries. Aggregate mode with --expected-shards 8 --fail-on-regression true also passes with no stale entries. This is a gate replay on existing CI evidence, not a new execution of the full Delta suite. The restored baseline has no duplicate entries and matches the observed aggregate exactly. License-header and whitespace checks passed.

The separate build-cudf-centos-9 failure reports that its hosted runner lost communication. No completed compiler failure/log was available, and retrying the failed workflow was denied with Must have admin rights to Repository. The new push can trigger fresh CI; a maintainer may need to retry the interrupted GPU job if it remains blocked.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot 1.0.102-0.unstable.r40141230562.g130d689 (GPT-6 Astra / gpt-6-astra)

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86