Repository navigation
[VL] Offload Kafka micro-batch scans - #13237
Open
ReemaAlzaid wants to merge 9 commits into
Open
ReemaAlzaid wants to merge 9 commits into
ReemaAlzaid wants to merge 9 commits into
Conversation
Split from the Kafka draft in apache#13130. Co-authored-by: Reema Alzaid <reema-alzaid@outlook.com>
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unsupported Java authentication settings can reach the native Kafka reader and cause valid authenticated streams to fail instead of falling back to Spark.
1 open issue
What changed in this PR
This PR connects the existing Kafka micro-batch scan transformer to the Velox backend, adding schema validation and Spark fallback for unsupported columns such as headers.
Changes:
- Register Kafka offloading through an optional Velox component and Maven profile.
- Validate native Kafka column names and types.
- Add schema and streaming tests, and enable them in Kafka CI.
| File | Description |
|---|---|
gluten-ut/spark35/src/test/backends-velox/org/apache/gluten/execution/VeloxKafkaScanValidationSuite.scala |
Tests supported and unsupported Kafka schemas. |
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala |
Adds Kafka column validation. |
backends-velox/src-kafka/test/scala/org/apache/gluten/execution/kafka/VeloxGlutenKafkaScanSuite.scala |
Tests native reads and headers fallback. |
backends-velox/src-kafka/main/scala/org/apache/gluten/component/VeloxKafkaComponent.scala |
Registers offloading when Spark Kafka classes are available. |
backends-velox/src-kafka/main/resources/META-INF/gluten-components/org.apache.gluten.component.VeloxKafkaComponent |
Enables component discovery. |
backends-velox/pom.xml |
Adds optional Kafka dependencies. |
.github/workflows/velox_kafka.yml |
Enables schema and streaming test steps. |
Review effort: Balanced
| } | ||
|
|
||
| override def injectRules(injector: Injector): Unit = { | ||
| OffloadKafkaScan.inject(injector) |
|
Run Gluten Clickhouse CI on x86 |
zhouyuan
reviewed
Oct 27, 2026
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Enable native Kafka micro batch scans, with Spark fallback for
unsupported columns such as headers. Split from #13130.
How was this patch tested?
Added schema validation and streaming tests for native reads
and headers fallback.
Tested on Linux aarch64 with Spark 3.5.5, Scala 2.12.18, and Java 17,
with Kafka and Iceberg profiles enabled.
schemas and rejection of unsupported columns/types.
micro batch reads and headers fallback, each returning all
1,000 expected records.
Both ScalaTest suites completed without failures or aborted suites.
Both Maven invocations reported BUILD SUCCESS.
Also passed a separate
spark-submitsmoke test. The nativequery contained one MicroBatchScanExecTransformer; the headers query
used Spark fallback with no native Kafka scan.