Repository navigation
[GLUTEN-13471][FLINK] Add support for TO_DATE / TO_TIMESTAMP / TO_TIMESTAMP_LTZ - #13473
TianJialong wants to merge 18 commits into
Conversation
|
@KevinyhZou @lgbo-ustc @philo-he @ggjh-159 Would you mind reviewing this PR? |
lgbo-ustc
left a comment
There was a problem hiding this comment.
This PR depends on bigo-sg/velox#94, but the Flink build still pins a Velox4j revision whose Velox dependency predates those functions. Please update the Velox and Velox4j references so the new offload mappings are built and tested against a version that contains the required implementations.
|
@TianJialong, thanks for the PR. Could you rebase the code and fix conflicts? The latest Gluten Flink uses the following file to maintain the referenced velox4j version. |
| export fmt_SOURCE=BUNDLED | ||
| export folly_SOURCE=BUNDLED | ||
| git clone -b gluten-20260829 https://github.com/bigo-sg/velox4j.git | ||
| git clone -b gluten-20260829 https://github.com/TianJialong/velox4j.git |
There was a problem hiding this comment.
Do we need to revert this before merging?
There was a problem hiding this comment.
@philo-he Yes, we need. But PRs in velox and velox4j haven't merged yet. I'll revert it after they merge.
There was a problem hiding this comment.
I see. Let's mark this PR as a draft to indicate it is not ready to merge. Thanks!
|
@TianJialong please update velox4j version |
|
philo-he
left a comment
There was a problem hiding this comment.
Looks good. Thanks! Pending @lgbo-ustc's review.
What changes are proposed in this pull request?
Fixes #13471
Maps the temporal string/number conversion expressions to their velox flinksql implementations (companion velox PR bigo-sg/velox#94, velox4j PR bigo-sg/velox4j#75):
RexCallConverterFactory:TO_DATE/TO_TIMESTAMP/TO_TIMESTAMP_LTZentries — plain name mappings; both arities of the first two are covered by the flinksql signatures,TO_TIMESTAMP_LTZ's numeric argument types bind through the same registration.RexNodeConverter:DATEliterals convert to the epoch-dayIntegerValue— Flink folds constantTO_DATEcalls in the planner, and the plan used to die at conversion with the DATE literal unsupported.TIMESTAMP_WITH_LOCAL_TIME_ZONEliterals share the TIMESTAMP conversion body (TimestampString.getMillisSinceEpoch→ boxed-Longmillis, with a null guard), so foldedTO_TIMESTAMP/TO_TIMESTAMP_LTZliterals reach velox. NULL literal branches emit a null variant (a velox4j serialization limitation currently drops typed NULL literals at the bridge; NULL rows arrive from data inputs instead).RexNodeConverter: identity-cast elimination — the planner emitsCAST(literal AS T)around every folded literal whose nullability differs from the sink's, and velox rejects same-type casts, soisSameTypeIgnoringNullability(recursive type equality ignoring nullability only; ANY never elided; composite component types must match) passes the operand through unchanged.RexNodeConverterTest: covers the two literal channels, the identity-cast behavior across scalar and composite types, and the NULL literal branches.How was this patch tested?
RexNodeConverterTest— DATE literal converts to the epoch-dayIntegerValue, LTZ literal to epoch millis; identity CAST elided on nullability-only casts for scalar and composite types (ARRAY/ROW/MAP casts with different component types stay real casts); NULL literal branches.DateTimeFunctionsTestinorg.apache.gluten.table.runtime.stream.custom— both arities ofto_date(incl. constant-format folding through the DATE literal channel),to_timestamp(SMART clamping,24:00rollover, fraction flooring, column/constant formats),to_timestamp_ltz(precision 0/3 unit semantics, epoch-domain bounds, double/decimal inputs), NULL propagation on all forms, and CASE/WHERE combinations.