Skip to content

[GLUTEN-13471][FLINK] Add support for TO_DATE / TO_TIMESTAMP / TO_TIMESTAMP_LTZ - #13473

Open
TianJialong wants to merge 18 commits into
apache:mainfrom
TianJialong:main
Open

TianJialong wants to merge 18 commits into
apache:mainfrom
TianJialong:main

Conversation

@TianJialong

@TianJialong TianJialong commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Fixes #13471

Maps the temporal string/number conversion expressions to their velox flinksql implementations (companion velox PR bigo-sg/velox#94, velox4j PR bigo-sg/velox4j#75):

  • RexCallConverterFactory: TO_DATE / TO_TIMESTAMP / TO_TIMESTAMP_LTZ entries — plain name mappings; both arities of the first two are covered by the flinksql signatures, TO_TIMESTAMP_LTZ's numeric argument types bind through the same registration.
  • RexNodeConverter: DATE literals convert to the epoch-day IntegerValue — Flink folds constant TO_DATE calls in the planner, and the plan used to die at conversion with the DATE literal unsupported. TIMESTAMP_WITH_LOCAL_TIME_ZONE literals share the TIMESTAMP conversion body (TimestampString.getMillisSinceEpoch → boxed-Long millis, with a null guard), so folded TO_TIMESTAMP / TO_TIMESTAMP_LTZ literals reach velox. NULL literal branches emit a null variant (a velox4j serialization limitation currently drops typed NULL literals at the bridge; NULL rows arrive from data inputs instead).
  • RexNodeConverter: identity-cast elimination — the planner emits CAST(literal AS T) around every folded literal whose nullability differs from the sink's, and velox rejects same-type casts, so isSameTypeIgnoringNullability (recursive type equality ignoring nullability only; ANY never elided; composite component types must match) passes the operand through unchanged.
  • RexNodeConverterTest: covers the two literal channels, the identity-cast behavior across scalar and composite types, and the NULL literal branches.

How was this patch tested?

  • Planner-level: RexNodeConverterTest — DATE literal converts to the epoch-day IntegerValue, LTZ literal to epoch millis; identity CAST elided on nullability-only casts for scalar and composite types (ARRAY/ROW/MAP casts with different component types stay real casts); NULL literal branches.
  • Execution-level (embedded gluten stack): DateTimeFunctionsTest in org.apache.gluten.table.runtime.stream.custom — both arities of to_date (incl. constant-format folding through the DATE literal channel), to_timestamp (SMART clamping, 24:00 rollover, fraction flooring, column/constant formats), to_timestamp_ltz (precision 0/3 unit semantics, epoch-domain bounds, double/decimal inputs), NULL propagation on all forms, and CASE/WHERE combinations.
  • End-to-end dual-run on Flink 1.19.2: 32 SQL scenarios across the family — both arities, constant vs column formats, NULL propagation, literal folding for DATE and LTZ, invalid-input and out-of-range classes — with 29 exact matches and 3 shared job-failure classes (illegal format, year overflow, unsupported LTZ precision: identical failures on both stacks), zero mismatches.

@github-actions github-actions Bot added the FLINK label Oct 27, 2026
@TianJialong

Copy link
Copy Markdown
Contributor Author

@KevinyhZou @lgbo-ustc @philo-he @ggjh-159 Would you mind reviewing this PR?

@lgbo-ustc lgbo-ustc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR depends on bigo-sg/velox#94, but the Flink build still pins a Velox4j revision whose Velox dependency predates those functions. Please update the Velox and Velox4j references so the new offload mappings are built and tested against a version that contains the required implementations.

@github-actions github-actions Bot added the INFRA label Oct 29, 2026
@TianJialong
TianJialong requested a review from lgbo-ustc October 29, 2026 03:46
@philo-he

philo-he commented Nov 5, 2026

Copy link
Copy Markdown
Member

@TianJialong, thanks for the PR. Could you rebase the code and fix conflicts?

The latest Gluten Flink uses the following file to maintain the referenced velox4j version.
https://github.com/apache/gluten/blob/main/gluten-flink/velox4j-ref.txt

Comment thread .github/workflows/flink.yml Outdated
export fmt_SOURCE=BUNDLED
export folly_SOURCE=BUNDLED
git clone -b gluten-20260829 https://github.com/bigo-sg/velox4j.git
git clone -b gluten-20260829 https://github.com/TianJialong/velox4j.git

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to revert this before merging?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@philo-he Yes, we need. But PRs in velox and velox4j haven't merged yet. I'll revert it after they merge.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Let's mark this PR as a draft to indicate it is not ready to merge. Thanks!

@philo-he
philo-he marked this pull request as draft November 9, 2026 08:01
@ggjh-159

Copy link
Copy Markdown
Contributor

@TianJialong please update velox4j version

@github-actions github-actions Bot removed the INFRA label Nov 11, 2026
@TianJialong
TianJialong marked this pull request as ready for review November 11, 2026 04:53
@TianJialong

TianJialong commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

@TianJialong please update velox4j version

@philo-he @ggjh-159 All references are updated.

@philo-he philo-he changed the title [GLUTEN-13471][FLINK] Support TO_DATE / TO_TIMESTAMP / TO_TIMESTAMP_LTZ offload to velox flinksql functions [GLUTEN-13471][FLINK] Add support for TO_DATE / TO_TIMESTAMP / TO_TIMESTAMP_LTZ Nov 11, 2026
Comment thread .gitignore Resolved

@philo-he philo-he left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks! Pending @lgbo-ustc's review.

@TianJialong TianJialong reopened this Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FLINK][VL] Support the TO_DATE / TO_TIMESTAMP / TO_TIMESTAMP_LTZ temporal expressions

4 participants