Skip to content

[GLUTEN-9003][VL] Add native counter functions for Delta's IncrementMetric - #13357

Open
malinjawi wants to merge 4 commits into
apache:mainfrom
malinjawi:feat/velox-increment-metric-counters
Open

malinjawi wants to merge 4 commits into
apache:mainfrom
malinjawi:feat/velox-increment-metric-counters

Conversation

@malinjawi

@malinjawi malinjawi commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Part of #9003. This is the native half of #13165, split out to make review easier.

Delta's IncrementMetric counts how many rows an expression is evaluated on, and Velox already tracks that per expression (ExprStats::numProcessedRows). This adds the native side of counting it there:

  • 32 fixed counter functions, increment_metric_0 to increment_metric_31. Each one is a pass-through, marked non-deterministic so Velox never constant-folds or dictionary-peels it, and without default null behavior so null inputs still count.
  • For a plan that calls a counter, WholeStageResultIterator forces operator_track_expression_stats=true and expression.eval_simplified=false after the user's overrides (simplified evaluation doesn't count rows), and exports the counters' processed-row counts with the operator metrics.
  • Counters belong in projections only: [GLUTEN-9003][VL] Count Delta's IncrementMetric natively #13165 emits them from Delta's project transformer alone, and Delta's filter transformer strips its counters and credits the filter's output rows. The plan converter fails the plan if a declared counter is not called by any projection, so a misplaced counter can't report a wrong count.

Nothing calls these functions yet, so plans, metric values and the metrics JSON don't change: an operator's metrics get an expressionStats object only when its plan calls a counter. #13358 reads the counts on the JVM side, and #13165 makes the Delta transformers emit the calls.

How was this patch tested?

IncrementMetricFunctionTest: pass-through including nulls; the shapes Delta emits (counters over literals, per-branch counts inside if with the result checked); a dictionary input, which must not be peeled; detecting counters in a plan; rejecting a counter outside a projection; and that the forced configs keep counting when a user sets eval_simplified=true and operator_track_expression_stats=false.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: IBM Bob

@malinjawi

Copy link
Copy Markdown
Contributor Author

cc: @zhouyuan @rui-mo @felipepessoto

…etric

Delta's IncrementMetric counts how many rows an expression is evaluated
on. Velox already keeps that count per expression
(ExprStats::numProcessedRows), so a counter can be a pass-through
function whose name is the key the JVM reads the count from.

- Register 32 fixed counter slots, increment_metric_0 to
  increment_metric_31: non-deterministic, so Velox never constant-folds
  or dictionary-peels them, and without default null behavior, so null
  inputs still count.
- For a plan that calls a counter, force operator_track_expression_stats
  on and expression.eval_simplified off after the user's overrides
  (simplified evaluation applies functions without counting them), and
  export the counters' processed-row counts in the operator metrics.

Nothing calls the counters yet, so plans and metrics are unchanged. The
Delta transformers start emitting them in a follow-up.
@malinjawi
malinjawi force-pushed the feat/velox-increment-metric-counters branch from 84f1bb3 to 438aca7 Compare October 19, 2026 12:05
@zhouyuan
zhouyuan requested a balanced review from Copilot October 23, 2026 14:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues
What changed in this PR

Adds Velox-native support for Delta’s IncrementMetric by introducing fixed-slot counter functions and wiring their per-expression processed-row counts into operator metrics, while forcing the necessary query config knobs when counters are present in the plan.

Changes:

  • Added increment_metric_<slot> (0–31) passthrough vector functions and registration.
  • Added plan detection + query-config forcing to ensure expression stats are tracked and simplified eval is disabled when counters are used.
  • Exported counter processed-row counts into operator metrics JSON under expressionStats; added unit tests.
File Description
cpp/​velox/​operators/​functions/​IncrementMetricFunction.h Declares fixed-slot counter APIs, plan detection, and config overrides.
cpp/​velox/​operators/​functions/​IncrementMetricFunction.cc Implements counter function registration, counter-name detection, plan traversal, and config override helper.
cpp/​velox/​operators/​functions/​RegistrationAllFunctions.cc Registers the new counter functions during startup registration.
cpp/​velox/​compute/​WholeStageResultIterator.h Adds a flag to track whether the plan uses counters (to force configs).
cpp/​velox/​compute/​WholeStageResultIterator.cc Forces configs when counters are detected; exports counter stats into operator metrics JSON.
cpp/​velox/​tests/​IncrementMetricFunctionTest.cc Adds tests for passthrough behavior, counting semantics, plan detection, and config-forcing.
cpp/​velox/​tests/​CMakeLists.txt Adds the new test file to the existing test target.
cpp/​velox/​CMakeLists.txt Adds the new implementation file to the build.

Review effort: Lite

…expressionStats

Review follow-ups:
- isIncrementMetricFunction accepts only increment_metric_0 to
  increment_metric_31, not any name with the prefix.
- An operator's metrics JSON gets an expressionStats key only when it
  has counter stats; the JVM already reads a missing key as empty.
- Rename trackExpressionStats_ to usesIncrementMetricFunctions_, which is
  what it stores.
- Say why only project and filter nodes are inspected: they are the only
  ones Gluten puts counters in.
…s counter shapes

Review follow-ups:
- Counters only appear in projections: the JVM emits them from Delta's
  project transformer alone, and its filter transformer strips its counters
  and credits the filter's output rows. The comments said "project and
  filter"; usesIncrementMetricFunctions now looks at projections only.
- checkIncrementMetricFunctionsInProjects fails the plan conversion when the
  Substrait plan declares a counter that no projection calls. FilterProject
  only reports the counts of its projection, so a counter anywhere else would
  read zero.
- Test the shapes Delta emits: counters over literals, per-branch counts with
  the IF result checked, a dictionary input that must not be peeled, and the
  new check.
- Rename the per-operator JSON object that shadowed the node's stats.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants