Skip to content

[Umbrella][rust] Consumer offset management and language bindings #5198

Open

Description

@naivedogger

Search before asking

Description

Add consumer offset management to the Rust client and expose it through the existing C++, Python, and Elixir bindings. This is the multi-language client work associated with #4608, covering manually assigned consumers. Group membership and automatic rebalancing are outside this scope.

The implementation is planned as five PRs:

  • 1. [rust] Add explicit consumer offset commits and restore ([rust] Add explicit consumer offset commits and restore #5200, PR [rust] Add explicit consumer offset commits and restore #5201). Implement internal owner discovery, caching, rediscovery and routing retries; synchronous/asynchronous commits with per-bucket results and metadata; committed-offset initializers with missing-offset fallbacks; and Admin queries for selected buckets across tables. Integrate record and Arrow scanners. Align authorization errors and protocol behavior with the server, preserve same-scanner commit ordering, and handle uncertain write outcomes explicitly.
  • 2. [rust] Add automatic consumer offset commits. Add opt-in periodic commits with documented position, failure and shutdown semantics. Keep automatic commits disabled by default and preserve explicit commit behavior.
  • 3. [rust] Add multi-table consumer group scanning. Provide the required multi-table scanner support and integrate a shared group ID, committed starting positions, and cross-table commits with per-bucket failure handling. Cross-table Admin queries alone do not satisfy this task.
  • 4. [rust] Add consumer group Admin operations. Support listing groups and all saved offsets, altering/resetting offsets, and deleting offsets/groups as the corresponding server APIs become available.
  • 5. [c++][python][elixir] Expose consumer offset support in the language bindings. Expose the Rust functionality through the existing bindings with language-appropriate asynchronous results, error propagation and resource lifetimes, plus documentation and integration tests for each language.

Each PR includes its own tests and documentation. The base Rust PR should validate restore, partial failures, metadata semantics, owner failover and routing retries against the merged server implementation. Authorization scenarios must be validated against the applicable group ACL implementation (#5073).

PRs 2–4 build on PR 1 and can progress independently where server support permits. PR 4 depends on the remaining server-side management work in #4608. PR 5 follows the Rust API stabilization; it reuses the Rust implementation rather than duplicating offset-management logic in each binding.

Relevant server work: owner discovery #4716, commits #4850, fetches #5138, and group ACLs #5073. Completion of this issue does not depend on automatic rebalancing.

Willingness to contribute

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions