Repository navigation
Description
Search before asking
- I searched existing issues. FIP-53: Log Scanner Offset(Consumer Offsets)Management #4608 tracks the server and Java client work; this issue tracks the Rust client and its language bindings.
Description
Add consumer offset management to the Rust client and expose it through the existing C++, Python, and Elixir bindings. This is the multi-language client work associated with #4608, covering manually assigned consumers. Group membership and automatic rebalancing are outside this scope.
The implementation is planned as five PRs:
- 1. [rust] Add explicit consumer offset commits and restore ([rust] Add explicit consumer offset commits and restore #5200, PR [rust] Add explicit consumer offset commits and restore #5201). Implement internal owner discovery, caching, rediscovery and routing retries; synchronous/asynchronous commits with per-bucket results and metadata; committed-offset initializers with missing-offset fallbacks; and Admin queries for selected buckets across tables. Integrate record and Arrow scanners. Align authorization errors and protocol behavior with the server, preserve same-scanner commit ordering, and handle uncertain write outcomes explicitly.
- 2. [rust] Add automatic consumer offset commits. Add opt-in periodic commits with documented position, failure and shutdown semantics. Keep automatic commits disabled by default and preserve explicit commit behavior.
- 3. [rust] Add multi-table consumer group scanning. Provide the required multi-table scanner support and integrate a shared group ID, committed starting positions, and cross-table commits with per-bucket failure handling. Cross-table Admin queries alone do not satisfy this task.
- 4. [rust] Add consumer group Admin operations. Support listing groups and all saved offsets, altering/resetting offsets, and deleting offsets/groups as the corresponding server APIs become available.
- 5. [c++][python][elixir] Expose consumer offset support in the language bindings. Expose the Rust functionality through the existing bindings with language-appropriate asynchronous results, error propagation and resource lifetimes, plus documentation and integration tests for each language.
Each PR includes its own tests and documentation. The base Rust PR should validate restore, partial failures, metadata semantics, owner failover and routing retries against the merged server implementation. Authorization scenarios must be validated against the applicable group ACL implementation (#5073).
PRs 2–4 build on PR 1 and can progress independently where server support permits. PR 4 depends on the remaining server-side management work in #4608. PR 5 follows the Rust API stabilization; it reuses the Rust implementation rather than duplicating offset-management logic in each binding.
Relevant server work: owner discovery #4716, commits #4850, fetches #5138, and group ACLs #5073. Completion of this issue does not depend on automatic rebalancing.
Willingness to contribute
- I'm willing to submit a PR!
Activity