Repository navigation
[build] Add a local development kit for contributors - #3851
Conversation
de731d7 to
ca4f94d
Compare
|
Hi @luoyuxia and @fresh-borzoni, I opened this PR based on my experience getting started with Fluss. I’d appreciate your thoughts on the overall direction and scope when you have time. Thanks! |
|
I recommend trying the DevKit introduced by this PR locally. Compared with starting tests from IntelliJ or building Docker images and assembling a custom Docker Compose setup, it provides a more convenient workflow and makes it easier for both developers and coding agents to launch Fluss for end-to-end testing and troubleshooting. I have frequently used a similar approach in the AutoMQ DevKit, and found it helpful for local development. I believe it could provide a similar improvement to the Fluss development experience. |
|
Hi @wuchong @luoyuxia @swuferhong, this PR adds a local DevKit for Fluss contributors. It is a development tooling contribution rather than a user-facing feature, so I’d appreciate your thoughts on whether it fits the project and any feedback when you have time. If it is not a good fit for the repository or the current scope, please feel free to let me know. Thanks! |
fresh-borzoni
left a comment
fresh-borzoni
left a comment
There was a problem hiding this comment.
@Gezi-lzq Thank you for the PR and sorry for the long turnaround.
I think it can be useful for the project overall.
First, nothing tests the distribution as shipped. ITCases build a Configuration in Java and start servers in-process, so server.yaml parsing, the bin/ scripts and the plugins/<format>/ classloaders never run. Only the server loads lake plugins through a real PluginManager, but LakeTieringJobBuilder, LakeSourceUtils and every test pass null and use the flat classpath.
Second, nothing tests against a remote filesystem. That's how HoodieFlinkEngineContext.DEFAULT slipped through as tiering ITCases write to a local temp dir where an empty Hadoop conf works fine.
Your S3-backed profile is what exposed it.
Right now the only place Fluss runs for real is someone's laptop, set up
by hand, which is why wrong config can sit in our docs for months. If this lands, the quickstart pages can point at it instead of carrying their own compose blocks, so what we document is what we run. After that, a new lake format or filesystem plugin ships with a profile and its packaging gets checked when it's added. And it gives the cross-process things: TLS listeners, the k8s health contract - somewhere to be run properly, which a single-JVM ITCase can't do.
+1 on the CI smoke tests you mentioned, shall we do them in this PR rather than later, otherwise breakage stays silent until someone tries to run it?
Also devkit/** isn't in our paths-ignore, so a compose-only edit runs the whole Java matrix today.
Should this live under docker/? docker/quickstart-flink/ already has init_paimon.sh /init_iceberg.sh and a sample SQL file.
I'm thinking we keep this PR to the devkit and split out the HudiWriteTableInfo change to fix this issue. WDYT?
|
Sorry for the delayed update—I only got a chance to update the PR today. Thanks for the detailed feedback and support. |
|
Hi @beryllw, I noticed the recent work on the Gateway lakehouse quickstart. #3851 also includes a contributor-focused Gateway scenario that builds and runs Gateway from the current checkout against a real Fluss cluster, covering readiness, metadata access, REST writes, and configuration or integration changes. |
fresh-borzoni
left a comment
fresh-borzoni
left a comment
There was a problem hiding this comment.
@Gezi-lzq Thank you, good changes, run the devkit, LGTM overall, collected some comments and question, PTAL
|
Hi @fresh-borzoni, apologies for the delayed follow-up, and thank you again for the detailed review. I’ve addressed the points you raised and pushed the changes to the PR. Please take another look when convenient. |
wuchong
left a comment
wuchong
left a comment
There was a problem hiding this comment.
Could you please rebase this branch onto the latest main instead of merging main into it? Keeping the history linear will make this PR easier to review and merge.
wuchong
left a comment
wuchong
left a comment
There was a problem hiding this comment.
Thanks for putting this together. I see real value in this PR. Beyond making local development and validation easier, what I value most is the foundation it provides for E2E testing suite in CI, an important gap in Fluss’s current test coverage.
Add a Docker Compose and just based development kit that runs locally built Fluss artifacts and supports core, Iceberg, Paimon, Lance, and Hudi profiles with manual tiering validation commands. Run Fluss processes under Docker init, enable JDWP and Prometheus metrics, and propagate configured Hadoop settings to the Hudi write client so the S3A-backed Hudi profile retains its endpoint and credentials.
Stage profile-declared dependencies and Compose overlays for both lake and non-lake profiles. Keep Server JAR cleanup ownership explicit, align local runtime versions, and document the configuration rationale and validation boundaries.
Markdown files are excluded by the repository RAT configuration, consistent with the DevKit README.
Keep the PR focused on the implementation and concise usage documentation, and remove the README link to the deleted design note.
DevKit changes do not affect Java sources, so exclude them from the full Java CI workflow while retaining the separate license check.
Add core SQL examples and DevKit smoke coverage, tune snapshot and tiering polling intervals, and remove the temporary Hudi profile.
Keep lake examples under devkit/examples and remove the format-specific shortcut recipes so the workflow is explicit and easier to explore. Update the smoke workflow to execute the Paimon SQL chain directly.
Compile the Gateway in its pinned Rust environment, mount the local executable into a lightweight runtime container, and validate REST connectivity in the DevKit smoke workflow.
Replace the separate build recipes with one explicit core, tiering, or gateway target and keep startup recipes focused on starting prebuilt artifacts.
Declare runtime components per profile so build and startup dependencies stay aligned. Load Flink and Gateway overlays from runtime targets, mount the standard Gateway configuration, and document how contributors tailor complete local test scenarios.
Align managed Paimon dependencies with the root POM, strengthen Gateway and Paimon smoke coverage, and clarify the profile-driven build and runtime lifecycle. Keep startup offline after an explicit build, validate readiness and the active Tiering Job, and remove the unrelated Hudi behavior change.
Run the DevKit smoke workflow only when files under devkit change.
Run the Paimon SQL examples directly without parsing or polling their result output.
Derive build steps from runtime targets and remove the duplicate build target files. Use the active Compose profile for runtime checks, rely on Compose healthchecks for Flink readiness, and remove unused build command overrides.
Document profiles as complete local test scenarios and define runtime.targets as the single declaration for built-in runtime groups.\n\nValidate target names, duplicates, and dependencies before emitting targets or downloading profile dependencies.
Lead contributors from choosing a test scenario through validation, iteration, and diagnostics before introducing the profile contract as reference material.
Replace the unavailable minio/mc image with rustfs/rc:v0.1.36 and use rc for the shared lake profile bucket initialization. AI-assisted with OpenAI Codex.
Keep profile build and runtime state explicit, preserve command arguments, verify downloaded artifacts, stream SQL output, and bind development ports to localhost.
Use supported scan modes, make lookup output observable, model Lance tiering with embedding events, and document each profile's user journey.
Exercise one minimal closed loop per profile, keep lifecycle recovery out of smoke coverage, and run the expensive workflow only for DevKit changes.
4afc93c to
215f664
Compare
Use unittest selection for profile smoke checks and reuse just for setup and cleanup. Keep CI steps named by profile and assert observable results in each case.
Share the RustFS Compose overlay across all profiles and stage the filesystem plugin for Flink in common profile setup. This lets the core smoke scan read from its S3 remote data directory.
Check full Core rows, Gateway column metadata, and Paimon union output. Keep Lance data checks active when Python optimization is enabled.
Thanks, I agree that this could be a useful foundation for E2E testing in CI. The current smoke tests already run a few small end-to-end flows against real services and check their results. I haven’t yet worked out how to structure a broader E2E suite or systematically cover Fluss’s different features. I think these profiles and smoke cases are a good starting point, and we can build on them as the testing needs become clearer. I also expect DevKit to be useful in day-to-day development for starting Fluss, reproducing issues, and validating changes locally. |
Purpose
Closes #3850.
Add a contributor-oriented DevKit for building the current checkout and running repeatable local end-to-end scenarios without building a custom Fluss image.
Brief change log
justenvironment for Core, Gateway, Paimon, Iceberg, and Lance scenarios with one or three TabletServersjust build <profile>compiles and downloads dependencies, whilejust up <profile>only stages, starts, and waits for prepared artifactsTests
rustfs/rc:v0.1.36; actual DevKit Compose initialization and repeated bucket creation passed againstrustfs/rustfs:1.0.0-alpha.83;git diff --checkand./mvnw -N spotless:check validatepassed./mvnw validatejust --fmt --checkactionlint .github/workflows/devkit-smoke.yamltiering-statussuccess and its stopped-cluster/no-active-profile failure pathsFull
mvn verifywas not run locally; this change is covered by Maven validation and the DevKit end-to-end checks above.API and Format
No public API or storage format changes.
Documentation
Add
devkit/README.mddefining the profile andruntime.targetscontracts and covering prerequisites, build/start responsibilities, example workflows, lifecycle operations, and endpoints.