Skip to content

[lake] Fail fast on duplicate lake format plugins - #4916

Draft
Hachiman566 wants to merge 1 commit into
apache:mainfrom
Hachiman566:fix/lake-format-duplicates-review-20261027
Draft

Hachiman566 wants to merge 1 commit into
apache:mainfrom
Hachiman566:fix/lake-format-duplicates-review-20261027

Conversation

@Hachiman566

Copy link
Copy Markdown

Generative AI disclosure: Yes — OpenAI Codex. Human review is pending.

Purpose

Linked issue: close #4601

When two lake plugins register the same format, LakeStoragePluginSetUp selects the first match. The result depends on discovery order. This change fails on the second match and reports the requested format and both implementation classes, following ConfigProviders.

Brief change log

  • Keep the existing PluginManager-then-SPI discovery order and fail as soon as a second plugin matches the requested format.
  • Keep PluginLakeStorageWrapper for a single match. Keep the existing exception type and message when no plugin matches.
  • Cover duplicates within PluginManager, within SPI, across both sources, and repeated discovery of the same implementation, plus unique/no-match behavior and immediate failure.
  • Add a tests-without-lake-spi server test jar for Paimon and Spark UT. It excludes the test SPI registration that conflicts with the real Paimon plugin and retains all other entries. The default server tests.jar remains available to other tests.

Review question: This change also rejects the same implementation discovered through both PluginManager and SPI. It does not deduplicate by class name because separate classloaders may load different versions with the same name. Should this case also count as a conflict? This behavior still needs maintainer confirmation.

Tests

Validated on main 8a92b4fd8 with Java 11 and Maven 3.8.6, using -Dskip.iceberg-1.12.

  • Common: 1929 unit tests (0 failures/errors, 1 existing JRE-conditioned skip) and 186 parser tests passed. The 12 targeted setup/wrapper tests passed; removing the fix caused 5 expected assertion failures, and restoring it made all 12 pass.
  • Paimon: LakeEnabledTableCreateITCase#testCreateLakeEnabledTable passed (1 integration test).
  • Spark: the latest reactor, including Spark tiering and Iceberg common, built successfully. SparkLakePaimonCatalogTest passed all 13 tests, with no failures or aborted suites.
  • SPI isolation: the dedicated server test jar removes only the lake SPI registration (461 → 460 entries); all retained contents match the default jar. The actual Spark test classpath discovers one real Paimon provider, and lookup returns PluginLakeStorageWrapper.
  • Checkstyle, Spotless, RAT and git diff --check passed.

Not run: the full repository matrix, full Paimon/Spark suites beyond these targets, or JDK 8/JDK 17 runs, including Iceberg 1.12.

API and Format

No public signatures or stored formats change. Duplicate requested-format registrations now fail instead of silently selecting the first implementation.

Documentation

No new configuration options or setup steps are introduced. No documentation changes are included.

@polyzos

polyzos commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

@Hachiman566 let me know once ready for review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[lake] Fail fast when two plugins register the same lake format

2 participants