Skip to content

fix(serializer): stop rendering meta.language as prose; emit lang attributes in HTML - #746

Open
wittjeff wants to merge 2 commits into
docling-project:mainfrom
wittjeff:fix/language-meta-serializers
Open

wittjeff wants to merge 2 commits into
docling-project:mainfrom
wittjeff:fix/language-meta-serializers

Conversation

@wittjeff

@wittjeff wittjeff commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Fixes #745.

What changed

Markdown / plain text. MarkdownMetaSerializer._serialize_meta_field had no branch for LanguageMetaField, so it fell through to str(field_val) and the export gained a paragraph like confidence=0.99 created_by='langdetect' code=<HumanLanguageLabel.DE: 'de'>. A language code is metadata, not content: it now renders only when mark_meta=True ([Language] de) and is skipped otherwise. Plain text inherits the same serializer.

HTML.

  • HTMLParams.html_lang was declared but never used; the root element was the literal "<html>". It is now <html lang="…"> from html_lang (default "en", unchanged) — WCAG 3.1.1 Language of Page.
  • Text items whose meta.language differs from html_lang get a lang attribute on their <p> / <hN> element via the existing attrs parameter of get_html_tag_with_text_direction — WCAG 3.1.2 Language of Parts. Items in the document language get no attribute, so output is unchanged for documents without language meta.
  • The language row in the meta <details> block is left as is (covered by test_semantic_base_meta_fields_roundtrip_and_html_rendering).

Tests

  • test_md_and_text_language_meta_is_not_rendered_as_prose and test_html_declares_document_and_element_languages in test/test_metadata.py.
  • Groundtruth regenerated with DOCLING_GEN_TEST_DATA=1: the 23 HTML files change only on the <html> line (PNG fixtures that regenerated with byte-level churn were reverted).
  • pre-commit run --all-files and the full suite pass locally (701 passed, 6 skipped).

Context

Downstream, yra-doc-pipeline populates meta.language from a deterministic detector and maps it to PDF/UA /Lang; with this change the same data reaches the HTML export instead of leaking into text exports.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit —

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@codecov

codecov Bot commented Sep 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

wittjeff and others added 2 commits September 25, 2026 08:26
…ributes in HTML

Fixes docling-project#745.

Markdown and plain text printed the pydantic repr of `LanguageMetaField`
("confidence=0.99 created_by='...' code=<HumanLanguageLabel.DE: 'de'>") as a
paragraph because `_serialize_meta_field` had no branch for it. A language
code is metadata, not content, so it now renders only under `mark_meta`
("[Language] de") and is skipped otherwise.

HTML: `HTMLParams.html_lang` was declared but never emitted; the root is now
`<html lang="...">` (WCAG 3.1.1). Text items whose `meta.language` differs
from `html_lang` get their own `lang` attribute on the <p>/<hN> element
(WCAG 3.1.2). The language row in the meta <details> block is unchanged.

Regression tests added in test/test_metadata.py; the 23 HTML groundtruth
files change only on the <html> line.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
@wittjeff
wittjeff force-pushed the fix/language-meta-serializers branch from de09bb7 to b46f4e8 Compare September 25, 2026 15:26
@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) ai:possible-duplicate AI triage: another PR may fix the same problem bug Something isn't working serialization Output serializers: Markdown, HTML, LaTeX, JSON, plain-text, outline labels Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

AI triage (advisory)

Checked commit b46f4e88f9. The analysis read the diff only. It did not run code from this pull request.

The PR stops MarkdownMetaSerializer from printing meta.language as a text paragraph. It shows the language only when mark_meta=True. In HTML, it writes html_lang into the <html lang> root tag and adds lang attributes to <p> and <hN> elements whose language differs.

Topic labels added: bug, serialization.

CI safety: no concern found. A maintainer must still approve CI.

Possible duplicates

PR State Signal Assessment
#869 open also references issue #745 duplicate: The same author opened it for the same issue #745. It hides meta.language in Markdown and adds <html lang> and lang attributes in HTML. It edits the same code and tests, so only one PR can merge.
2 related pull request(s) in the same code
PR State Signal Assessment
#701 open changes the same code related: It changes list item marker styles in html.py. It does not touch language handling, and the two changes can merge together.
#933 open changes the same code related: It changes how headings with inline children render in html.py and markdown.py. It does not touch language metadata.

Reference data changes

Change Files
text 23
  • text tests/data/doc/2206.01062.yaml.html: +1 / -1 lines
  • text tests/data/doc/2408.09869_p1_split.gt.html: +1 / -1 lines
  • text tests/data/doc/2408.09869v3_enriched_p1_include_annotations_false.gt.html: +1 / -1 lines
  • text tests/data/doc/2408.09869v3_enriched_p1_include_annotations_true.gt.html: +1 / -1 lines
  • text tests/data/doc/2408.09869v3_enriched_p2_p3_p5.gt.html: +1 / -1 lines
  • text tests/data/doc/2408.09869v3_enriched_split.gt.html: +1 / -1 lines
  • text tests/data/doc/2408.09869v3_enriched_split_p2.gt.html: +1 / -1 lines
  • text tests/data/doc/activities.gt.html: +1 / -1 lines
  • text tests/data/doc/activities_p1.gt.html: +1 / -1 lines
  • text tests/data/doc/activities_p2.gt.html: +1 / -1 lines
  • text tests/data/doc/barchart.gt.html: +1 / -1 lines
  • text tests/data/doc/concatenated.html: +1 / -1 lines
  • text tests/data/doc/constructed_doc.embedded.html.gt: +1 / -1 lines
  • text tests/data/doc/constructed_doc.placeholder.html.gt: +1 / -1 lines
  • text tests/data/doc/constructed_doc.referenced.html.gt: +1 / -1 lines
  • text tests/data/doc/constructed_document.yaml.html: +1 / -1 lines
  • text tests/data/doc/constructed_orig_false.gt.html: +1 / -1 lines
  • text tests/data/doc/constructed_orig_true.gt.html: +1 / -1 lines
  • text tests/data/doc/dummy_doc.yaml.html: +1 / -1 lines
  • text tests/data/doc/inline_and_formatting.gt.html: +1 / -1 lines
  • text tests/data/doc/polymers.gt.html: +1 / -1 lines
  • text tests/data/doc/rich_table.gt.html: +1 / -1 lines
  • text tests/data/docling_document/export/formula_mathml.html: +1 / -1 lines

Assessment: expected. Each of the 23 HTML files changes by one line (+1/-1). This matches the change of the root tag from <html> to <html lang="en"> in HTMLDocSerializer. The summary shows no other changes.

The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI first review (advisory): no blocker found

This review covers the full diff at b46f4e88f9. It read the code and did not run it.

No merge-deciding defect found. I checked the Markdown and plain-text language skip, the HTML <html lang> and element lang attributes, the attribute escaping in get_html_tag_with_text_direction, the subclasses of MarkdownMetaSerializer, and the new tests. The <html> line is the only change in the HTML reference data, as the PR describes.

A maintainer makes the merge decision. This review never approves the pull request.

@github-actions github-actions Bot added the ai:review-lgtm AI first review found no blocker or major issue (advisory) label Nov 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:ci-safe AI triage found no CI-safety concern (advisory) ai:possible-duplicate AI triage: another PR may fix the same problem ai:review-lgtm AI first review found no blocker or major issue (advisory) bug Something isn't working serialization Output serializers: Markdown, HTML, LaTeX, JSON, plain-text, outline

Projects

None yet

1 participant