Skip to content

feat(utils): add a shared symbol-font to Unicode table - #889

Open
wittjeff wants to merge 1 commit into
docling-project:mainfrom
wittjeff:feat/symbol-font-unicode-tables
Open

wittjeff wants to merge 1 commit into
docling-project:mainfrom
wittjeff:feat/symbol-font-unicode-tables

Conversation

@wittjeff

Copy link
Copy Markdown
Contributor

Shared data for docling-project/docling#4610 (DOCX, PR docling-project/docling#4622), docling-project/docling-parse#416 (PDF, PR docling-project/docling-parse#417) and docling-project/docling.rs#588.

What

One table that maps the byte codes of the symbol-encoded fonts to Unicode, for Symbol, ZapfDingbats, Wingdings, Wingdings 2, Wingdings 3 and Webdings:

  • docling_core/utils/symbol_fonts.json: data only. Each entry is a byte code (0x20–0xFF) mapped to a code point, both in hexadecimal. The file also gives its sources and the Unicode notice that must stay with the Adobe-derived part.
  • docling_core/utils/symbol_fonts.py: a small accessor.
    • SymbolFont.from_name() accepts the names Office files use (Wingdings 2) and the PDF names (ABCDEF+Wingdings-Regular, SymbolMT, Symbol,Bold). Unicode-encoded fonts such as Symbola and Segoe UI Symbol give None.
    • symbol_font_char() and decode_symbol_font_text() accept the byte (a set in Symbol) or its private-use form (U+F061, as in w:t text and w:sym w:char). A code without a mapping is kept, so no text is lost.
  • scripts/generate_symbol_fonts.py: builds the JSON from the Unicode Consortium sources. --check reports a stale table.

Why one shared table

Three readers now decode these fonts, and each has its own table. They already give different text for the same Symbol glyphs:

Symbol code docling PR #4622 docling.rs docling-parse PR #417 (Adobe Glyph List)
0x44, 0x57, 0x6D Δ Ω μ Δ Ω μ ∆ U+2206, Ω U+2126, µ U+00B5
0xD2–0xD4, 0xE2–0xE4 ® © ™ U+F6Dx / U+F8Ex (private use) U+F6Dx / U+F8Ex (private use)

So the same Word document gives different text when it is converted as DOCX and as PDF. The JSON format lets docling-parse (C++, which already reads JSON with nlohmann) and docling.rs read the same file. docling-core is the only Python package that both docling-slim (DOCX without docling-parse) and docling-parse install.

Sources and choices

  • Symbol, ZapfDingbats: the Adobe mapping files that Unicode publishes (MAPPINGS/VENDORS/ADOBE/symbol.txt, zdingbat.txt).
    • Where a code has two values, the table uses the character a reader expects: the Greek letters, not INCREMENT, OHM SIGN or MICRO SIGN; SPACE, not NO-BREAK SPACE.
    • The Corporate Use Subarea glyphs resolve to standard characters. The ZapfDingbats ornaments resolve by their Unicode names (U+2768–U+2775). The Symbol delimiter pieces use U+239B–U+23AD, U+23AF and U+23D0, the same rewrite that docling-parse already applies to the Adobe Glyph List. The serif and sans serif ® © ™ use the plain signs. The radical extender has no Unicode equivalent, so it is not in the table.
  • Wingdings, Wingdings 2, Wingdings 3, Webdings: the normative "Source references for wingdings and webding symbols" at the end of L2/12-368. This is a UCS;ID list for 866 of the 867 glyphs; the one exception is the Windows logo, w-1255, which is not encoded. I also read the detailed tables of section 6 of the same document, and they agree with the list on all 866 glyphs. Byte 0x20 is the space glyph in every font.
  • The table never maps to a private-use or unassigned code point. A test checks this.

These Wingdings values differ from pandoc's table (Text.Pandoc.Readers.Docx.Symbols) in 107 entries. Pandoc often uses an older character that L2/12-368 gives only as a cross-reference. Two examples: Wingdings 0x30 is U+1F5C0 FOLDER, where pandoc has U+1F4C1 FILE FOLDER; Wingdings 0x8D is U+278B (sans-serif circled digit two), where pandoc has U+2777. The table follows the Unicode document, not pandoc, so no GPL data is involved.

Next steps (not in this PR)

  • docling: the Word backend can use decode_symbol_font_text() for Symbol, Wingdings and Webdings runs and w:sym, after #4622 merges.
  • docling-parse: ship a copy of the JSON in pdf_resources, use it in the private-use fold of PR fix: improve meta migration and warning handling #417 for all six fonts, and add a test that checks the copy is identical to the docling-core file.
  • docling.rs: can generate its arrays from the same file.

Checks

  • uv run pytest tests/test_symbol_fonts.py: 28 passed.
  • uv run python scripts/generate_symbol_fonts.py --check: the table is current.
  • pre-commit run (ruff, mypy, full pytest, docs): all passed.
  • The JSON is in the wheel (package-data "*.json").

🤖 Generated with Claude Code

Symbol, ZapfDingbats, Wingdings, Wingdings 2, Wingdings 3 and Webdings
place their glyphs on byte codes, so Office files and PDFs carry their
text as bytes or as U+F020-U+F0FF private-use code points. Add one
data-only table for all six fonts (docling_core/utils/symbol_fonts.json),
a small accessor (SymbolFont.from_name, symbol_font_char,
decode_symbol_font_text) and the script that generates the table from
the Unicode Consortium sources: the Adobe Symbol and ZapfDingbats
mapping files and the normative source references of L2/12-368 for the
Wingdings family and Webdings. The table never maps to a private-use
code point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit —

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

artiz added a commit to docling-project/docling.rs that referenced this pull request Oct 17, 2026
…742)

The Wingdings / Webdings tables in `symbol_fonts.rs` were described as the
Unicode encoding of their glyphs (L2/11-052R) but equalled pandoc's
`Text.Pandoc.Readers.Docx.Symbols`, which differs from the normative
L2/12-368 `UCS;ID` source references in 107 entries (53 Wingdings, 29
Wingdings 2, 16 Wingdings 3, 9 Webdings), and the Symbol table mapped
Adobe's corporate-use pieces to private-use code points.

The hand-kept arrays are replaced by docling-core's shared, data-only
`docling_core/utils/symbol_fonts.json` (docling-project/docling-core#889),
generated from the Unicode Consortium sources and never mapping to a
private-use code point. It is vendored byte-identical as
`backend/symbol_fonts.json` and parsed on first use, so docling.rs, Python
docling's DOCX reader and docling-parse produce the same characters.

- Wingdings `8D` is ➋ (was ❷), `6C` ⚫ (was ●), `FC` ✓ (was ✔), …; Symbol's
  bracket pieces and serif / sans-serif ® © ™ are the standard characters;
  the radical extender, which Unicode does not encode, is kept as stored.
- `ZapfDingbats` is mapped too, and font names resolve like docling-core's
  `SymbolFont.from_name` (subset tag and style suffix ignored: `SymbolMT`,
  `ABCDEF+Wingdings-Regular`, `ITC Zapf Dingbats`).
- Tests check the vendored table's guarantees (every font, codes 0x20–0xFF,
  no private-use values, the Unicode notice kept) and the issue's examples;
  the four symbol-font fixtures change in exactly those glyphs.
- `scripts/sync_symbol_fonts.sh [--check] [REF]` re-vendors the file from
  docling-core.

Closes #742

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
@github-actions github-actions Bot added ai:ci-needs-care AI triage: check CI-safety findings before you approve CI enhancement New feature or request labels Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

AI triage (advisory)

Checked commit 3620d5e547. The analysis read the diff only. It did not run code from this pull request.

This PR adds a mapping table and helper functions to convert symbol-encoded font glyph codes to Unicode.

Topic labels added: enhancement.

CI safety: needs maintainer care before you approve CI.

  • scripts/generate_symbol_fonts.py: changes maintenance scripts

The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI first review (advisory): no blocker found

This review covers the full diff at 3620d5e547. It read the code and did not run it.

The PR adds a shared symbol-font to Unicode mapping table, accessor functions, and generation script for decoding legacy symbol fonts across Docling components. I checked the font name matching, character decoding logic for byte and PUA code points, package configuration, and test coverage, and found no issues.

A maintainer makes the merge decision. This review never approves the pull request.

@github-actions github-actions Bot added the ai:review-lgtm AI first review found no blocker or major issue (advisory) label Nov 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:ci-needs-care AI triage: check CI-safety findings before you approve CI ai:review-lgtm AI first review found no blocker or major issue (advisory) enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant