Repository navigation
Conversation
Symbol, ZapfDingbats, Wingdings, Wingdings 2, Wingdings 3 and Webdings place their glyphs on byte codes, so Office files and PDFs carry their text as bytes or as U+F020-U+F0FF private-use code points. Add one data-only table for all six fonts (docling_core/utils/symbol_fonts.json), a small accessor (SymbolFont.from_name, symbol_font_char, decode_symbol_font_text) and the script that generates the table from the Unicode Consortium sources: the Adobe Symbol and ZapfDingbats mapping files and the normative source references of L2/12-368 for the Wingdings family and Webdings. The table never maps to a private-use code point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
|
✅ DCO Check Passed Thanks @wittjeff, all your commits are properly signed off. 🎉 |
Merge Protections🔴 1 of 2 protections blocking · waiting on 👀 reviews
🔴 Require two reviewer for test updatesWaiting for
This rule is failing.When test data is updated, we require two reviewers
Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
…742) The Wingdings / Webdings tables in `symbol_fonts.rs` were described as the Unicode encoding of their glyphs (L2/11-052R) but equalled pandoc's `Text.Pandoc.Readers.Docx.Symbols`, which differs from the normative L2/12-368 `UCS;ID` source references in 107 entries (53 Wingdings, 29 Wingdings 2, 16 Wingdings 3, 9 Webdings), and the Symbol table mapped Adobe's corporate-use pieces to private-use code points. The hand-kept arrays are replaced by docling-core's shared, data-only `docling_core/utils/symbol_fonts.json` (docling-project/docling-core#889), generated from the Unicode Consortium sources and never mapping to a private-use code point. It is vendored byte-identical as `backend/symbol_fonts.json` and parsed on first use, so docling.rs, Python docling's DOCX reader and docling-parse produce the same characters. - Wingdings `8D` is ➋ (was ❷), `6C` ⚫ (was ●), `FC` ✓ (was ✔), …; Symbol's bracket pieces and serif / sans-serif ® © ™ are the standard characters; the radical extender, which Unicode does not encode, is kept as stored. - `ZapfDingbats` is mapped too, and font names resolve like docling-core's `SymbolFont.from_name` (subset tag and style suffix ignored: `SymbolMT`, `ABCDEF+Wingdings-Regular`, `ITC Zapf Dingbats`). - Tests check the vendored table's guarantees (every font, codes 0x20–0xFF, no private-use values, the Unicode notice kept) and the issue's examples; the four symbol-font fixtures change in exactly those glyphs. - `scripts/sync_symbol_fonts.sh [--check] [REF]` re-vendors the file from docling-core. Closes #742 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
AI triage (advisory)Checked commit This PR adds a mapping table and helper functions to convert symbol-encoded font glyph codes to Unicode. Topic labels added: CI safety: needs maintainer care before you approve CI.
The |
There was a problem hiding this comment.
AI first review (advisory): no blocker found
This review covers the full diff at 3620d5e547. It read the code and did not run it.
The PR adds a shared symbol-font to Unicode mapping table, accessor functions, and generation script for decoding legacy symbol fonts across Docling components. I checked the font name matching, character decoding logic for byte and PUA code points, package configuration, and test coverage, and found no issues.
A maintainer makes the merge decision. This review never approves the pull request.
Shared data for docling-project/docling#4610 (DOCX, PR docling-project/docling#4622), docling-project/docling-parse#416 (PDF, PR docling-project/docling-parse#417) and docling-project/docling.rs#588.
What
One table that maps the byte codes of the symbol-encoded fonts to Unicode, for Symbol, ZapfDingbats, Wingdings, Wingdings 2, Wingdings 3 and Webdings:
docling_core/utils/symbol_fonts.json: data only. Each entry is a byte code (0x20–0xFF) mapped to a code point, both in hexadecimal. The file also gives its sources and the Unicode notice that must stay with the Adobe-derived part.docling_core/utils/symbol_fonts.py: a small accessor.SymbolFont.from_name()accepts the names Office files use (Wingdings 2) and the PDF names (ABCDEF+Wingdings-Regular,SymbolMT,Symbol,Bold). Unicode-encoded fonts such asSymbolaandSegoe UI SymbolgiveNone.symbol_font_char()anddecode_symbol_font_text()accept the byte (aset in Symbol) or its private-use form (U+F061, as inw:ttext andw:sym w:char). A code without a mapping is kept, so no text is lost.scripts/generate_symbol_fonts.py: builds the JSON from the Unicode Consortium sources.--checkreports a stale table.Why one shared table
Three readers now decode these fonts, and each has its own table. They already give different text for the same Symbol glyphs:
Δ Ω μΔ Ω μ∆U+2206,ΩU+2126,µU+00B5® © ™So the same Word document gives different text when it is converted as DOCX and as PDF. The JSON format lets docling-parse (C++, which already reads JSON with nlohmann) and docling.rs read the same file. docling-core is the only Python package that both docling-slim (DOCX without docling-parse) and docling-parse install.
Sources and choices
MAPPINGS/VENDORS/ADOBE/symbol.txt,zdingbat.txt).® © ™use the plain signs. The radical extender has no Unicode equivalent, so it is not in the table.UCS;IDlist for 866 of the 867 glyphs; the one exception is the Windows logo, w-1255, which is not encoded. I also read the detailed tables of section 6 of the same document, and they agree with the list on all 866 glyphs. Byte 0x20 is the space glyph in every font.These Wingdings values differ from pandoc's table (
Text.Pandoc.Readers.Docx.Symbols) in 107 entries. Pandoc often uses an older character that L2/12-368 gives only as a cross-reference. Two examples: Wingdings 0x30 isU+1F5C0 FOLDER, where pandoc hasU+1F4C1 FILE FOLDER; Wingdings 0x8D isU+278B(sans-serif circled digit two), where pandoc hasU+2777. The table follows the Unicode document, not pandoc, so no GPL data is involved.Next steps (not in this PR)
decode_symbol_font_text()for Symbol, Wingdings and Webdings runs andw:sym, after #4622 merges.pdf_resources, use it in the private-use fold of PR fix: improve meta migration and warning handling #417 for all six fonts, and add a test that checks the copy is identical to the docling-core file.Checks
uv run pytest tests/test_symbol_fonts.py: 28 passed.uv run python scripts/generate_symbol_fonts.py --check: the table is current.pre-commit run(ruff, mypy, full pytest, docs): all passed.package-data "*.json").🤖 Generated with Claude Code