Skip to content

fix(fonts): font and ToUnicode decoding fixes, consolidated per #428 - #431

Merged
PeterStaar-IBM merged 10 commits into
docling-project:mainfrom
wittjeff:fix/font-decoding
Nov 3, 2026
Merged

PeterStaar-IBM merged 10 commits into
docling-project:mainfrom
wittjeff:fix/font-decoding

Conversation

@wittjeff

@wittjeff wittjeff commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Font and ToUnicode decoding topic from #428. This PR replaces #419, #388, #327, #389, #417 and #322, which are closed and link here. Commits 1 to 6 are one commit per fix, in that order, and commits 7 and 8 are review changes. The branch has no merge commits.

Fixes #145. Fixes #133. Fixes #317. Refs #362 (case 1, opt-in through do_char_normalization).

commit replaces fix
1 #419 (fixes #145, #133) /ToUnicode source codes are read as big-endian integers, not as UTF-16 text. Codes 0xD800–0xDFFF were lost, and their text overwrote code 0x47.
2 #388 (case 1 of #362) The sanitizer keeps U+2012–U+2015 and U+2212 instead of folding them to -. Commit 8 changes this: the fold is on by default. U+2010 and U+2011 always fold, because docling joins line-end hyphens by looking for -.
3 #327 (fixes #317) A base-14 font dict with the obsolete /Name label decodes through the base font's built-in encoding, as the width and bbox lookups already do.
4 #389 Hexadecimal glyph-index names (Skia g%X) are unmapped glyphs, not text.
5 #417 Symbol and ZapfDingbats private-use /ToUnicode values (U+F020–U+F0FF) fold through the font's own encoding.
6 #322 (follow-up to #299) Symbolic fonts decode through their declared encoding or the embedded program's builtin encoding, and a code that nothing resolves becomes a GLYPH marker instead of fabricated text.
7 review The builtin encoding table is a std::unordered_map.
8 review New DecodeConfig.do_char_normalization, default True. True folds U+2012–U+2015 and U+2212 to -, as main does. False keeps them. The other replacements run in both modes.

Changes made while consolidating

Evidence and ground truth changes

I regenerated the parser ground truth with this branch against the pinned dataset (21374211) and diffed the *.word.txt exports page by page. 33 of the 940 sampled pages change. Every changed page is assigned to the fix that changes it below. Render ground truth does not change. test_render_reference_documents_from_filenames also checks the parsed text against the parser ground truth, so test_regression_threaded_render.py stays red until the parser ground truth is updated.

Each section gives the reason, the first changed words of the cited pages (before → after), and the full page list. The maintainers update the ground truth when merging; the regenerated files are available if useful.

#419: 3 pages in 2 documents

Ground truth changes because ToUnicode source codes in the surrogate range are read as numbers, so CJK and Latin characters that were lost or replaced come back.

skipped_2pages.pdf p2:

before after
헬스GLYPH<c=56526,font=/AAAAAP+NotoSansCJKkr-Bold>어 헬스케어
GLYPH<c=55926,font=/AAAAAP+NotoSansCJKkr-Bold>단재료 첨단재료
Manu치acturing) Manufacturing)
De치ense) Defense)
농GLYPH<c=56163,font=/AAAAAP+NotoSansCJKkr-Bold>산 농축산

skipped_1page.pdf p2:

before after
블록GLYPH<c=55938,font=/AAAAAI+NotoSansCJKkr-Bold>인 블록체인
엣지GLYPH<c=56514,font=/AAAAAI+NotoSansCJKkr-Bold>퓨팅 엣지컴퓨팅
GLYPH<c=56502,font=/AAAAAI+NotoSansCJKkr-Bold>트롤 인GLYPH<c=57086,font=/AAAAAI+NotoSansCJKk… 컨트롤 인터페이스
GLYPH<c=55798,font=/AAAAAI+NotoSansCJKkr-Bold>세대 차세대
GLYPH<c=55798,font=/AAAAAI+NotoSansCJKkr-Bold>세대 차세대
All pages

skipped_1page.pdf p2, skipped_2pages.pdf p2, skipped_2pages.pdf p3

#388: 0 pages by default

The ground truth for dashes does not change, because do_char_normalization defaults to True. These are examples with do_char_normalization=False:

elsevier-00.pdf p9:

default False
339-349. 339–349.
863-865. 863–865.
1932-1937. 1932–1937.

19246029784032167-4.pdf p3:

default False
- −
- −

I checked both modes on the 218 pages that previously changed only because of dashes. The default matches the stored ground truth on 217 pages. False matches the earlier keep-the-dashes output on the same 217 pages. The other page, cropbox_versus_mediabox_02.pdf p3, differs in the same way in both modes, because my comparison script splits lines at the \x1e characters in that page's text.

#327: 1 pages in 1 documents

Ground truth changes because the ZapfDingbats font labelled /Name /F3 decodes through its builtin encoding (code 108 is a71, the black circle).

annots_01.pdf p1:

before after
l ●
l ●
l ●
l ●
All pages

annots_01.pdf p1

#389: 2 pages in 2 documents

Ground truth changes because bare hexadecimal glyph names (Skia g%X/G%X) become GLYPH<name:…> markers instead of text.

9acb62b4-4449-48d0-8127-f2be26349a6a-6.pdf p1:

before after
GDB GLYPH<name:GDB>
GFEGLYPH<name:G27> GLYPH<name:G30>GLYPH<name:G23>G3BGLYPH<name:G31> GLYPH<name:G21>GLYPH<… GLYPH<name:GFE>GLYPH<name:G27> GLYPH<name:G30>GLYPH<name:G23>GLYPH<name:G3B>GLYPH<name:G3…
GLYPH<name:G35>GLYPH<name:G59>G8BGB4GLYPH<name:G25>GLYPH<name:G44>GD8GA4G4AG3FGLYPH<name:… GLYPH<name:G35>GLYPH<name:G59>GLYPH<name:G8B>GLYPH<name:GB4>GLYPH<name:G25>GLYPH<name:G44…
G2AGLYPH<name:G25>GLYPH<name:G33>GE8GLYPH<name:G48>GDFGAAGLYPH<name:G33>GLYPH<name:G67>GL… GLYPH<name:G2A>GLYPH<name:G25>GLYPH<name:G33>GLYPH<name:GE8>GLYPH<name:G48>GLYPH<name:GDF…
G5CG3A G8BGLYPH<name:G24>GCBGBBGLYPH<name:G25>GF4GLYPH<name:G99>GLYPH<name:G44>G5DG5CGLYP… GLYPH<name:G5C>GLYPH<name:G3A> GLYPH<name:G8B>GLYPH<name:G24>GLYPH<name:GCB>GLYPH<name:GB…
All pages

9acb62b4-4449-48d0-8127-f2be26349a6a-6.pdf p1, ba55e6a2-1a86-460d-bf2c-e01a6771657b-2.pdf p1

#417: 11 pages in 9 documents

Ground truth changes because Symbol private-use values (U+F0xx) fold through the Symbol encoding.

16293520921922171261-1.pdf p2:

before after
•
•

b766d7d18edf92b1_0002.pdf p1:

before after
=
  θ =
1 =1
       θ γ - = + + =
=

eac6df0b2a6f9cf0_0002.pdf p1:

before after
(m)、内部摩擦角φs(deg)、砂礫密度 (m)、内部摩擦角φs(deg)、砂礫密度σ
)、流体密度 )、流体密度ρ
   τ τ ≤
            ( )( ) ( ) { } ρ ρ σ θ τ
     + + - + =
All pages

11033738705194660817-2.pdf p15, 11033738705194660817-2.pdf p16, 13828292398120620566-1.pdf p1, 16293520921922171261-1.pdf p1, 16293520921922171261-1.pdf p2, 16687286043211380927-1.pdf p15, 6439951649404873078-1.pdf p1, 8313873609013042575-4.pdf p5, b766d7d18edf92b1_0002.pdf p1, eac6df0b2a6f9cf0_0002.pdf p1, redp5110_sampled.pdf p14

#322: 16 pages in 4 documents

Ground truth changes because symbolic fonts decode through their builtin encoding; the result agrees with pdfium's text extraction.

10017889119412695965-12.pdf p5:

before after
ˇGLYPH<240>ŁGLYPH<229>GLYPH<236>GLYPH<237>ŁŒ æŁªGLYPH<237>GLYPH<224>ºGLYPH<238>GLYPH<226>… Приемник сигналов пульта ДУ Дисплей Кнопка тестового включения Кнопка ВКЛ./ВЫКЛ. Кнопка б…
˚GLYPH<237>GLYPH<238>GLYPH<239>ŒGLYPH<224> GLYPH<240>GLYPH<229>ªGLYPH<243>ºŁGLYPH<240>GLY… Кнопка регулировки
˚GLYPH<237>GLYPH<238>GLYPH<239>ŒGLYPH<224> GLYPH<228>GLYPH<238>GLYPH<239>GLYPH<238>ºGLYPH… Кнопка дополнительных функций Кнопка режима работы Кнопка скорости вентилятора ÏÀÍÅËÜ ÓÏÐ…
GLYPH<226>ߌºGLYPH<254>GLYPH<247>GLYPH<224>GLYPH<229>GLYPH<242>æGLYPH<255>. GLYPH<149> ˚G… âûêëþ÷àåòñÿ. • Êíîïêà ðåæèìà ðàáîòû: ïîçâîëÿåò âûáðàòü îäèí èç ïÿòè ðåæèìîâ ðàáîòû êîíäèö…
GLYPH<192>GLYPH<226>GLYPH<242>GLYPH<238>GLYPH<236>GLYPH<224>GLYPH<242>ŁGLYPH<247>GLYPH<22… Àâòîìàòè÷åñêèé: êîíäèöèîíåð âûáèðàåò ðåæèì àâòîìàòè÷åñêè â çàâèñèìîñòè îò óñëîâèé (òåìïåð…

5462902444445925132-4.pdf p1:

before after
" π
œ¨¨¨¨ ˙»»»»
ˇ¨¨¨¨ œ»»»»
œ¨¨¨¨ ˙»»»»
GLYPH<246>˙˙˙˙ ˆ««««

4e5aa9238544d683_0009.pdf p1:

before after
GLYPH<32>GLYPH<54>:GLYPH<50>GLYPH<54> GLYPH<54>:GLYPH<50>GLYPH<54>
GLYPH<32>7:GLYPH<32>GLYPH<52> 7: GLYPH<52>
GLYPH<32>7:GLYPH<32>8 7: 8
GLYPH<32>7:GLYPH<49>GLYPH<51> 7:GLYPH<49>GLYPH<51>
GLYPH<32>7:GLYPH<49>GLYPH<53> 7:GLYPH<49>GLYPH<53>
All pages

10017889119412695965-12.pdf p11, 10017889119412695965-12.pdf p2, 10017889119412695965-12.pdf p4, 10017889119412695965-12.pdf p5, 10017889119412695965-12.pdf p9, 10976580690960943929_004.pdf p1, 10976580690960943929_004.pdf p12, 10976580690960943929_004.pdf p4, 10976580690960943929_004.pdf p7, 10976580690960943929_004.pdf p9, 4e5aa9238544d683_0009.pdf p1, 5462902444445925132-4.pdf p1, 5462902444445925132-4.pdf p2, 5462902444445925132-4.pdf p3, 5462902444445925132-4.pdf p4, 5462902444445925132-4.pdf p5

Effects to review

  • Line-end U+2212 (only with do_char_normalization=False). Two documents encode line-end hyphens as the minus sign: 1322063424305812499-5.pdf (Greek, 133 words such as κα−) and 11867823218465466809_003.pdf (68 words). With False, those words end in U+2212, which docling's dehyphenation does not join. The default is not affected.
  • Mojibake that the font itself encodes (commit 6). 10017889119412695965-12.pdf p2, p4, p9 and p11 change from GLYPH<…> markers to text such as Ñîäåðæàíèå, which is "Содержание" in Windows-1251 read as Latin-1. The font names its Cyrillic glyphs with Latin glyph names, and pdfium extracts the same text. On p5 the same document's other font decodes to correct Cyrillic (Приемник сигналов).

For commit 6, I compared the changed pages with pdfium's text extraction: 10017889119412695965-12.pdf p2 and p5, 5462902444445925132-4.pdf p1 (˙»»»», œ»»»», ˆ««««) and 4e5aa9238544d683_0009.pdf p1 (7: 8) agree.

Tests

  • tests/test_unit_*.py: 335 passed, 1 skipped.
  • test_regression_parse.py, test_regression_threaded_parse.py and test_regression_threaded_render.py against the pinned dataset with the regenerated parser ground truth: 66 passed.

Checklist (#428):

🤖 Generated with Claude Code

wittjeff and others added 6 commits October 28, 2026 13:05
cmap_parser::get_source() decoded each bfchar/bfrange source code as
UTF-16 and set_mapping/set_range read the key back with utf8::next.
Codes 0xD800-0xDFFF are lone surrogates, so the decode threw and the
fallback "GLYPH<cmap:...>" string was keyed as 'G' (0x47): the code
lost its text and code 0x47 took the text of the bad entry. Large CJK
fonts use this range; before 7.16.0 the lost mapping also crashed the
parse with "Invalid code point" (docling#1111).

Read source codes as big-endian integers with to_uint32(), as the
codespacerange bounds already are, and pass integer codes to
set_mapping and set_range.

Fixes docling-project#145
Fixes docling-project#133

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
…cell text

`text_constants::replacements` folded U+2012..U+2015 and U+2212 to `-`
together with the hyphen variants U+2010 and U+2011. The fold is
unconditional and `orig` is a copy of the folded text, so nothing
downstream could recover an en dash in `pages 3–5` or a minus sign in
`−3`. Only the two hyphen variants keep folding, because the
line-joining dehyphenation in docling recognises an ASCII hyphen alone.

Covers case 1 of docling-project#362. The ligature
fallback (case 2) and the kerning-gap split (case 3) are separate code
paths and will come as separate PRs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
…ies a /Name label

A base-14 font dictionary carrying the obsolete /Name entry (e.g.
<< /BaseFont /Symbol /Name /F5 >>, as ReportLab emits) reports the label
as its font name. get_correct_character() matched only that name against
the base-14 metrics, so Symbol/ZapfDingbats text missed the built-in
encoding tables and fell back to StandardEncoding: Symbol's arrowright
(0256) decoded as the 'fi' ligature and ZapfDingbats' a35 star as 'H'.
The bogus ligature then merged into the neighbouring line under the
sanitator's cross-font ligature exemption, which is why issue docling-project#317
presented as a merging bug.

Mirror the font_name-or-base_font fallback that every width/bbox lookup
in page_font.h already uses.

Fixes docling-project#317.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
Skia/PDF names Type 3 glyphs with printf("g%X", gid), so /g15C or /gA1
carry hexadecimal digits and escaped the decimal-only glyph-index
pattern. Such names then leaked into the extracted text as 'g15C'
instead of the GLYPH<name:...> marker (the docling-project#238 / docling-project#302 failure class).

Widen the character class to [0-9A-Fa-f]; regex_match still requires a
full match, so AGL names such as gbreve or gcaron are unaffected.

Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
…e font encoding

Office exporters embed Symbol, SymbolMT and ZapfDingbats as Identity-H fonts
whose /ToUnicode map sends every glyph to the "symbol charset" private-use
block, U+F000 plus the byte code. Taken at face value, a bullet is U+F0B7 and
an equals sign U+F03D: invisible characters that no search finds. On the 940
regression pages, 10 documents carry 815 such characters in Symbol fonts.

After the /ToUnicode stream is parsed, fold single code points in
U+F020-U+F0FF back through the matched base font's AFM glyph names and the
glyph list when that base font is Symbol or ZapfDingbats. Codes the encoding
lacks keep the map's value, and every other font is untouched, so CJK gaiji
and Arabic presentation fonts that use the block for real text are unaffected.

closes docling-project#416

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
A symbolic or Type0/CID font whose /ToUnicode fails to parse, or that has
none, fell through to the Standard/WinAnsi tables and fabricated text
(for example '!"#', or 'Ł' for Cyrillic 'и'). Follow-up to docling-project#299.

- A simple symbolic font with no /Encoding maps its codes through the
  embedded program's builtin encoding (PDF 32000-1 9.6.6.2, 9.6.6.4).
  FreeType walks codes 0-255 through the builtin charmaps and the text
  comes from the glyph name (AGL, uniXXXX, uXXXX) or the program's
  Unicode cmap. This mirrors the renderer's char_code_to_glyph_index, so
  text and drawn glyph agree.
- A simple symbolic font that declares a simple encoding (/WinAnsiEncoding
  or an /Encoding dict) keeps decoding through it: many fonts carry a
  spurious symbolic flag.
- When neither a parsed cmap, a declared encoding nor the builtin encoding
  resolves a code, a GLYPH marker is emitted instead of fabricated text.
  Type0/CID fonts get the marker unconditionally.

Precedence: /Differences, /ToUnicode, builtin, then the name-matched
base-font tables and the marker guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@wittjeff

Copy link
Copy Markdown
Contributor Author

Correction to the PR description. It says: "Render ground truth does not change: test_regression_threaded_render.py passes against the stored ground truth." That is not correct as written.

test_render_reference_documents_from_filenames also checks the parsed text against the parser ground truth. In CI run 40751310128, it fails on the first changed page: 10003567098752154021-14.pdf p18, - → – (commit 2). The render ground truth (images) still does not change. But this test stays red until the parser ground truth is updated, like the other two regression tests.

The CI failures match the description: the regression-tests job fails on 251 distinct pages, and they are the same 251 pages that the description lists. No page is extra and no page is missing. All other jobs pass.

@PeterStaar-IBM PeterStaar-IBM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking forward to merge this in, can we just make it unordered_map, it improves the performance generally (and I dont see the need for sorting)

@PeterStaar-IBM

Copy link
Copy Markdown
Member

looking forward to merge this in, can we just make it unordered_map, it improves the performance generally (and I dont see the need for sorting)

@wittjeff just fyi ^^. If we fix this, we can merge!

Comment thread src/parse/pdf_sanitators/constants.h Resolved
wittjeff and others added 2 commits October 30, 2026 09:14
Requested in the review of docling-project#431. Nothing iterates the table in code
order: it is filled once per font and then read by code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
…shes

Requested in the review of docling-project#431. U+2012..U+2015 and U+2212 fold to `-`
again by default, so existing callers see the same text as on main.
DecodeConfig(do_char_normalization=False) keeps the characters that the
document encodes (docling-parse#362). The other replacements, including
the hyphen variants U+2010 and U+2011, run in both modes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
@PeterStaar-IBM
PeterStaar-IBM merged commit 8aa5cac into docling-project:main Nov 3, 2026
81 of 83 checks passed
@wittjeff
wittjeff deleted the fix/font-decoding branch November 3, 2026 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants