Repository navigation
fix(fonts): font and ToUnicode decoding fixes, consolidated per #428 - #431
Conversation
cmap_parser::get_source() decoded each bfchar/bfrange source code as UTF-16 and set_mapping/set_range read the key back with utf8::next. Codes 0xD800-0xDFFF are lone surrogates, so the decode threw and the fallback "GLYPH<cmap:...>" string was keyed as 'G' (0x47): the code lost its text and code 0x47 took the text of the bad entry. Large CJK fonts use this range; before 7.16.0 the lost mapping also crashed the parse with "Invalid code point" (docling#1111). Read source codes as big-endian integers with to_uint32(), as the codespacerange bounds already are, and pass integer codes to set_mapping and set_range. Fixes docling-project#145 Fixes docling-project#133 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
…cell text `text_constants::replacements` folded U+2012..U+2015 and U+2212 to `-` together with the hyphen variants U+2010 and U+2011. The fold is unconditional and `orig` is a copy of the folded text, so nothing downstream could recover an en dash in `pages 3–5` or a minus sign in `−3`. Only the two hyphen variants keep folding, because the line-joining dehyphenation in docling recognises an ASCII hyphen alone. Covers case 1 of docling-project#362. The ligature fallback (case 2) and the kerning-gap split (case 3) are separate code paths and will come as separate PRs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
…ies a /Name label A base-14 font dictionary carrying the obsolete /Name entry (e.g. << /BaseFont /Symbol /Name /F5 >>, as ReportLab emits) reports the label as its font name. get_correct_character() matched only that name against the base-14 metrics, so Symbol/ZapfDingbats text missed the built-in encoding tables and fell back to StandardEncoding: Symbol's arrowright (0256) decoded as the 'fi' ligature and ZapfDingbats' a35 star as 'H'. The bogus ligature then merged into the neighbouring line under the sanitator's cross-font ligature exemption, which is why issue docling-project#317 presented as a merging bug. Mirror the font_name-or-base_font fallback that every width/bbox lookup in page_font.h already uses. Fixes docling-project#317. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
Skia/PDF names Type 3 glyphs with printf("g%X", gid), so /g15C or /gA1
carry hexadecimal digits and escaped the decimal-only glyph-index
pattern. Such names then leaked into the extracted text as 'g15C'
instead of the GLYPH<name:...> marker (the docling-project#238 / docling-project#302 failure class).
Widen the character class to [0-9A-Fa-f]; regex_match still requires a
full match, so AGL names such as gbreve or gcaron are unaffected.
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
…e font encoding Office exporters embed Symbol, SymbolMT and ZapfDingbats as Identity-H fonts whose /ToUnicode map sends every glyph to the "symbol charset" private-use block, U+F000 plus the byte code. Taken at face value, a bullet is U+F0B7 and an equals sign U+F03D: invisible characters that no search finds. On the 940 regression pages, 10 documents carry 815 such characters in Symbol fonts. After the /ToUnicode stream is parsed, fold single code points in U+F020-U+F0FF back through the matched base font's AFM glyph names and the glyph list when that base font is Symbol or ZapfDingbats. Codes the encoding lacks keep the map's value, and every other font is untouched, so CJK gaiji and Arabic presentation fonts that use the block for real text are unaffected. closes docling-project#416 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
A symbolic or Type0/CID font whose /ToUnicode fails to parse, or that has none, fell through to the Standard/WinAnsi tables and fabricated text (for example '!"#', or 'Ł' for Cyrillic 'и'). Follow-up to docling-project#299. - A simple symbolic font with no /Encoding maps its codes through the embedded program's builtin encoding (PDF 32000-1 9.6.6.2, 9.6.6.4). FreeType walks codes 0-255 through the builtin charmaps and the text comes from the glyph name (AGL, uniXXXX, uXXXX) or the program's Unicode cmap. This mirrors the renderer's char_code_to_glyph_index, so text and drawn glyph agree. - A simple symbolic font that declares a simple encoding (/WinAnsiEncoding or an /Encoding dict) keeps decoding through it: many fonts carry a spurious symbolic flag. - When neither a parsed cmap, a declared encoding nor the builtin encoding resolves a code, a GLYPH marker is emitted instead of fabricated text. Type0/CID fonts get the marker unconditionally. Precedence: /Differences, /ToUnicode, builtin, then the name-matched base-font tables and the marker guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
|
✅ DCO Check Passed Thanks @wittjeff, all your commits are properly signed off. 🎉 |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
|
Correction to the PR description. It says: "Render ground truth does not change:
The CI failures match the description: the regression-tests job fails on 251 distinct pages, and they are the same 251 pages that the description lists. No page is extra and no page is missing. All other jobs pass. |
PeterStaar-IBM
left a comment
There was a problem hiding this comment.
looking forward to merge this in, can we just make it unordered_map, it improves the performance generally (and I dont see the need for sorting)
@wittjeff just fyi ^^. If we fix this, we can merge! |
Requested in the review of docling-project#431. Nothing iterates the table in code order: it is filled once per font and then read by code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
…shes Requested in the review of docling-project#431. U+2012..U+2015 and U+2212 fold to `-` again by default, so existing callers see the same text as on main. DecodeConfig(do_char_normalization=False) keeps the characters that the document encodes (docling-parse#362). The other replacements, including the hyphen variants U+2010 and U+2011, run in both modes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Font and ToUnicode decoding topic from #428. This PR replaces #419, #388, #327, #389, #417 and #322, which are closed and link here. Commits 1 to 6 are one commit per fix, in that order, and commits 7 and 8 are review changes. The branch has no merge commits.
Fixes #145. Fixes #133. Fixes #317. Refs #362 (case 1, opt-in through
do_char_normalization)./ToUnicodesource codes are read as big-endian integers, not as UTF-16 text. Codes0xD800–0xDFFFwere lost, and their text overwrote code0x47.-. Commit 8 changes this: the fold is on by default. U+2010 and U+2011 always fold, because docling joins line-end hyphens by looking for-./Namelabel decodes through the base font's built-in encoding, as the width and bbox lookups already do.g%X) are unmapped glyphs, not text./ToUnicodevalues (U+F020–U+F0FF) fold through the font's own encoding.GLYPHmarker instead of fabricated text.std::unordered_map.DecodeConfig.do_char_normalization, defaultTrue.Truefolds U+2012–U+2015 and U+2212 to-, asmaindoes.Falsekeeps them. The other replacements run in both modes.Changes made while consolidating
mainaddedmatched_font_name()/matched_base_font()topage_font.h. The fix now usesmatched_base_font()inget_character_from_encoding(). That helper does what fix: consult base-font metrics for character decoding when the font dict carries a /Name label #327 did (font name first, then/BaseFont) and is cached. The member comment says which call sites use which helper.и,2,ش) instead of the text line: since fix: emit glyph marker for symbolic/composite fonts with no usable cmap #322 was opened,main's line assembly puts the right-to-leftشfirst, which is not part of this fix. Commit 6 squashes the three commits of fix: emit glyph marker for symbolic/composite fonts with no usable cmap #322.HF_DATASET_REVISIONis not included (Consolidating the open rendering, text and font PRs: scope and evidence requirements #428 section 4).Evidence and ground truth changes
I regenerated the parser ground truth with this branch against the pinned dataset (
21374211) and diffed the*.word.txtexports page by page. 33 of the 940 sampled pages change. Every changed page is assigned to the fix that changes it below. Render ground truth does not change.test_render_reference_documents_from_filenamesalso checks the parsed text against the parser ground truth, sotest_regression_threaded_render.pystays red until the parser ground truth is updated.Each section gives the reason, the first changed words of the cited pages (before → after), and the full page list. The maintainers update the ground truth when merging; the regenerated files are available if useful.
#419: 3 pages in 2 documents
Ground truth changes because ToUnicode source codes in the surrogate range are read as numbers, so CJK and Latin characters that were lost or replaced come back.
skipped_2pages.pdfp2:헬스GLYPH<c=56526,font=/AAAAAP+NotoSansCJKkr-Bold>어헬스케어GLYPH<c=55926,font=/AAAAAP+NotoSansCJKkr-Bold>단재료첨단재료Manu치acturing)Manufacturing)De치ense)Defense)농GLYPH<c=56163,font=/AAAAAP+NotoSansCJKkr-Bold>산농축산skipped_1page.pdfp2:블록GLYPH<c=55938,font=/AAAAAI+NotoSansCJKkr-Bold>인블록체인엣지GLYPH<c=56514,font=/AAAAAI+NotoSansCJKkr-Bold>퓨팅엣지컴퓨팅GLYPH<c=56502,font=/AAAAAI+NotoSansCJKkr-Bold>트롤 인GLYPH<c=57086,font=/AAAAAI+NotoSansCJKk…컨트롤 인터페이스GLYPH<c=55798,font=/AAAAAI+NotoSansCJKkr-Bold>세대차세대GLYPH<c=55798,font=/AAAAAI+NotoSansCJKkr-Bold>세대차세대All pages
skipped_1page.pdfp2,skipped_2pages.pdfp2,skipped_2pages.pdfp3#388: 0 pages by default
The ground truth for dashes does not change, because
do_char_normalizationdefaults toTrue. These are examples withdo_char_normalization=False:elsevier-00.pdfp9:False339-349.339–349.863-865.863–865.1932-1937.1932–1937.19246029784032167-4.pdfp3:False-−-−I checked both modes on the 218 pages that previously changed only because of dashes. The default matches the stored ground truth on 217 pages.
Falsematches the earlier keep-the-dashes output on the same 217 pages. The other page,cropbox_versus_mediabox_02.pdfp3, differs in the same way in both modes, because my comparison script splits lines at the\x1echaracters in that page's text.#327: 1 pages in 1 documents
Ground truth changes because the ZapfDingbats font labelled
/Name /F3decodes through its builtin encoding (code 108 isa71, the black circle).annots_01.pdfp1:l●l●l●l●All pages
annots_01.pdfp1#389: 2 pages in 2 documents
Ground truth changes because bare hexadecimal glyph names (Skia
g%X/G%X) becomeGLYPH<name:…>markers instead of text.9acb62b4-4449-48d0-8127-f2be26349a6a-6.pdfp1:GDBGLYPH<name:GDB>GFEGLYPH<name:G27> GLYPH<name:G30>GLYPH<name:G23>G3BGLYPH<name:G31> GLYPH<name:G21>GLYPH<…GLYPH<name:GFE>GLYPH<name:G27> GLYPH<name:G30>GLYPH<name:G23>GLYPH<name:G3B>GLYPH<name:G3…GLYPH<name:G35>GLYPH<name:G59>G8BGB4GLYPH<name:G25>GLYPH<name:G44>GD8GA4G4AG3FGLYPH<name:…GLYPH<name:G35>GLYPH<name:G59>GLYPH<name:G8B>GLYPH<name:GB4>GLYPH<name:G25>GLYPH<name:G44…G2AGLYPH<name:G25>GLYPH<name:G33>GE8GLYPH<name:G48>GDFGAAGLYPH<name:G33>GLYPH<name:G67>GL…GLYPH<name:G2A>GLYPH<name:G25>GLYPH<name:G33>GLYPH<name:GE8>GLYPH<name:G48>GLYPH<name:GDF…G5CG3A G8BGLYPH<name:G24>GCBGBBGLYPH<name:G25>GF4GLYPH<name:G99>GLYPH<name:G44>G5DG5CGLYP…GLYPH<name:G5C>GLYPH<name:G3A> GLYPH<name:G8B>GLYPH<name:G24>GLYPH<name:GCB>GLYPH<name:GB…All pages
9acb62b4-4449-48d0-8127-f2be26349a6a-6.pdfp1,ba55e6a2-1a86-460d-bf2c-e01a6771657b-2.pdfp1#417: 11 pages in 9 documents
Ground truth changes because Symbol private-use values (U+F0xx) fold through the Symbol encoding.
16293520921922171261-1.pdfp2:••b766d7d18edf92b1_0002.pdfp1:= θ =1=1 θ γ - = + + ==eac6df0b2a6f9cf0_0002.pdfp1:(m)、内部摩擦角φs(deg)、砂礫密度(m)、内部摩擦角φs(deg)、砂礫密度σ)、流体密度)、流体密度ρ τ τ ≤ ( )( ) ( ) { } ρ ρ σ θ τ + + - + =All pages
11033738705194660817-2.pdfp15,11033738705194660817-2.pdfp16,13828292398120620566-1.pdfp1,16293520921922171261-1.pdfp1,16293520921922171261-1.pdfp2,16687286043211380927-1.pdfp15,6439951649404873078-1.pdfp1,8313873609013042575-4.pdfp5,b766d7d18edf92b1_0002.pdfp1,eac6df0b2a6f9cf0_0002.pdfp1,redp5110_sampled.pdfp14#322: 16 pages in 4 documents
Ground truth changes because symbolic fonts decode through their builtin encoding; the result agrees with pdfium's text extraction.
10017889119412695965-12.pdfp5:ˇGLYPH<240>ŁGLYPH<229>GLYPH<236>GLYPH<237>ŁŒ æŁªGLYPH<237>GLYPH<224>ºGLYPH<238>GLYPH<226>…Приемник сигналов пульта ДУ Дисплей Кнопка тестового включения Кнопка ВКЛ./ВЫКЛ. Кнопка б…˚GLYPH<237>GLYPH<238>GLYPH<239>ŒGLYPH<224> GLYPH<240>GLYPH<229>ªGLYPH<243>ºŁGLYPH<240>GLY…Кнопка регулировки˚GLYPH<237>GLYPH<238>GLYPH<239>ŒGLYPH<224> GLYPH<228>GLYPH<238>GLYPH<239>GLYPH<238>ºGLYPH…Кнопка дополнительных функций Кнопка режима работы Кнопка скорости вентилятора ÏÀÍÅËÜ ÓÏÐ…GLYPH<226>ߌºGLYPH<254>GLYPH<247>GLYPH<224>GLYPH<229>GLYPH<242>æGLYPH<255>. GLYPH<149> ˚G…âûêëþ÷àåòñÿ. • Êíîïêà ðåæèìà ðàáîòû: ïîçâîëÿåò âûáðàòü îäèí èç ïÿòè ðåæèìîâ ðàáîòû êîíäèö…GLYPH<192>GLYPH<226>GLYPH<242>GLYPH<238>GLYPH<236>GLYPH<224>GLYPH<242>ŁGLYPH<247>GLYPH<22…Àâòîìàòè÷åñêèé: êîíäèöèîíåð âûáèðàåò ðåæèì àâòîìàòè÷åñêè â çàâèñèìîñòè îò óñëîâèé (òåìïåð…5462902444445925132-4.pdfp1:"πœ¨¨¨¨˙»»»»ˇ¨¨¨¨œ»»»»œ¨¨¨¨˙»»»»GLYPH<246>˙˙˙˙ˆ««««4e5aa9238544d683_0009.pdfp1:GLYPH<32>GLYPH<54>:GLYPH<50>GLYPH<54>GLYPH<54>:GLYPH<50>GLYPH<54>GLYPH<32>7:GLYPH<32>GLYPH<52>7: GLYPH<52>GLYPH<32>7:GLYPH<32>87: 8GLYPH<32>7:GLYPH<49>GLYPH<51>7:GLYPH<49>GLYPH<51>GLYPH<32>7:GLYPH<49>GLYPH<53>7:GLYPH<49>GLYPH<53>All pages
10017889119412695965-12.pdfp11,10017889119412695965-12.pdfp2,10017889119412695965-12.pdfp4,10017889119412695965-12.pdfp5,10017889119412695965-12.pdfp9,10976580690960943929_004.pdfp1,10976580690960943929_004.pdfp12,10976580690960943929_004.pdfp4,10976580690960943929_004.pdfp7,10976580690960943929_004.pdfp9,4e5aa9238544d683_0009.pdfp1,5462902444445925132-4.pdfp1,5462902444445925132-4.pdfp2,5462902444445925132-4.pdfp3,5462902444445925132-4.pdfp4,5462902444445925132-4.pdfp5Effects to review
do_char_normalization=False). Two documents encode line-end hyphens as the minus sign:1322063424305812499-5.pdf(Greek, 133 words such asκα−) and11867823218465466809_003.pdf(68 words). WithFalse, those words end in U+2212, which docling's dehyphenation does not join. The default is not affected.10017889119412695965-12.pdfp2, p4, p9 and p11 change fromGLYPH<…>markers to text such asÑîäåðæàíèå, which is "Содержание" in Windows-1251 read as Latin-1. The font names its Cyrillic glyphs with Latin glyph names, and pdfium extracts the same text. On p5 the same document's other font decodes to correct Cyrillic (Приемник сигналов).For commit 6, I compared the changed pages with pdfium's text extraction:
10017889119412695965-12.pdfp2 and p5,5462902444445925132-4.pdfp1 (˙»»»»,œ»»»»,ˆ««««) and4e5aa9238544d683_0009.pdfp1 (7: 8) agree.Tests
tests/test_unit_*.py: 335 passed, 1 skipped.test_regression_parse.py,test_regression_threaded_parse.pyandtest_regression_threaded_render.pyagainst the pinned dataset with the regenerated parser ground truth: 66 passed.Checklist (#428):
main, no merge commits-#388, fix: consult base-font metrics for character decoding when the font dict carries a /Name label #327, fix: treat hexadecimal glyph-index names (Skia g%X) as unmapped glyphs #389, fix(fonts): fold Symbol-font private-use /ToUnicode values through the font encoding #417, fix: emit glyph marker for symbolic/composite fonts with no usable cmap #322)🤖 Generated with Claude Code