Repository navigation
fix: fall back from noncharacter ToUnicode mappings - #395
Merged
PeterStaar-IBM merged 1 commit intoOct 23, 2026
Merged
Conversation
Since docling-project#296 the /ToUnicode cmap wins over /Differences glyph names. Some LaTeX Type1 subsets (eg LinLibertine in arXiv 2408.09869v5) map the ligature codes /f_f, /f_i and /f_f_i to <ffff>, so every ligature came out as U+FFFF. docling-project#352 already treats U+0000 as unusable; extend that to U+FFFD and the Unicode noncharacters (U+FDD0..U+FDEF, U+xFFFE/U+xFFFF) so the glyph-name recovery applies again. Fixes docling-project#394 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
Contributor
|
✅ DCO Check Passed Thanks @wittjeff, all your commits are properly signed off. 🎉 |
Contributor
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
PeterStaar-IBM
approved these changes
Oct 23, 2026
wittjeff
added a commit
to wittjeff/docling-parse
that referenced
this pull request
Oct 27, 2026
…tests The tests added with docling-project#395 expected U+2212 to fold to "-". This PR keeps the minus sign, so the three minus cases now expect U+2212. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #394.
Problem
Since #296 (7.7.1),
init_differences()resolves each/Differencescode through the/ToUnicodecmap first and uses the glyph name only whenhas_to_unicode()rejects the cmap entry. That check rejects empty entries, U+FFFD and (since #352) U+0000. It accepts U+FFFF. The LinLibertine Type1 subsets in arXiv 2408.09869v5 map/f_f,/f_iand/f_f_ito<ffff>, so those codes never reach the existing ligature decomposition. On that file, 30 text-line cells on pages 7–9 contain U+FFFF (e�ort,speci�c,A�liation).Fix
has_to_unicode()decodes the cmap value. When the value is a single code point and that code point is U+0000, U+FFFD or a Unicode noncharacter (U+FDD0..U+FDEF, U+xFFFE/U+xFFFF in every plane), the code falls through to the glyph-name methods, the same way #352 handles U+0000. Every other/ToUnicodevalue stays authoritative, including multi-character values.Verification
A large effort went,very specific,Scientific Articles,Author and Affiliation,annotation staff.tests/test_unit_nul_mapping.py: 9 parametrized cases (<FFFF>,<FFFE>,<FDD0>×/f_f_i,/f_i,/minus). They fail on the 7.22.2 wheel and pass with this change.tests/test_unit_*.py: 241 passed, 6 skipped.tests/test_regression_parse.py,test_regression_embedded_fonts.py,test_regression_cell_contraction.pyagainst a fresh copy of the pinned HF dataset: 54 passed, no groundtruth change.Limit: a font without
/Differences(for example a CID font with Identity-H) that maps a code to U+FFFF has no glyph name to fall back to, so it keeps the current output.🤖 Generated with Claude Code