[wittjeff]

Summary

/ToUnicode source codes from 0xD800 to 0xDFFF were lost. A second mapping was also overwritten.

cmap_parser::get_source() decoded each source code as UTF-16 text (to_utf8 → utf8::utf16to8). It then read the key back with utf8::next. A source code is a byte sequence, not text, and the values 0xD800–0xDFFF are lone surrogates in UTF-16. For those codes:

1. utf16to8 throws, and the catch returns the string "GLYPH<cmap:dfa8>".
2. set_mapping / set_range read the first character of that string as the key: 'G' = 0x47.

The result is that the code itself has no mapping, and code 0x47 gets the text of the bad entry. Large CJK fonts use this range a lot. Before 7.16.0, the missing mapping also crashed the parse with RuntimeError: Invalid code point (docling#1111). The is_code_point_valid guard from #325 stopped the crash, but the mapping was still lost.

The fix reads source codes as big-endian integers with the existing to_uint32(). codespacerange bounds are already read this way. set_mapping and both set_range overloads now take integer codes. For codes outside the surrogate range, the keys are the same as before.

Fixes #145. Fixes #133.

Evidence

The issue files from docling#1111, parsed with 7.22.1 and with this branch (characters that come out as U+FFFD):

file	7.22.1	this branch
pranger-3.pdf p1 (NotoSansCJKjp, <DC77>)	1	0
NASA_23_p1.pdf p1 (<DFA7>–<DFAC> = curly quotes)	8	0
default.pdf p265 (Haansoft Batang, <DFCF>, <DFA2>)	1	0

On NASA_23_p1.pdf, the <0047> <0020> mapping was overwritten four times, so every word space in that font came out as ”. With this branch, the text matches pdftotext: 대략 ‘23m 전후’, 직경은 대략 ‘2m 전후’.

Not fixed by this PR: 25.8._._0807.1.pdf pages 1–2 use codes such as <DF2B> that are not in their /ToUnicode at all (the CMap has 37 entries, the highest is <526C>). pdftotext also gets no text from these runs.

Behavior change

A 4-byte source code now keeps all 32 bits. Before, utf16to8 masked it to 16 bits. A 1- or 2-byte code that is not a surrogate gives the same key as before.

Tests

• tests/test_unit_cmap_encoding.py: two new tests, for a bfchar source code and for bfrange bounds in the surrogate range. Both fail on 7.22.1 and pass with this branch.
• Full suite (local, Linux x86_64, test data at the pinned revision): 398 passed, 1 skipped, 3 failed. All three failures are the groundtruth difference described below. They are test_regression_parse.py::test_reference_documents_from_filenames, test_regression_threaded_parse.py::test_threaded_reference_documents_from_filenames and test_regression_threaded_render.py::test_render_reference_documents_from_filenames.
• In test_reference_documents_from_filenames, 9 page/mode checks differ, on skipped_1page.pdf p2 and skipped_2pages.pdf p2–p3. In every case, the groundtruth at the pinned revision 6d3458e2 holds the bug output (GLYPH<c=55938,font=/AAAAAI+NotoSansCJKkr-Bold>, code 0xDA82), and this branch gives the correct text (블록체인, 헬스케어). pdftotext gives the same text. No other page in the corpus changed. The groundtruth for these 3 pages needs to be regenerated. I can prepare the dataset PR if you want me to.

🤖 Generated with Claude Code

[github-actions]

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[cau-git]

Thanks for this contribution. This PR is on hold until it has been consolidated as described in #428. Please resubmit it as part of the topic PR listed there, with the evidence the issue asks for.

[wittjeff]

Superseded by #431, which consolidates the font and ToUnicode decoding fixes as described in #428. This fix is commit 1 of 6 there, rebased on current main.