[wittjeff]

Summary

The cell text sanitizer folds the U+2018..U+201F quote block to ASCII before any text leaves the parser, and mapped the four double quotes “ ” „ ‟ (U+201C, U+201D, U+201E, U+201F) to ' alongside the single quotes. That changes meaning: 24” x 36” (inches) came out as 24' x 36' (feet). The fold runs outside the do_sanitization flag and the exported orig field is a copy of the folded text, so nothing downstream could recover the character.

Fixes #335. Also the root cause of docling-project/docling#1433 (Apr 2025), where the parser was identified but no docling-parse issue was opened.

Cause

text_constants::replacements in src/parse/pdf_sanitators/constants.h. The whole block dates to #94, which gives no reason for folding double quotes to an apostrophe. docling's own PageAssembleModel.sanitize_text already maps “ ” to ", so the parser was the layer out of step with its consumer.

Change

Map U+201C, U+201D, U+201E, U+201F to ". The single-quote entries (U+2018, U+2019, U+201B to ') and U+201A to , are unchanged. Four lines.

Test

tests/test_unit_quote_sanitization.py, self-contained via tests/pdf_builder.py:

• core-14 Helvetica with /WinAnsiEncoding and no embedded program, setting 0x91..0x94 and 0x84 (quoteleft, quoteright, quotedblleft, quotedblright, quotedblbase), so the glyph decode is not in question;
• a non-embedded Identity-H font whose /ToUnicode bfrange covers the whole block, so U+201B and U+201F are exercised too;
• an orig == text check.

Against 7.19.0 the WinAnsi tests fail with ["24' x 36'"] == ['24" x 36"']; with the change all five pass and the 169 existing test_unit_* tests are unaffected.

The issue's own repro, after the change:

24" x 36", 10' poles, it's

Regression groundtruth

Not regenerated here: the HF corpus was unreachable from my environment. Pages whose source PDFs contain curly double quotes will change ' to " in their cell text; at most 26 of the 110 parser groundtruth pages contain a ' at all. Happy to push the dataset update once CI shows the affected set, or leave it to whoever normally bumps HF_DATASET_REVISION.

Downstream, a docling-parse bump carrying this will shift docling's PDF groundtruth in the same way.

🤖 Generated with Claude Code

https://claude.ai/code/session_01X7AdKLZDGfNTS2LJGmhehR

[github-actions]

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[wittjeff]

Following up on the groundtruth question from the description now that I can reach the corpus.

I ran tests/test_regression_parse.py::test_reference_documents_from_filenames locally on this branch against regression-dataset-for-docling-parse @ 571f34b:

• 148 pages in 77 documents differ from the stored groundtruth, and every one of them differs only by ' → " at the same positions in the cell text — verified cell by cell after re-parsing each page, not just from the failure table; no cell-count changes, no other characters, and apostrophes that should stay ' (sull'ordinamento, NEHA's) do. The list is in the details block below.
• Nothing else in the suite changes.

The regression job on this PR has not run yet: the "Run CI" workflow run is sitting in action_required, so it needs a maintainer to approve it for the fork.

For the groundtruth update: I can regenerate the affected pages with --update-groundtruth and open the dataset PR, or leave it to whoever normally bumps HF_DATASET_REVISION — whichever you prefer. Downstream, docling's own PDF groundtruth will move the same way once it picks up a docling-parse release with this in it.

Affected pages (77 documents, 148 pages)

10003567098752154021-14.pdf  pages 5, 6, 7, 15, 18
1020000062573-15.pdf  pages 13
1020187007810-230.pdf  pages 294
10358507436534803868_011.pdf  pages 1
10446922681186632184_002.pdf  pages 14
10458169359009391401-5.pdf  pages 3, 4, 5, 6, 7
10588776208842536523_006.pdf  pages 2
10749817875312063576_009.pdf  pages 9, 10
10989047281457174669_003.pdf  pages 1
11033738705194660817-2.pdf  pages 10, 11, 16
11077159831207327532_003.pdf  pages 3
1111108925592240862_007.pdf  pages 1
11315887125936234472_013.pdf  pages 3, 6, 10, 13
11516920924344231708_004.pdf  pages 3, 4, 6
11586287754644715199-16.pdf  pages 28, 30
11816493302388412838-4.pdf  pages 1, 2, 6, 7, 8
11831013430589524848.pdf  pages 1
11926187848345237766-2.pdf  pages 1, 2
11939885835628903831_004.pdf  pages 3, 6
12066842041297507170_028.pdf  pages 33
12089643962602557549_003.pdf  pages 14, 15, 18, 19
12134468270109370579_001.pdf  pages 3
12345475637115547305-1.pdf  pages 1, 2
13570074629782512488-1.pdf  pages 1, 2, 3, 5
13728511855768238546-20.pdf  pages 10, 14, 16, 35
13785621751355796362-1.pdf  pages 1
1407778270224909650-1.pdf  pages 2, 3
14267954770681214922-1.pdf  pages 1
14289803404128846560-14.pdf  pages 2, 3, 5, 10, 12
14570626672493314948-53.pdf  pages 35
15044306777075682274-3.pdf  pages 1, 2, 3
15523818099946337472-5.pdf  pages 2, 5
15910972127254098494-3.pdf  pages 2, 5, 6, 7, 8
16293520921922171261-1.pdf  pages 1, 3
16680759319942965852-8.pdf  pages 4, 5, 7, 8
16687286043211380927-1.pdf  pages 15, 17
17068186561688296387-1.pdf  pages 1
17748028314546730327-1.pdf  pages 1
18392306937729366053-2.pdf  pages 2, 3
19246029784032167-4.pdf  pages 6
2203.01017v2.pdf  pages 2, 8, 12
2206.01062.pdf  pages 3, 4, 7
2213713933117287130-1.pdf  pages 1
2881248815239273277-1.pdf  pages 1
3848385316711437784-4.pdf  pages 1, 2, 3, 4
4865216256588543301.pdf  pages 5, 6, 7, 8
4c9bb33c9466bc71_0001.pdf  pages 1
5462902444445925132-4.pdf  pages 1, 3, 4, 5
5626934442963300514-10.pdf  pages 16
5a2e3eec-f6a1-487a-9929-35f9f1661679-1.pdf  pages 1
5d969923927df690_0006.pdf  pages 1
6442bc6d-74bb-4310-981e-66dc381ccc2e-3.pdf  pages 1
6480366468566741514.pdf  pages 1
762975394748001135-1.pdf  pages 1
7829021aca2be7b4_0002.pdf  pages 1
7949049574853423109-3.pdf  pages 1, 2, 3, 4
8793130488931799579-2.pdf  pages 1
9402bf128c06d7b1_0001.pdf  pages 1
94e3fe046fa21a9c_0001.pdf  pages 1
9835735744180501405-1.pdf  pages 1
985a9a85d020ac41_0003.pdf  pages 1
annots_01.pdf  pages 1
c8702e75db7ff148_0001.pdf  pages 1
complex_invisible_fonts_02.pdf  pages 1
complex_invisible_fonts_04.pdf  pages 1
complex_invisible_fonts_05.pdf  pages 1
df49758bb5df51b4_0002.pdf  pages 1
dln_123cd9c15dca0dbc8147101bae1c2d1829ac8b15fcfc5dc319ce3a49f3cbf7d9.pdf  pages 1
dln_51b441c0038311652fc5a461aa2ccabf0558992b447dfa2ae91df4d801460ffe.pdf  pages 1
dln_835d4dd76390a0ccb3d5b257f3ce021d3a03e2b1249d68558821930fb8bdf069.pdf  pages 1
dln_bf963befd079febf2803a959166b7c61251d9fcc0a8fdaf779bd0101532c3cde.pdf  pages 1
dln_cfb45c751397e1d2aae56e6c17ef7b4f0317086bfc80bdd0e487d95bf083093c.pdf  pages 1
e6f72c8639b5513c_0012.pdf  pages 1
e7a3a534-9609-4dc9-9446-2d3a13c0b2ff-5.pdf  pages 1
indexed_device_n.pdf  pages 1
ligatures_01.pdf  pages 1, 3, 4
normal_4pages.pdf  pages 1, 2, 3

[wittjeff]

Dataset groundtruth PR for the 148 pages above, regenerated on 4eaebf4 and verified by hash to touch nothing else: https://huggingface.co/datasets/docling-project/regression-dataset-for-docling-parse/discussions/5

Note for sequencing: 5462902444445925132-4.pdf pages 1, 3, 4, 5 are also changed by the groundtruth for #322 (dataset PR 4); whichever of the two lands second needs those four pages regenerated once more on a build carrying both fixes.

[PeterStaar-IBM]

lgtm!

[wittjeff]

Rebased onto main at 7.22.0 (95d24e3; no code change, still just the sanitizer table and the unit test) and regenerated the groundtruth against the current dataset revision 65fd81c7. Dataset discussion #5 was generated on 571f34b1 and would have undone the word/line bbox precision from #351, so it is closed in favour of:

https://huggingface.co/datasets/docling-project/regression-dataset-for-docling-parse/discussions/6

588 files, 147 pages in 76 documents. Verified file by file rather than by hash alone: the .json.gz were decompressed and compared structurally, the text exports character by character, and every difference is ' becoming " at the same position. Nothing else under groundtruth/parser/ moves. Once that lands, HF_DATASET_REVISION on this branch needs to point at the merge commit; I can push that bump or leave it to you, whichever you prefer.

One page fewer than the list above: 6480366468566741514.pdf page 1. Its „Heute: KOHLHAAS“ used to fold to 'Heute: KOHLHAAS', but since #352 main decodes those two glyphs as ãHeute: KOHLHAASÒ. 0xE3 and 0xD2 are „ and “ in Mac Roman but ã and Ò in Latin-1, so the null-cmap fallback looks like it picks the wrong base table for that font. Unrelated to this PR; I will look at it separately.

Also noticed while diffing: dataset main carries 753 *.cells.delta.txt / *.word.delta.txt files under groundtruth/parser/. Those are the mismatch diagnostics test_regression_parse.py writes and deletes on success, so they probably got swept up in 3826b7b7. Harmless for the tests, just noise.