[wittjeff]

Re-scoped on 2026-10-05: #355 was fixed by #361 (7.22.1). This PR now fixes the regression that #361's flat cap introduced on letter-spaced text.

Problem

Since 7.22.1, contract() caps the unmarked word-gap threshold at a flat 0.35 advance on every line. Letter spacing opens every character gap on a line, often past 0.35, so tracked text splits into letters. Line cells:

file	7.22.0	7.22.2 (main)	this PR
elsevier-00.pdf p1	ARTICLE INFO	A R T I C L E I N F O	ARTICLE INFO
14289803404128846560-14.pdf p14	Tinta	T i nt a	Tinta
14289803404128846560-14.pdf		Ga b i ne t e de Ap o i o à s F r e g ue s i a s	Gabinete de Apoio às Freguesias
2305.03393v1-pg9.pdf (#355)	HyperParameterOptimization (7.21–7.22.0)	5.1 Hyper Parameter Optimization	5.1 Hyper Parameter Optimization

Change

infer_word_gap() keeps the cap, but lets it rise with the line's own letter spacing:

cap = max(0.35, 1.5 × median + 0.20)

The median is taken over the gaps between non-dot glyphs that fall below the word threshold. Leader dots are spaced like words and would otherwise pass for tracking (maximum mass of vehicle: . . . . collapsed to maximummassofvehicle: without that exclusion). The cap only rises when at least three such gaps exist: a single gap, as between a lone math symbol or one table digit and its neighbour, is no evidence of tracking.

• Runs without space glyphs: threshold = min(k-means midpoint, cap). On a line that is not letter-spaced this is exactly main's min(0.35, midpoint).
• Runs with space glyphs: threshold = min(1.0, cap), in place of the flat 0.35 from fix: updated the page_item_sanitators/cells.h and its associated ground-truth #361.

Tests:

• tests/test_unit_word_gap_threshold.py: a Tc-tracked heading placed with TJ offsets, the same heading with a space glyph, and the LaTeX section heading from Words lose their separating spaces in 7.21.0 ("Related Work" -> "RelatedWork"), bisected to #351 #355. The two tracked cases fail on main.
• tests/test_regression_cell_contraction.py::test_letter_spaced_text_stays_whole: article info in elsevier-00.pdf and Tinta in 14289803404128846560-14.pdf.

Results

Regression dataset, line cells compared with main (7.22.2), 447 files:

	lines	files	pages
boundaries removed	926	33	82
boundaries added	0	0	0

• None of the 926 merges glues two dictionary words together; 109 join letters into a dictionary word (Subst i tut ing → Substituting, heff ingen → heffingen, C H A P T E R 5 → CHAPTER 5, F INANCI ALS → FINANCIALS).
• 665 of them restore the 7.22.0 line text exactly. Most of the rest are the Portuguese letter-spaced file above (617 lines in all), where 7.22.0 had its own partial splits.
• Merges that are not tracked text: isolated math and music glyphs (F t dt → Ftdt in 19246029784032167-4.pdf, œ œ œ note heads in 4561257296328379730-2.pdf), and a Japanese name list where 7.22.0 also merged (岩 田 守 弘 → 岩田 守弘).
• Every line listed in Words lose their separating spaces in 7.21.0 ("Related Work" -> "RelatedWork"), bisected to #351 #355 is unchanged from main.

Ground truth. The 82 pages above are the ones whose line cells differ from main, so test_regression_parse.py fails on them and the dataset revision would need regenerating. I have not touched tests/constants.py.

Not addressed

• a b s t r a c t in elsevier-00.pdf becomes abs trac t: the Tc on that line puts two gaps above the cap. Still closer than main.
• The single-digit grid in 10458169359009391401-5.pdf stays as on main (the three-gap rule keeps it), although 7.22.0 merged some of its cells.

🤖 Generated with Claude Code

[github-actions]

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[PeterStaar-IBM]

@wittjeff I think this has been fixed by now, no?

[wittjeff]

@PeterStaar-IBM yes, #361 fixes #355: on 7.22.2 the page-9 heading reads 5.1 Hyper Parameter Optimization, and the section-heading test from this PR passes on main.

One thing this PR's second test surfaces: the flat 0.35 cap in contract() splits letter-spaced headings. elsevier-00.pdf's ARTICLE INFO was whole in 7.22.0 and is A R T I C L E I N F O in 7.22.1+ (it was also split in 7.20.0). I'm re-scoping this PR to that regression: replace the flat cap with the median-relative one, which handles both the section-number gap and tracked text. I'll update the description and numbers against current main.

[wittjeff]

Re-scoped and pushed (579ef02). The flat 0.35 cap now rises with the line's own letter spacing: max(0.35, 1.5 × median letter gap + 0.20), with gaps next to leader dots excluded and at least three letter gaps required. On a line that is not letter-spaced the threshold is exactly main's.

Against main on the regression set, the change only removes boundaries: 926 lines in 33 files (82 pages), none gluing two dictionary words, 665 of them back to their 7.22.0 text. No new boundaries anywhere, and every line from #355 is unchanged. Description and tables updated; the ground truth for those 82 pages would need regenerating.

[cau-git]

Thanks for this contribution. This PR is on hold until it has been consolidated as described in #428. Please resubmit it as part of the topic PR listed there, with the evidence the issue asks for.

[wittjeff]

Superseded by #437, the word and cell segmentation topic PR from #428. It carries this change as one commit, rebased on main, with the evidence and the ground-truth page list in one place.