Repository navigation
fix(render): stroke clipping, tiling patterns and soft masks, consolidated per #428 - #439
Merged
PeterStaar-IBM merged 10 commits intoNov 3, 2026
Merged
Conversation
An object painted under an ExtGState whose /SMask names a transparency group was painted fully opaque: the key was parsed to a three-value enum and the renderer had nothing to composite. InDesign builds drop shadows and feathered photo edges from this mask, so those effects came out as opaque boxes over the photos. The mask is now composited (ISO 32000-1, 11.6.5): - page_grph.h keeps the /SMask dictionary as a soft_mask_spec: the /G group, /S, /BC as device RGB and /TR sampled through pdf_function. - grph.h snapshots the CTM at the gs, which is the mask's coordinate system; /SMask /None clears it; the mask survives q/Q and reaches forms. - stream.h sub-decodes the group right after the gs, the way Do decodes a form, into containers of its own and once per entry and CTM. Every text, shape, bitmap and shading instruction painted under the mask shares the decoded soft_mask_instruction, like clip_state. - blend2d_renderer.h paints the group once per page over its backdrop, reads it back as an A8 plane through /TR (backdrop outside /BBox) and caches it per mask. A masked instruction re-enters its paint routine with page_context() redirected to a layer; the layer is multiplied by the plane and blitted with the instruction's blend mode. render_config.render_soft_masks switches the compositing off; the time spent is reported under render_page.soft_mask. A mask set inside a mask group, knockout and isolated attributes, and canvases above 8192 px per side stay unhandled and are reported. Fixes docling-project#384 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
A page can install a new soft mask with every `gs` for next to nothing, and each decoded mask cost an instruction list at parse time and a canvas-sized A8 plane at render time, so a hostile page could hold gigabytes. - The renderer keeps at most 256 MB of mask planes per page; past that the cache is dropped and planes are rebuilt as needed. - The decoder decodes at most 512 distinct masks per page; the rest of the page paints unmasked, reported once. - The CTM in the decode cache key is quantised to 1e-3, so float noise between two `gs` of the same mask does not make two decodes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
A transparency group painted under a soft mask is masked as a whole, and a mask that the group's content installs acts inside the group (ISO 32000-1, 11.6.6). The group's alpha and blend mode were already pushed down onto its content; the soft mask was not. A `gs` inside the group replaced the outer mask, so only the inner mask took effect, and `/SMask /None` inside the group removed the outer mask. The graphics state now keeps the mask in force at the `Do` as the group mask. Paints inside the group carry the intersection of the group mask and their own mask, and the renderer builds its plane as the product of the two planes. Intersections are cached per pair, like decoded masks. Reported by dlaszlo on docling-project#384: a 50% mask inside a group painted under a 50% mask now leaves a quarter of the paint, (191, 0, 64), as pdfium does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
The stroke branch of render_shape painted straight onto the page, so a stroke under a curved clip (or under one W over several rectangles) ran past the clip. The clip tests also used the path's own bbox: a horizontal or vertical rule has a zero-area bbox, the mask area came out empty, and the rule disappeared (a regression since 7.15.0). A stroke whose path lies just outside a rectangular clip, but whose width reaches inside it, was dropped for the same reason. - size the clip tests and the mask window by the stroke's reach - paint the stroke through the clip mask, like the fill - rasterise fill and stroke into an A8 coverage window and composite with fill_mask, instead of a PRGB32 layer - build the mask once over the clip box and reuse it while consecutive shapes share the clip; fall back to a per-shape mask when the clip box is too large to cache - build_clip_mask: treat a group of several rectangle subpaths as needing the mask, so a shape outside the union is not painted unclipped Fixes docling-project#378 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
Every fill whose colour is a tiling pattern replays the pattern cell on a lattice of up to 17x17 nested stream decoders. Without shape tracking the path being filled is never recorded, so the lattice falls back to the page box and runs in full for every fill, and every stroke it produces is then dropped by the same shape guards. A CAD sheet hatched with thousands of pattern fills took minutes to decode for text alone. Return early from the tiling branch when `keep_shapes` is false, the same way the shading-pattern branch already does, and parse the pattern cell once per resource instead of once per fill. Fixes docling-project#375 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
… the cell A tiling-pattern fill replayed the pattern cell on a lattice of up to 17x17 nested stream decoders, registering every cell stroke as a page shape and a render instruction. A CAD sheet hatched with thousands of regions produced millions of them and took minutes to decode with shapes kept, which the threaded backend always does for its page render. The cell is now decoded once per pattern (and per colour for an uncoloured one) into its own instruction list, in pattern space. A fill emits one shape instruction for its path carrying that cell as a tiling paint. The Blend2D renderer rasterises the cell into a tile at canvas resolution, caches it per page, and fills the path with a repeating BLPattern through the usual clip, mask and blend handling. Cells wider than their step are drawn again shifted by a step so the overlap wraps into the tile. Cells larger than the page keep the direct replay. The page keeps one shape item per hatched region, its outline, instead of the replayed strokes. Text inside a pattern cell is tile content and is no longer reported as page text. Refs docling-project#375 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
…ing-project#378 against pypdfium2 Three pages the fixes were shown on join the enforced pypdfium2 comparison, each with its own tolerance: the magazine cover whose photo is a transparency group under a soft mask with a second mask inside (mean absolute error 11.89 -> 1.58), a drop shadow under a title that was a solid grey block (4.41 -> 1.54), and a card whose rounded border ran past its curved clip (6.46 -> 5.03). All three fail on main. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
This was referenced Nov 2, 2026
Contributor
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
Contributor
|
✅ DCO Check Passed Thanks @wittjeff, all your commits are properly signed off. 🎉 |
PeterStaar-IBM
self-requested a review
November 3, 2026 04:59
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
5 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Renderer topic from #428: stroke clipping, tiling patterns and soft masks. This PR replaces #413, #397, #390 and #386, which are closed and link here. It has one commit per fix, rebased on
main(2fb6525), with no merge commits.Fixes #384. Fixes #378. Fixes #375. Refs docling-project/docling#4490.
/SMaskis composited through the mask group's luminosity or alpha, honouring/BBox,/BC,/TRand the CTM at thegs. Commit 2 bounds the per-page caches; commit 3 keeps the outer mask inside a masked transparency group.RenderConfig.render_soft_masksswitches it off.Wis clipped away instead of painted unclipped. Fill and stroke composite through A8 coverage windows and the clip mask is cached over the clip box.BLPattern, instead of being replayed on a lattice of up to 17x17 nested decoders.Changes made while consolidating
mainmoved the construction ofbitmap_instructionintopdf_state<BITMAP>::add_bitmap_instruction()(fix: decode image XObject samples only when a page draws them #425) and added the finer re-decode of reduced bitmaps (fix(render): decode reduced bitmaps again for a finer re-render #429): the soft mask is attached at the new site, andrender_bitmap()tests the soft mask before the re-decode, so a re-rendered image is masked once (the masked paint re-entersrender_bitmap()withcontext_override_set, which disables the mask inside the layer).multiply_a8_by_a8()sits next to fix(render): composite ExtGState soft masks (/SMask) #413'smultiply_prgb32_by_a8_window();render_shape()keeps fix(render): composite ExtGState soft masks (/SMask) #413's soft-mask entry above fix(render): clip strokes through non-rectangular clip masks #397's coverage-window body.render_shape(). feat(render): tile pattern fills in the renderer instead of replaying the cell #390 filled a pattern through the old PRGB32 clip layer; here the tile is aBLPatternset as the context's fill style, and the clipped fill composites it withfill_mask(origin, coverage)(no colour argument) while a plain fill keepsfill_mask(origin, coverage, colour).shape_instructioncarries the soft mask and the tiling paint side by side.HF_DATASET_REVISIONto regenerated ground truth (feat(render): tile pattern fills in the renderer instead of replaying the cell #390'sbb2ca62, fix(render): composite ExtGState soft masks (/SMask) #413'sbfb443c) are not included (Consolidating the open rendering, text and font PRs: scope and evidence requirements #428 section 4).Evidence
Rendering is measured as
tests/test_regression_pypdfium_render.pymeasures it: every sampled page of the regression corpus at the pinned dataset revision (6774fc0, 1968 pages), rendered by this branch and bymain(2fb6525) with the same binary configuration, compared with pypdfium2 at the test's scale. The per-page table, the CSVs and the three-panel images are on theevidence/renderer-topicbranch of my fork.Corpus-wide. 43 of 1968 pages move: 27 closer to pypdfium2, 16 further. No page gets worse by more than the tolerance (9.0 / 0.09), none moves from inside it to outside, and no page changes render status. The largest move away is +0.01 mean absolute error, on pages of
11033738705194660817-2.pdfwhose background pattern is now one tile instead of replayed bitmaps. The full list is incompare.md.Cited pages (mean_abs_error / changed_pixels_ratio against pypdfium2):
complex_invisible_fonts_01.pdfp1, magazine cover, photo in a transparency group under a mask with a second mask inside17898535961965146676-4.pdfp1, drop shadow under a title, was a solid grey block14327672304519301096-2.pdfp2, signal-strength bars, were solid black921558209587984065-1.pdfp1, rounded card border stroked under a curved clip15424333971388669153-2.pdfp4, table rules under clips14570626672493314948-53.pdfp34, table rules under clips7143857365833452481-1.pdfp1, dashed border pdfium clips awayPDF32000_2008.pdfp752, Annex L tiling figuresc9612bab-d235-4788-8194-6cc78b4d7456-1.pdfp1, vector hatches11273518440839632455_002.pdfp1, image-cell hatchesEnforced test. Commit 7 adds three cases to
tests/test_regression_pypdfium_cited_pages.py: the cover (extgstate-soft-mask, limits 3.0 / 0.05), the drop shadow (soft-masked-drop-shadow, 2.5 / 0.05) and the card border (stroke-through-curved-clip, 5.8 / 0.142). All three fail onmainand pass here; the four existing cases still pass.Tiling patterns and pixels. Commits 5 and 6 change how a pattern fill is produced, not what it looks like: the pattern pages move by at most 0.01 against pypdfium2, so no image test can fail on
mainfor them. What changes is the cost and the page items. On real pages of the corpus, decode plus render through the threaded parser with shapes kept (one thread, scale 1, median of 3),main→ this branch:c9612bab-d235-4788-8194-6cc78b4d7456-1.pdfp1, vector hatchesPDF32000_2008.pdfp752, Annex L tiling figures3763025099401294496-2.pdfp2, vector hatches11273518440839632455_002.pdfp1, image-cell hatches14570626672493314948-53.pdfp34, image-cell hatchesThe corpus has no page like the reporter's A1 CAD sheet (#375, docling#4490), where the lattice replay took minutes; #390's synthetic measurement of that case (500 hatched regions, 30-stroke cell: 185 s → 1.4 s with shapes kept) still holds for commit 6, and #386's for the text-only decode. Text inside a cell. Commit 6 stops reporting text drawn inside a tiling cell as page text; the lattice replay had reported it once per placement. I checked what that costs. In the whole regression dataset and in 5,366 PDFs on my machine, the only tiling cells that draw text are the Annex L figures of
PDF32000_2008.pdf(and pdfium's test-corpus transcriptions of them): four ZapfDingbats suit glyphs per cell. On page 745,mainreports 344 suit-glyph words from the lattice replay plus the 4 words of the figure's enlarged illustration, which is ordinary page text; this branch reports the 4 illustration words and none from the cells. pdfium, MuPDF and poppler each extract exactly those 8 illustration glyphs and nothing from the cells, so the branch agrees with all three. The unit tests of #390 enforce one shape item per hatched region with the region still painted at the cell's coverage, the replay fallback for a page-sized cell, a cell wider than its step, an image cell reported once, and a cell that paints with its own pattern terminating.Ground truth
Listed as #428 section 4 asks; the maintainers update the ground truth when merging. I did not pin a regenerated revision.
Render ground truth. One page leaves the render tolerance (10.0 / 0.15) of
test_rendered_pages_match_groundtruth, and it is the one the soft-mask fix is about:complex_invisible_fonts_01.pdfp1, mean absolute error 10.54 against the storedfull_page.png(which was rendered without the mask). Its new rendering is 1.58 from pypdfium2. Three other pages are above the tolerance in that run,7829021aca2be7b4_0002.pdfp1,15523818099946337472-5.pdfp5 and1020010076157-4.pdfp3, but they render identically onmainand on this branch (they do not appear among the 43 pages whose pypdfium2 metrics move), so they are not caused by this PR.test_render_reference_documents_from_filenamesalso fails, at its first mismatch, on the bitmap metadata of11033738705194660817-2.pdf("bitmap resource count mismatch: expected 4, got 1"): the pattern-cell images below are now one page image per fill. The pages are the ones in the parser list.Parser ground truth. 44 of the 950 sampled pages change, in 7 documents, all from commits 5 and 6 (tiling patterns) and all in the page items (
verify_SegmentedPdfPage: shape and bitmap counts), not in the text, except where noted. This is the same list #390 reported.PDF32000_2008.pdfc9612bab-d235-4788-8194-6cc78b4d7456-1.pdf3763025099401294496-2.pdf11033738705194660817-2.pdf11273518440839632455_002.pdf14570626672493314948-53.pdf14911938403731335361-1.pdftest_threaded_reference_documents_from_filenamesstops at the same first mismatch (11033738705194660817-2.pdf). Everything else in the two parse suites passes (15 of 17 tests), as do the 24 other render tests and the unit lane (357 passed, 1 skipped).Known limits
🤖 Generated with Claude Code