Repository navigation
feat(benchmarks): score OCR on olmOCR-Bench with its own tests - #457
burntcookiedough wants to merge 15 commits into
Conversation
Without MinerU installed, `--use-ocr` returned empty text, the reader dropped every PDF, and the run ended with "No documents found" and exit code 0, so a missing dependency looked like an empty folder. Raise ImportError with the install command instead. It has to be ImportError: SimpleDirectoryReader logs and skips a file on any other exception type.
Auto-detect only looked at the whole PDF, so a PDF with a normal text layer and a few scanned pages lost those pages. It now finds image-only pages and OCRs just those. Also: - add --ocr-tier, since basic is much faster than standard on CPU - OCR .png, .jpg, and .jpeg files when --use-ocr is set - say that the first OCR run downloads models - skip files with no text and name them, instead of indexing empty chunks
|
This answers the three objections that got the benchmark stripped from #190, and — the part that matters — it actually produces a measurement this time, which the previous version could not. Confirmed against the earlier review:
18/18 tests pass locally (5 benchmark + 13 from the OCR change). Merge ordering: this branch strictly contains #456 — I confirmed with Thanks for reporting the results as a rough guide rather than a published score, and for stating the hardware and sample size. 10 PDFs per split is small, and saying so is the right call. LGTM, after #456. |
62fdcaf to
30c99b4
Compare
An image has no text layer, so auto-detect now sends .png, .jpg and .jpeg files to OCR as well. Before, they loaded as empty documents and were skipped unless --use-ocr was set. Print the MinerU install hint only when a skipped file is a PDF or an image, since OCR cannot help an empty text file.
MinerU's Markdown embeds each figure as a base64 data URI. On the olmOCR-Bench sample, these made up 92% of all OCR output, which would fill the index with noise. Remove them after rendering. Captions and recognized text are separate blocks, so no text is lost.
The OCR reader returned no document for a file without text, so the file vanished without a message. Return the empty document instead, so drop_empty_documents names it with the other skipped files.
30c99b4 to
51f9ad6
Compare
|
Heads-up, since the numbers changed after your review: the script now scores the way olmOCR's To check it, I ran olmOCR's The table in the description now covers all four tiers. #456 also gained a fix that removes the base64 images MinerU embeds in its Markdown (22f68d6). They were 92% of the OCR text, and no score changed. |
Mixed PDFs put each OCR page back at its own position instead of appending all OCR text after the text layer. Scanned-page detection rotates image boxes to match the page, so rotated scans are found. Skipped files are reported by path, so two files with the same name in different folders are told apart.
51f9ad6 to
fafe980
Compare
olmOCR adds a baseline test to every PDF that lacks one and reports the average pass rate of each test file and the baseline group. The script pooled all tests instead and skipped the baseline, so it understated every tier by 6 to 9 points. Add the baseline group, report olmOCR's score, and add the advanced tier to the results. Running olmOCR's benchmark.py on the same OCR output gives the same score for standard and advanced.
fafe980 to
9d9b302
Compare
What does this PR do?
Adds
benchmarks/ocr_benchmark/, which scores LEANN's OCR path on olmOCR-Bench. Part of #158.The benchmark was removed from #190 because it could not produce a valid score. This version follows the approach @ASuresh0524 outlined in that review:
huggingface_hub, because the datasets-server API still returns HTTP 500 for this dataset.benchmark.pydoes: every PDF also gets olmOCR's baseline test, and the score is the average pass rate of the splits and the baseline group.This PR builds on #456, which adds the
tierargument it needs. Please review #456 first. Only the last five commits are new here.Results
10 PDFs per split (
--max-pdfs 10 --seed 0) on MinerU 4.0.12, Windows 11, and an RTX 3060 Laptop GPU (6 GB). This is a small sample, so it is a rough guide and not a published score. The math splits were not run, because they need a Playwright browser.flashbasicstandardadvancedold_scansmulti_columnlong_tiny_textheaders_footerstable_testsI checked the scoring by running olmOCR's
benchmark.pyon the same OCR output. It gives the same score forstandardandadvanced. Forflashandbasicit gives about 1.5 points more (66.3% and 74.6%), because each returned no text for oneheaders_footersPDF, and olmOCR lets that empty output pass the PDF's "absent" tests.advancedscored highest, mostly on tables, and took 4% longer thanstandard.standardstays the default, because it is MinerU's own default forparse().flashwrites tables as plain text instead of Markdown tables, so it fails almost every table test even though the cell text is there.GPU memory use peaked at 3.5 GB with
standardandadvancedand at 1.6 GB withflash, against 1.2 GB when idle.Related Issues
Part of #158.
Checklist
uv run pytest): the new tests pass, with and without MinerU installed. The full suite has the same failures asmainon my laptop, as described in feat(apps): OCR scanned pages and images in document RAG #456.ruff formatandruff check)pre-commit run --all-files)