Repository navigation
Conversation
Students ask in everyday words and course material answers in its own terms, so the passage that holds the answer never reaches the window. keyword_terms_for_query also drops any query term whose document frequency is zero, so a casual word that appears nowhere in the collection is discarded before BM25 runs. Preparation can now write one everyday-language sentence for each passage and store it as that chunk's search terms. The sentence is indexed for keyword search and embedded with the passage; chunk text is never modified, so the stored source, the source panel and the answer window are unchanged and no chunk is added or displaced. The option is off by default, works in Basic and AI preparation, needs a preparation model, caches by passage text so an unchanged document makes no model call again, and never fails preparation when the model declines. Measured on a generated database textbook with gemma4:e4b writing the sentences and 56 casual student phrasings scored strictly: retrieval 40 to 45 of 56 with every prose passage given a sentence, 28 of 28 textbook-worded controls unchanged, and answers 41 to 47 of 56. Preparation costs about 3 seconds per passage once. See docs/plain-language-search-terms.md for the limits and the negative results. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
I ran an independent holdout evaluation of this PR at What the independent test found
The casual retrieval gain (+8.3 percentage points) is close to the PR's full-enrichment result (+8.9 points). The answer test had two wins and one regression, a net +4.2 points; its paired bootstrap interval was wide (-8.3 to +16.7 points). Partial-or-complete supported answers were 19/24 versus 21/24, so the strict complete-answer metric should not be read as saying every other answer was useless. Original passage text stayed identical, and all 48 answers completed without truncation. Limitations: these were assistant-authored questions and assistant scoring of blinded A/B answers, not actual student submissions or an independent human panel. This is one textbook and one local model pair. We checked for exact overlap with existing local benchmark strings, but could not check your private question sets or model pretraining. The samples and grading rules differ, so the answer-rate difference is not a controlled comparison between the two evaluations. Suggested improvements to this PR
My recommendation is to retain the optional design, clarify the targeted answer result and end-to-end cost, and strengthen the full-corpus holdout evaluation. The independent result supports “this can help”; it does not yet establish that the usual answer improvement repays the preparation time. |
|
The complete independent evaluation is now available for reproduction and regrading: Evaluation package and step-by-step README It is published on
To start: git clone --single-branch --branch codex/pr157-independent-evaluation https://github.com/georgia-tech-db/TokenSmith.git TokenSmith-pr157-eval
cd TokenSmith-pr157-eval
python3 tests/experiments/pr157-holdout-2026-10-14/verify.py --source .That audit needs only standard Python and makes no model calls. The README then covers dependencies, exact model digests, and The publication checks passed: all frozen input hashes match, the reported statistics reproduce, and all 48 recorded prompts, source counts, and budget limits recreate offline. Local machine paths and unrelated model inventory were removed from metadata; the changes are documented. For an independent regrade, use |
|
Follow-up to the review. I kept the feature as it was (one plain-language sentence per eligible passage, stored in
Retrieval on your holdout (fresh profiles, same models, my machine), casual questions, target / complete:
Paired against the PR as reviewed: +5/-1 target at four sources, +3/-0 at eight. Bootstrap by concept 0 to 9, so the interval touches zero at 24 concepts. Technical controls unchanged. Query latency unchanged (median 15 ms either way). Retrieval on a larger set. 120 new casual phrasings, five per concept, generated from the original passages with the same gold groups and criteria:
Paired against the PR at four sources: +8/-2 target and +11/-3 complete, bootstrap intervals by concept 0 to 13 and 1 to 15. Answers on the larger set, gemma4:e4b at four sources, 360 answers graded blind against the frozen criteria with arm labels hidden (0 wrong or unsupported, 1 partial, 2 complete):
Every row moves the right way. Against the plain app the feature with this change raises complete answers from 30 to 40 and cuts wrong answers from 42 to 26. What the change itself adds on top of the PR is at the bottom of the table: seven wrong answers become partial ones (33 to 26), and the count of complete answers holds (39 to 40, paired +9/-8). So the extra passages the change brings into the window stop the model from answering wrongly more often than they produce a fully complete answer, which is the expected shape when retrieval improves and the answering step stays the same. Twenty answers were identical across arms and were scored by different graders, and all twenty scores matched. On your original 24 pairs the picture is consistent but too small to separate from grader noise: my first grading gave 16 / 18 / 20 complete at four sources and 17 / 22 / 22 at eight, second formulation 14 / 13 / 17, and a second blind pass over all 288 answers kept the direction in every cell with net differences of two or less. Things I tested and did not adopt: three sentences per passage, three student-style questions, a more distinctive vocabulary, half a sentence, a separate embedding for the sentence, BM25 column weights, a larger keyword-term budget. All within the one-to-two question noise of regenerating the sentence, several worse. The one stable regression is the page I/O question, where a distractor's sentence contains two of the query's words. Baseline reproduced your numbers exactly. The PR arm differs by one or two because the sentences were regenerated (4 of 251 identical, Ollama 0.35.1 vs 0.31.1). Full packet, patch, blind packets and scores are in the holdout folder on my machine and I can push them to the evaluation branch if useful. Baseline reproduced your numbers exactly. The PR arm differs by one or two because the sentences were regenerated (4 of 251 identical, Ollama 0.35.1 vs 0.31.1). The 120 phrasings, the sentences the on profile held, the retrieval results, all 360 answers, the blind mapping and scores, the grader prompt, the second grading of your 288 holdout answers, and the scripts to rerun it are in |
Summary
Students ask in everyday words and course material answers in its own terms, so the passage that holds the answer never reaches the window.
keyword_terms_for_queryalso drops any query term whose document frequency is zero, so a casual word that appears nowhere in the collection is discarded before BM25 runs.Preparation can now write one everyday-language sentence for each passage and store it as that chunk's search terms. The sentence is indexed for keyword search and embedded with the passage.
chunk_textis never modified, so the stored source, the source panel and the answer window are unchanged, and no chunk is added or displaced. Off by default.This is the follow-up to the closed bridge-chunks PR, built the way that review suggested: write into the existing passage instead of adding a generated one. The earlier approach failed structurally, because a generated passage competes for a window slot and the embedder cannot tell the right one from the wrong one (admitted on 120 of 128 questions, off topic on 92).
What changed
tokensmith_preparation.py:plain_language_termswrites one sentence per source unit from that unit's own text, through the existingcompletion_clientwith a JSON schema, cached by passage text under the job directory like boundaries. Sentences that describe the passage instead of rephrasing it are dropped.tokensmith_preparation_job.py: runs the step in both Basic and AI mode when the option is on, before the report is written.tokensmith_store.py/tokensmith_engine.py: the sentence is persisted in the existingchunks.keywordscolumn (already indexed by thechunks_ftstriggers, never selected into source rows) and joins the embedding input.Measured
Generated BuzzDB textbook, nomic-embed-text, K=4 hybrid, gemma4:e4b writing the sentences from the passage text alone. 56 casual student phrasings over 14 concepts, scored strictly (one retrieved chunk must carry at least two of the concept's answer keywords), plus 28 textbook-worded controls. Every index rebuilt from empty.
Answers with gemma4:e4b reading the same windows: 41 of 56 without terms, 47 with them on those 13 passages. On a broader set (150 passages sampled across the book, one casual question each, written by a different model) giving every prose passage a sentence raised passage recall from 98 to 112 of 150.
Costs: about 3 seconds per passage once, cached; index time unchanged (62 to 75 s for this book either way); search latency unchanged; answer window unchanged because the sentence stays out of the passage text.
Limits, including the negative results, are in
docs/plain-language-search-terms.md: the gain is retrieval rather than reading, a question only benefits when it shares words with the sentence, two sentences per passage reached further on the broad set but was the only configuration that lost human questions, and a second textbook gained far less because its baseline was already 89%.Note on an existing check
tests/ui/library-preparation.jsxasserted that Basic preparation must never submit a preparation model. The terms step needs one in either mode, so that invariant is now conditional and the opposite case is asserted too.Validation
npm run typecheckandnpm run buildpass on 0.1.15.chunks_ftsonly in the index built with the option on, and never appear in a shown source.🤖 Generated with Claude Code