Skip to content

feat(vault-graph): surgical vault search, mentions and section reads - #36

Merged
mm-weber merged 206 commits into
mainfrom
surgical-search
Nov 4, 2026
Merged

mm-weber merged 206 commits into
mainfrom
surgical-search

Conversation

@mm-weber

@mm-weber mm-weber commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

TL;DR

  • Agents stop reading whole vault notes. Three new read-only vault-graph tools return the blocks around each mention of a note (vault_mentions), a small note whole or a big one as an outline (vault_read_note), and single sections (vault_read_section). Four skills now read in that order.
  • On a real 469-note vault, worldbuilding around the biggest hub reads 19k tokens instead of 254k (13.1x less), and session prep 23k instead of 82k (3.6x). 90% of linked notes get every mention in one call, and warm calls on the biggest hubs take under 300 ms.
  • No new dependencies, and the six graph tools are unchanged. 299 tests pass and CI is green.

Adds Obsidian-style surgical reads to the vault-graph MCP server, so an agent reads the Markdown blocks it needs instead of whole notes. Spec: specs/v0.2/surgical-search.md (R1-R91). Backlog item BL-6 in specs/v0.2/v0.2-milestone-plan.md.

What ships

  • mcp-servers/vault-graph/note_parser.py (new): a pure line scanner for frontmatter, aliases, headings, code spans, links and blocks.
  • mcp-servers/vault-graph/vault_index.py (new): a stat-checked cache of the vault with Obsidian-style link resolution. It is the only module that reads files, and
    it checks file stats on every call, so results never go stale.
  • Three read-only tools in server.py (17 tools in total):
    • vault_mentions: linked mentions by default, plain-text ones with unlinked=True. Each mention comes with its Markdown block, labelled path:line and heading
      path. Excerpts fill budget_tokens (default 4,000); a map lists every mention not shown, and paths=[...] pulls more.
    • vault_read_note: the whole note when small, otherwise an outline.
    • vault_read_section: whole sections by heading.
  • Two optional settings: VAULT_SKIP_FOLDERS and VAULT_NAME_ENDINGS.
  • Skills: worldbuilding, chronik, session-creation and vault-scout follow a read ladder (mentions, then a note or its outline, then sections). Their copies in
    examples/ttrpg/skills/ stay byte-identical.
  • Docs: README, docs/mcp-tools.md, quickstart, design, demo walkthrough, CHANGELOG, the vault CLAUDE.md example and template, and BL-6 in the v0.2 milestone plan.

The six graph tools are unchanged. A follow-up moves them onto the live index and retires .graph.json and vault_rebuild_graph.

Two amendments after the first review

R59, the per-note cap. When not everything fits the budget, each note's excerpts now stop at the larger of a quarter of the budget and an even share among the
notes with mentions. Before, a paths call that narrowed the listing to one heavy note returned the same excerpts as the call before it and left three quarters of
the budget unused. Example on a synthetic vault, default budget 4,000, one note with 60 mentions (about 5.6k tokens of excerpts):

Call Before After
vault_mentions("Mira") (four notes listed) mentions 1-10 of the heavy note the same
vault_mentions("Mira", paths=["Sessions/Session 12"]) the same mentions 1-10, ~920 of 4,000 tokens mentions 1-43, ~4.0k tokens

R48, long excerpts are trimmed instead of cut (new R90 and R91). An excerpt over 1,500 characters used to keep its top 1,500 characters; the mentions after the
cut went to the map, where no paths call could ever show them. Now it keeps its mention lines and the lines that frame them (a table's header and separator, a
callout's first line, parent list items, a property's key), with … for skipped lines and a marker such as [trimmed to its mentions; the block runs L12-61]. If
that is still too long, it comes in parts ([part 2 of 3, ...]), and a single very long line shows a window around the mention. Every mention is now shown in some
excerpt, so the map only holds what the budget leaves out. On the measured vault this frees 24 linked mentions (all notes as targets) and 133 unlinked ones (top 20
hubs) that the cut used to strand.

Verification

  • Tests: 299 passed (203 before), and CI's test step passes on df2c90f. The three new test files hold 96 tests. ruff check . and ruff format --check . are
    clean.
  • A second review of the two amendments found no Critical issues. Its fix wave added five guard tests (each verified to fail under the mutant it targets), a column
    fix for decomposed (NFD) text in unlinked windows, and a fix for a negative budget with no mentions. A scoped re-review confirmed every finding addressed, with a
    differential fuzz showing no regression.
  • Measured on a real 469-note vault (counts only), after both amendments:
    • cold parse 0.37 s; warm vault_mentions 240-270 ms on the ten biggest hubs (gate: 300 ms);
    • 302 of 334 linked notes (90%) complete in one call (target: 80%; 86% before the trim);
    • the biggest hub's first call is 4,896 tokens and reaches all 7 top-level folders; no note takes more than 27% of a first call's excerpts;
    • every mention is shown or mapped on all ten hubs;
    • every mention fits in some excerpt and no excerpt body exceeds 1,500 characters, for every note as a target (linked) and the top 20 hubs (unlinked).
  • Context cost through the read ladder vs. whole-note reads (measured before the trim amendment): worldbuilding on the biggest hub drops from 254k to 19k tokens
    (13.1x); prep from the last two sessions drops from 82k to 23k (3.6x). Target: 3x.
  • After merge: V5, the maintainer's own check with -next copies of the skills.

Known limitations (accepted)

Recorded in the spec's section 8. In the measured vault there are no nested fences and no fences in callouts; 64 notes have no headings, and 67 have exactly one
linking note.

  • Code fences close at the next fence with the same character, whatever its length or info string, and fences inside callouts are not recognized. A note that shows
    a fenced block inside a four-backtick block reads the inner code as Markdown, and a lone inner fence line turns the rest of the note into code.
  • vault_read_section on a note without headings answers with an empty heading list, and an empty headings list returns nothing.
  • note arguments take a title, path, .md or alias, not ./, a leading or trailing /, backslashes, [[...]], |text, #heading or a copied path:line
    label. For such a string, vault_mentions answers as for an unwritten note ("0 notes link here"). paths takes a list, not a single string.
  • A count of one reads "1 notes".

CI note

CI installs unpinned ruff>=0.7 (0.17.1 now), which also formats Python code blocks inside Markdown. Two blocks in specs/v0.2/v0.2-milestone-plan.md, older than
this branch, failed ruff format --check ., so the lint step failed and the test step never ran; main fails the same check today. df2c90f formats them
(formatting only), and CI is green. Pinning ruff, so a new release can't turn CI red without a code change, fits BL-5 (Dependabot).

mm-weber and others added 30 commits October 18, 2026 12:48
EARS spec for Obsidian-style linked and unlinked mentions plus note and
section reads in the vault-graph server: a pure line parser, a live
per-file index, and three read-only tools (vault_mentions,
vault_read_note, vault_read_section). Covers the read ladder for the
skills, 83 requirements traced to verification, measured baseline,
success criteria and the PR rollout order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
An independent review found that a heading mention's excerpt (its whole
section) plus the 1,500-character cut could drop mentions from both the
excerpts and the map. Heading excerpts now end with their first block,
and every mention a cut removes moves to the map.

Also adds a terms table for the line scanner, escaped table links,
alias lookups and alias links, the lead in outlines, a gap marker, an
attachment extension list, and a per-note cap of a quarter of the
budget when not everything fits. The baseline is re-measured with the
section 6 rules. V1 shares a test between requirements that describe
one behavior (68 tests), and V6 adds a measurement before merge. The
v0.2 milestone is confirmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…D-1, R16)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…EEN-1, R16)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
… or one (RED-2, R18)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
… or one (GREEN-2, R18)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…aliases (RED-3, R17)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…aliases (GREEN-3, R17)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
… and #tag (RED-4, R19)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
… and #tag (GREEN-4, R19)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
… values (RED-5, R21)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
… values (GREEN-5, R21)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
… are not (RED-7, R22)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
… are not (GREEN-7, R22)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…r fewer # (RED-8, R30)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…r fewer # (GREEN-8, R30)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…es (RED-9, R23)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…es (GREEN-9, R23)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…unts as (RED-10, R24)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…unts as (GREEN-10, R24)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…cestor (RED-11, R25)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…cestor (GREEN-11, R25)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…ows (RED-12, R26)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…ows (GREEN-12, R26)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…un (RED-13, R27)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…un (GREEN-13, R27)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…of lines (RED-14, R28)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…of lines (GREEN-14, R28)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
mm-weber and others added 29 commits October 20, 2026 03:32
…cepted limitations (R46, R48, R90, R91)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…tems (RED-90, R48)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…tems (GREEN-90, R48)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…-91, R48)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…ED-93, R90)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…REEN-93, R90)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…-94, R91)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…EN-94, R91)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…N-95, R91)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…RED-96, R59)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…GREEN-96, R59)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…D-97, R91)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…EEN-97, R91)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…ire the column locator

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…review Minors 1 and 8)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
… (RED-102, R90)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
…as if complete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
Recent ruff releases also format Python code blocks inside Markdown, and
CI installs unpinned ruff>=0.7 (0.17.1 today), so `ruff format --check .`
failed on two blocks in this plan and the test step never ran. The blocks
predate this branch, and main fails the same check. Formatting only: two
blank lines before a def and one wrapped signature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
Found while trying the branch in a real setup: an MCP venv built for
the old server has mcp but no PyYAML, so the new note parser fails at
import and the server takes the six graph tools down with it.
requirements.txt already lists pyyaml; the note tells upgraders to
install it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J14rqN83ifDuCsEiHU4tz9
@mm-weber
mm-weber merged commit d839e00 into main Nov 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:mcp-servers MCP servers enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant