Repository navigation
Description
This was generated by AI during triage.
This is the input for a wayfinder session that will chart the indexer redesign as a map (a new wayfinder:map issue). It brings together what's on main as of ec91b82 (after #63 and #64), what's already settled, and what's still open. The two comments below are folded in here.
Proposed destination
A decided design for ldacapi's indexer, clear enough to cut into implementation issues: where indexing runs and what triggers it, where index state lives, how entities shared between crates are stored and deleted, which parts move into arocapi, and how src/ is laid out afterwards. Planning only: the map ends when nothing is left to decide, not when it's built.
The charting session should confirm or redraw this destination first.
Notes for the map
- Domain:
GLOSSARY.mdanddocs/adr/. ADRs that constrain the design:- 0003 (every entity is visible)
- 0004 (entity metadata is a virtual crate, which
entityAsCratebuilds)
- arocapi owns the
EntityandFilePrisma models (prisma/arocapi, symlinked from the package), so ldacapi can't add columns or indexes to them. Any schema change there is an arocapi change. Our own models live inprisma/models. - Upstream repositories (crate-works/arocapi, ocfl-js): check with the maintainer before filing issues or PRs there.
- Production scale: the largest collection has about 30,000 entities. Crate licence metadata is inconsistent; only Sydney Speaks uses
ldac:authorizationWorkflowcorrectly.
Already settled (don't reopen)
- Index types: the two index types are the database index and the search index (Rename the structural index to the database index, and pnpm seed to pnpm import #55), from one shared
IndexTypelist. SogetStatecan't be asked about an unknown type (Align with arocapi 5 and rewrite the README in its terms #63). - Counting: both indexers count a crate by
meta.crateId. The database index counts rows, and the search index counts documents through arocapi'sSearchBackend, in batches (Find a crate's rows by meta.crateId, and remove search documents through arocapi's backend #59, Align with arocapi 5 and rewrite the README in its terms #63). - Deleting: deletes go through arocapi's
remove(ids), search index before database index, and stop at the first failure. Deleting the database index also deletes the search index, andpnpm indexdeletes everything before rebuilding (Align with arocapi 5 and rewrite the README in its terms #63). rootCollection: it's the top-level collection, found across distributed crates by followingmemberOfup through parent crates (rootCollection follows arocapi: the top-level collection, across distributed crates #60). A crate whose parent can't be read is treated as top-level, with a warning. A cycle ofmemberOflinks is logged, naming the collections involved (Align with arocapi 5 and rewrite the README in its terms #63).- Documents without rows: there's no search document for an entity without a database row (Index no search document for an entity without a database row #61).
accessControl: it's decided at index time and stored inEntity.meta.accessControl(feat: send Oni each entity's accessControl and counts #49, via Callbacks receive the stored row: transformers and access getrow(incl. meta) crate-works/arocapi#137). Today it'sPublicwhen the content licence is inopenLicenses, otherwiseAuthorizationByApplication. The redesign owns this rule.- OCFL library: OCFL access is
@ocfl/ocfl-fs1.0 beta. Opening and creating the storage root lives insrc/ocfl-repository.ts(OCFL: drop local workarounds once ocfl-js ships the fixes #53, build!: move to @ocfl/ocfl-fs 1.0 beta and arocapi 5.0.0-beta.10 #64).
What's there today
src/repository.ts: theRepositoryinterface has one implementation.initRepository('ocfl')dynamically imports./ocfl.tsand uses the module itself as the repository.src/ocfl.ts: module-level singletons (INDEXER,repository,backend,stateCache,src/ocfl.ts:18) run both indexers over crates, find each crate's top collection, and keep each crate's index state (queued/indexing/deleting/indexed) in memory.src/indexer/: anIndexerbase class, withDatabaseIndexer(PrismaEntity/Filerows) andSearchIndexer(OpenSearch documents) as subclasses.- Triggers: the admin pages start indexing and deleting in the API process without waiting for them to finish (
src/admin/actions.ts:36,:57).pnpm index(scripts/index.ts) deletes and rebuilds everything. - Oni counts are worked out on each request: one recursive
memberOfquery per collection or object, across crates (feat: send Oni each entity's accessControl and counts #49).
Known problems
- Index state is in memory, so it's lost on restart and wrong with more than one replica. Nothing persists the queue.
- Indexing runs inside the API process.
- Shared entities (Deleting a crate's database index removes entities other crates share #35, now a sub-issue):
prisma.entity.create(src/indexer/database.ts:71) fails with a duplicate key for every crate after the first that contains an entity. That logs an ERROR with a full stack trace each time. Deleting bymeta.crateId(database.ts:84) removes the entity for every crate that shares it. - Unindexed lookups: lookups by
meta.crateIdhave no database index (it's arocapi's model), so they scan the entity table twice per crate on each admin page load. - Partial reindex: if a forced reindex removes a crate's search documents and then its database delete fails, the crate keeps its rows but has no documents. The admin page's Search state does show this.
- Search refresh: there's one explicit OpenSearch refresh per crate (
src/indexer/search.ts:221,:266), because arocapi'sOpenSearchBackendhas no per-call refresh option. - Index settings: these are a
putIndexTemplateworkaround inSearchIndexer.init(search.ts:206). - Missing parent crate: no test covers it.
- The "entities" count: "Indexed crate … entities: N" counts entities attempted, not rows written.
- Metadata-only crates (e.g. COOEE), which list files they don't contain, log
Cannot read fileat ERROR once per missing file in each indexer (database.ts:59,search.ts:186). - Leftovers:
_indexEntityis unused (src/indexer/indexer.ts:147).Indexer.indexreadscrate.root.licensebefore checking thatcrate.rootexists (indexer.ts:112).objects(prefix, refresh)ignores both arguments.indexType(LDaC profile URIs) andopenLicensesare LDaCA constants, not configuration, yet they live insrc/configuration.ts:8,:15.src/utils.tsholds one indexer helper.
- Test data: the
bundledtest crate referencessub_col_2.txt, which isn't intest-data/bundled.
Open questions
Rough dependency order. Each is a likely starting ticket or fog.
-
Where indexing runs and what triggers it. Options: the API process, a separate worker, or a CLI only. Triggers: admin, deposit, schedule. Most of the other questions depend on this one.
-
Where index state lives (Postgres, in
prisma/models?), how the queue survives a restart and works across replicas, and what the admin page reads. Depends on 1. -
Shared entities (Deleting a crate's database index removes entities other crates share #35): how entities shared between crates are written and deleted. Options: upsert plus a crate↔entity join table in our own models, reference counting, or
roCrateIdsas in the spec's deposit extension. This also decides how lookups by crate avoid the table scan. -
The arocapi boundary: which generic parts move upstream. Candidates:
deriveUniqueEntityIds,pickSingleMemberOfandentityAsCrate- building file storage meta
- shared-entity upserts
- a
meta.crateIdindex - OpenSearch index settings and per-call refresh as
OpenSearchBackendoptions
LDaC-specific parts stay here: the profile filter (
recordType), search fields and facets, andldac:property handling. This is likely research on arocapi first, then grilling. -
Aggregates for Oni: should the indexer precompute
counts, plus thelanguage,communicationModeandmediaTypelists gathered from each subtree, instead of computing them per request? And how are ancestors in other crates refreshed when a crate changes? Ancestors stored on each search document and aggregated in OpenSearch could serve all four. -
The
accessControlrule: what decides it, given how inconsistent the licence metadata is. -
Packaging: should the indexer be its own package (and so make the repo a pnpm workspace), e.g. if it runs as a worker or is shared with other arocapi deployments? Depends on 1 and 4.
-
Layout: what
src/looks like afterwards (e.g.api/,oidc/,admin/,storage/,indexer/). This comes last.
The known-problem leftovers are small fixes, not decisions. The map can list them as out of scope or fold them into the implementation issues it hands off.
Related
- Sub-issue: Deleting a crate's database index removes entities other crates share #35 (shared entities). When the map is created, move it under the map or link it from the relevant ticket.
- Context: feat: send Oni each entity's accessControl and counts #49 (Oni entity fields), Align with arocapi 5 and rewrite the README in its terms #52 / Align with arocapi 5 and rewrite the README in its terms #63 (alignment with arocapi 5), OCFL: drop local workarounds once ocfl-js ships the fixes #53 / build!: move to @ocfl/ocfl-fs 1.0 beta and arocapi 5.0.0-beta.10 #64 (
@ocfl/ocfl-fs1.0 beta), Rename the structural index to the database index, and pnpm seed to pnpm import #55, Find a crate's rows by meta.crateId, and remove search documents through arocapi's backend #59, rootCollection follows arocapi: the top-level collection, across distributed crates #60, Index no search document for an entity without a database row #61, Callbacks receive the stored row: transformers and access getrow(incl. meta) crate-works/arocapi#137.
Original request
Title: Indexer: redesign storage, indexing and index state
Rough scope for a wayfinder run. The indexer is the part of ldacapi that hasn't been reworked yet, and most of the structural problems left in src/ sit around it.
What's there today
src/repository.tsdeclares aRepositoryinterface with one implementation.initRepository('ocfl')dynamically imports./ocfl.tsand uses the module itself as the repository.src/ocfl.tsdoes three jobs with module-level singletons (INDEXER,repository,backend,stateCache):- opening and creating the OCFL storage
- running the structural and search indexers over crates
- keeping each crate's index state (
queued/indexing/deleting/indexed) in memory
src/indexer/has anIndexerbase class withStructuralIndexer(PrismaEntity/Filerows) andSearchIndexer(OpenSearch documents) subclasses.- The admin pages start indexing in the API process, fire-and-forget, from a form post.
Problems
- Index state is in memory: it's lost on restart, wrong with more than one replica, and nothing persists the queue.
- Indexing runs inside the API process.
- Shared entities:
entity.createplus delete-by-rootCollectionbreaks when an entity appears in more than one crate, and deleting one crate's index removes entities other crates share (Deleting a crate's database index removes entities other crates share #35). count()disagrees between indexers (structural countsid = crateId, search countsrootCollection), andgetStatedepends on it.- Leftovers:
_indexEntityis unused;Indexer.indexreadscrate.rootbefore checking it exists;getStatereturnsundefinedfor an unknown index type. - LDaCA constants live in
configuration.tsthough they aren't configuration:indexType(LDaC profile URIs) andopenLicenses.src/utils.tsholds one indexer helper. - Crates that list files they don't contain (metadata-only, e.g. COOEE) log
Cannot read fileonce per missing file. - The
bundledtest crate referencessub_col_2.txt, which isn't intest-data.
Questions to settle
- Does indexing run in the API process, a separate worker, or a CLI only? What triggers it (admin, deposit, schedule)?
- Where does index state live (Postgres?), and what does the admin page read?
- How are entities shared between crates written and deleted (upsert, reference counting,
roCrateIdsas in the spec's deposit extension)? - Which parts are generic and belong in arocapi, which defines the
Entity/Filetables andOcflStorageMetabut nothing that writes them? Candidates:deriveUniqueEntityIds,pickSingleMemberOf,entityAsCrate, building file storage meta, shared-entity upserts. LDaC-specific parts stay here: the profile filter (recordType), search fields and facets,ldac:property handling. - Does OpenSearch index settings (the
putIndexTemplateworkaround inSearchIndexer.init) become an option on arocapi'sOpenSearchBackend? - Should the indexer become its own package (and so a pnpm workspace), e.g. if it runs as a worker or is shared with other arocapi deployments?
- What does
src/look like afterwards (e.g.api/,oidc/,admin/,storage/,indexer/)? - Does the indexer need to produce anything for the Oni transformer (subtree counts, licence metadata,
language/communicationMode/mediaTypeaggregates) rather than computing it per request?
Activity
johnf commentedon Nov 4, 2026
From the Oni transformer work (
feat/oni-entity-fields):accessControlis now decided at index time and stored inEntity.meta.accessControlby the structural indexer. Today it'sPublicwhen the content licence is inopenLicenses, otherwiseAuthorizationByApplication. The transformer passes it on through arocapi's newrowargument (Callbacks receive the stored row: transformers and access getrow(incl. meta) crate-works/arocapi#137, in 5.0.0-beta.9). The redesign owns that rule: production crates' licence metadata is inconsistent (only Sydney Speaks usesldac:authorizationWorkflowcorrectly), so any use of crate metadata needs care.countsare worked out per request: one recursivememberOfquery per collection or object, followingmemberOfacross crates. In production the largest collection has about 30,000 entities. Decide whether the indexer should precompute them, and refresh ancestors in other crates when a crate changes.language,communicationModeandmediaTypeon every entity: lists gathered from the subtree, which ldacapi doesn't send yet. They are the same kind of aggregate ascounts, so one approach (e.g. ancestors stored on each search document and aggregated in OpenSearch, or precomputed in the indexer) could serve all four.rootCollectionis its own crate rather than the top collection (e.g.distributed_content_restricted's is itself, notdistributed_root).johnf commentedon Nov 4, 2026
Update after #63, which aligned the indexer with arocapi 5. Items from the list above that it addressed:
count(): the two indexers no longer disagree. The database index counts a crate's rows bymeta.crateId, and the search index counts real documents through arocapi'sSearchBackend, in batches.getStateand unknown index types: index types are now one sharedIndexType, sogetStatecan't be asked about an unknown type.remove(ids), search before database, and stop at the first failure. Deleting the database index also deletes the search index, andpnpm indexdeletes everything before rebuilding.New limitations to fold into the redesign:
meta.crateIdhave no database index, because arocapi owns the Entity model and our Prisma schema can't add one. They scan the entity table, twice per crate per admin page load.OpenSearchBackendhas no per-call refresh option.johnf commentedon Nov 17, 2026
Triaged as
enhancement+ready-for-human. The body has been rewritten as the input for a wayfinder charting session; the original is kept under Original request, and both comments are folded in. #35 is now a sub-issue.Why not an agent: the open questions are architecture calls (where indexing runs, where index state lives, how shared entities are stored, what moves into arocapi). They need the maintainer in a grilling session, not an AFK agent.
Next step: run
/wayfinderwith this issue to chart the map. That session confirms the proposed destination, creates thewayfinder:mapissue and its first tickets, and moves #35 under the map.