Skip to content

Indexer: redesign storage, indexing and index state #47

Open
0 of 1 issue completed
Open

Description

@johnf

This was generated by AI during triage.

This is the input for a wayfinder session that will chart the indexer redesign as a map (a new wayfinder:map issue). It brings together what's on main as of ec91b82 (after #63 and #64), what's already settled, and what's still open. The two comments below are folded in here.

Proposed destination

A decided design for ldacapi's indexer, clear enough to cut into implementation issues: where indexing runs and what triggers it, where index state lives, how entities shared between crates are stored and deleted, which parts move into arocapi, and how src/ is laid out afterwards. Planning only: the map ends when nothing is left to decide, not when it's built.

The charting session should confirm or redraw this destination first.

Notes for the map

  • Domain: GLOSSARY.md and docs/adr/. ADRs that constrain the design:
    • 0003 (every entity is visible)
    • 0004 (entity metadata is a virtual crate, which entityAsCrate builds)
  • arocapi owns the Entity and File Prisma models (prisma/arocapi, symlinked from the package), so ldacapi can't add columns or indexes to them. Any schema change there is an arocapi change. Our own models live in prisma/models.
  • Upstream repositories (crate-works/arocapi, ocfl-js): check with the maintainer before filing issues or PRs there.
  • Production scale: the largest collection has about 30,000 entities. Crate licence metadata is inconsistent; only Sydney Speaks uses ldac:authorizationWorkflow correctly.

Already settled (don't reopen)

What's there today

  • src/repository.ts: the Repository interface has one implementation. initRepository('ocfl') dynamically imports ./ocfl.ts and uses the module itself as the repository.
  • src/ocfl.ts: module-level singletons (INDEXER, repository, backend, stateCache, src/ocfl.ts:18) run both indexers over crates, find each crate's top collection, and keep each crate's index state (queued/indexing/deleting/indexed) in memory.
  • src/indexer/: an Indexer base class, with DatabaseIndexer (Prisma Entity/File rows) and SearchIndexer (OpenSearch documents) as subclasses.
  • Triggers: the admin pages start indexing and deleting in the API process without waiting for them to finish (src/admin/actions.ts:36, :57). pnpm index (scripts/index.ts) deletes and rebuilds everything.
  • Oni counts are worked out on each request: one recursive memberOf query per collection or object, across crates (feat: send Oni each entity's accessControl and counts #49).

Known problems

  • Index state is in memory, so it's lost on restart and wrong with more than one replica. Nothing persists the queue.
  • Indexing runs inside the API process.
  • Shared entities (Deleting a crate's database index removes entities other crates share #35, now a sub-issue): prisma.entity.create (src/indexer/database.ts:71) fails with a duplicate key for every crate after the first that contains an entity. That logs an ERROR with a full stack trace each time. Deleting by meta.crateId (database.ts:84) removes the entity for every crate that shares it.
  • Unindexed lookups: lookups by meta.crateId have no database index (it's arocapi's model), so they scan the entity table twice per crate on each admin page load.
  • Partial reindex: if a forced reindex removes a crate's search documents and then its database delete fails, the crate keeps its rows but has no documents. The admin page's Search state does show this.
  • Search refresh: there's one explicit OpenSearch refresh per crate (src/indexer/search.ts:221, :266), because arocapi's OpenSearchBackend has no per-call refresh option.
  • Index settings: these are a putIndexTemplate workaround in SearchIndexer.init (search.ts:206).
  • Missing parent crate: no test covers it.
  • The "entities" count: "Indexed crate … entities: N" counts entities attempted, not rows written.
  • Metadata-only crates (e.g. COOEE), which list files they don't contain, log Cannot read file at ERROR once per missing file in each indexer (database.ts:59, search.ts:186).
  • Leftovers:
    • _indexEntity is unused (src/indexer/indexer.ts:147).
    • Indexer.index reads crate.root.license before checking that crate.root exists (indexer.ts:112).
    • objects(prefix, refresh) ignores both arguments.
    • indexType (LDaC profile URIs) and openLicenses are LDaCA constants, not configuration, yet they live in src/configuration.ts:8, :15.
    • src/utils.ts holds one indexer helper.
  • Test data: the bundled test crate references sub_col_2.txt, which isn't in test-data/bundled.

Open questions

Rough dependency order. Each is a likely starting ticket or fog.

  1. Where indexing runs and what triggers it. Options: the API process, a separate worker, or a CLI only. Triggers: admin, deposit, schedule. Most of the other questions depend on this one.

  2. Where index state lives (Postgres, in prisma/models?), how the queue survives a restart and works across replicas, and what the admin page reads. Depends on 1.

  3. Shared entities (Deleting a crate's database index removes entities other crates share #35): how entities shared between crates are written and deleted. Options: upsert plus a crate↔entity join table in our own models, reference counting, or roCrateIds as in the spec's deposit extension. This also decides how lookups by crate avoid the table scan.

  4. The arocapi boundary: which generic parts move upstream. Candidates:

    • deriveUniqueEntityIds, pickSingleMemberOf and entityAsCrate
    • building file storage meta
    • shared-entity upserts
    • a meta.crateId index
    • OpenSearch index settings and per-call refresh as OpenSearchBackend options

    LDaC-specific parts stay here: the profile filter (recordType), search fields and facets, and ldac: property handling. This is likely research on arocapi first, then grilling.

  5. Aggregates for Oni: should the indexer precompute counts, plus the language, communicationMode and mediaType lists gathered from each subtree, instead of computing them per request? And how are ancestors in other crates refreshed when a crate changes? Ancestors stored on each search document and aggregated in OpenSearch could serve all four.

  6. The accessControl rule: what decides it, given how inconsistent the licence metadata is.

  7. Packaging: should the indexer be its own package (and so make the repo a pnpm workspace), e.g. if it runs as a worker or is shared with other arocapi deployments? Depends on 1 and 4.

  8. Layout: what src/ looks like afterwards (e.g. api/, oidc/, admin/, storage/, indexer/). This comes last.

The known-problem leftovers are small fixes, not decisions. The map can list them as out of scope or fold them into the implementation issues it hands off.

Related

Original request

Title: Indexer: redesign storage, indexing and index state

Rough scope for a wayfinder run. The indexer is the part of ldacapi that hasn't been reworked yet, and most of the structural problems left in src/ sit around it.

What's there today

  • src/repository.ts declares a Repository interface with one implementation. initRepository('ocfl') dynamically imports ./ocfl.ts and uses the module itself as the repository.
  • src/ocfl.ts does three jobs with module-level singletons (INDEXER, repository, backend, stateCache):
    • opening and creating the OCFL storage
    • running the structural and search indexers over crates
    • keeping each crate's index state (queued/indexing/deleting/indexed) in memory
  • src/indexer/ has an Indexer base class with StructuralIndexer (Prisma Entity/File rows) and SearchIndexer (OpenSearch documents) subclasses.
  • The admin pages start indexing in the API process, fire-and-forget, from a form post.

Problems

  • Index state is in memory: it's lost on restart, wrong with more than one replica, and nothing persists the queue.
  • Indexing runs inside the API process.
  • Shared entities: entity.create plus delete-by-rootCollection breaks when an entity appears in more than one crate, and deleting one crate's index removes entities other crates share (Deleting a crate's database index removes entities other crates share #35).
  • count() disagrees between indexers (structural counts id = crateId, search counts rootCollection), and getState depends on it.
  • Leftovers: _indexEntity is unused; Indexer.index reads crate.root before checking it exists; getState returns undefined for an unknown index type.
  • LDaCA constants live in configuration.ts though they aren't configuration: indexType (LDaC profile URIs) and openLicenses. src/utils.ts holds one indexer helper.
  • Crates that list files they don't contain (metadata-only, e.g. COOEE) log Cannot read file once per missing file.
  • The bundled test crate references sub_col_2.txt, which isn't in test-data.

Questions to settle

  • Does indexing run in the API process, a separate worker, or a CLI only? What triggers it (admin, deposit, schedule)?
  • Where does index state live (Postgres?), and what does the admin page read?
  • How are entities shared between crates written and deleted (upsert, reference counting, roCrateIds as in the spec's deposit extension)?
  • Which parts are generic and belong in arocapi, which defines the Entity/File tables and OcflStorageMeta but nothing that writes them? Candidates: deriveUniqueEntityIds, pickSingleMemberOf, entityAsCrate, building file storage meta, shared-entity upserts. LDaC-specific parts stay here: the profile filter (recordType), search fields and facets, ldac: property handling.
  • Does OpenSearch index settings (the putIndexTemplate workaround in SearchIndexer.init) become an option on arocapi's OpenSearchBackend?
  • Should the indexer become its own package (and so a pnpm workspace), e.g. if it runs as a worker or is shared with other arocapi deployments?
  • What does src/ look like afterwards (e.g. api/, oidc/, admin/, storage/, indexer/)?
  • Does the indexer need to produce anything for the Oni transformer (subtree counts, licence metadata, language/communicationMode/mediaType aggregates) rather than computing it per request?

Activity

  1. johnf commented on Nov 4, 2026

    @johnf
    CollaboratorAuthor

    From the Oni transformer work (feat/oni-entity-fields):

    • accessControl is now decided at index time and stored in Entity.meta.accessControl by the structural indexer. Today it's Public when the content licence is in openLicenses, otherwise AuthorizationByApplication. The transformer passes it on through arocapi's new row argument (Callbacks receive the stored row: transformers and access get row (incl. meta) crate-works/arocapi#137, in 5.0.0-beta.9). The redesign owns that rule: production crates' licence metadata is inconsistent (only Sydney Speaks uses ldac:authorizationWorkflow correctly), so any use of crate metadata needs care.
    • counts are worked out per request: one recursive memberOf query per collection or object, following memberOf across crates. In production the largest collection has about 30,000 entities. Decide whether the indexer should precompute them, and refresh ancestors in other crates when a crate changes.
    • Oni also expects language, communicationMode and mediaType on every entity: lists gathered from the subtree, which ldacapi doesn't send yet. They are the same kind of aggregate as counts, so one approach (e.g. ancestors stored on each search document and aggregated in OpenSearch, or precomputed in the indexer) could serve all four.
    • A distributed sub-crate's rootCollection is its own crate rather than the top collection (e.g. distributed_content_restricted's is itself, not distributed_root).
  2. johnf commented on Nov 4, 2026

    @johnf
    CollaboratorAuthor

    Update after #63, which aligned the indexer with arocapi 5. Items from the list above that it addressed:

    • count(): the two indexers no longer disagree. The database index counts a crate's rows by meta.crateId, and the search index counts real documents through arocapi's SearchBackend, in batches.
    • getState and unknown index types: index types are now one shared IndexType, so getState can't be asked about an unknown type.
    • Deletes: they go through arocapi's remove(ids), search before database, and stop at the first failure. Deleting the database index also deletes the search index, and pnpm index deletes everything before rebuilding.

    New limitations to fold into the redesign:

    • Unindexed lookups: lookups by meta.crateId have no database index, because arocapi owns the Entity model and our Prisma schema can't add one. They scan the entity table, twice per crate per admin page load.
    • Partial reindex: if a forced reindex removes a crate's search documents and then its database delete fails, the crate keeps its rows but has no documents. The admin Search state does show this.
    • Missing parent crate: the walk stops at the highest crate it could read and logs a warning, so the nested crate looks top-level. No test covers this.
    • Duplicate ids: entity ids shared between crates (Deleting a crate's database index removes entities other crates share #35) log an ERROR with a full stack, where a one-line warning would do.
    • The "entities" count: "Indexed crate … entities: N" counts entities attempted, not rows written.
    • Search refresh: there's one explicit refresh per crate, and arocapi's OpenSearchBackend has no per-call refresh option.
  3. added
    enhancementNew feature or request
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Nov 17, 2026
  4. johnf commented on Nov 17, 2026

    @johnf
    CollaboratorAuthor

    This was generated by AI during triage.

    Triaged as enhancement + ready-for-human. The body has been rewritten as the input for a wayfinder charting session; the original is kept under Original request, and both comments are folded in. #35 is now a sub-issue.

    Why not an agent: the open questions are architecture calls (where indexing runs, where index state lives, how shared entities are stored, what moves into arocapi). They need the maintainer in a grilling session, not an AFK agent.

    Next step: run /wayfinder with this issue to chart the map. That session confirms the proposed destination, creates the wayfinder:map issue and its first tickets, and moves #35 under the map.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions