Repository navigation
Description
Problem Statement
ldacapi mounts arocapi 5 (5.0.0-beta.9, matching arocapi's latest code) and mostly plugs into it the way arocapi documents. A comparison against arocapi's docs, README and example app found no hard pivot (no patching, route overrides or forked arocapi code, and no v4-era APIs), but it did find places where ldacapi has drifted from arocapi's meaning or contract, mostly without a recorded decision:
rootCollectionholds the id of the crate an entity came from, even on the crate's own root. arocapi defines it as the top-level collection an entity sits under, and empty on a top-level collection. API clients see a top-level collection as its own collection. Oni's "Collection" facet splits a distributed collection into one entry per crate.- Admin deletes and counts on the search index call the OpenSearch client directly, filtering on
rootCollection. That depends on how arocapi'sOpenSearchBackendlays out its documents, instead of going through the backend'sremove(ids). The index is also refreshed by hand instead of through the backend'srefreshoption. - If writing an entity's database row fails, its search document is still indexed. arocapi then drops the hit but still counts it in
totaland the facets, so a search says "20 results" and shows 19. - The search facet
typeholds the raw@typelist, where arocapi's examples useentityTypefor the entity row's single class URI. - Caddy adds its own, weaker
sandboxContent-Security-Policy to everything under/api/*. arocapi 5 now sandboxes delivered crate files itself, and deliberately leaves audio and video out because Chrome won't play media opened directly under a CSP. Caddy's header puts it back on media. ADR 0002 and the README still say crate files are served with no sandbox. - A REMS error response is treated as holding no licences, but a network failure reaching REMS becomes a 500.
- "Structural index" is an ldacapi-only term that arocapi doesn't use and our glossary doesn't define.
pnpm seedis used in production to bring real crates in, despite its name.
The README also repeats arocapi's docs in places, gets some of arocapi's behaviour wrong or leaves it out (the licence check also covers metadata, where the enrolment link appears, 401 vs 403), misquotes the Caddy header, and links to none of arocapi's docs.
Solution
Bring ldacapi back in line with arocapi wherever it has drifted, record the places where we differ on purpose, and rewrite the README so it tells operators and developers how ldacapi uses arocapi in arocapi's own terms, linking to arocapi's docs for how arocapi works. Everything ships in one PR, one commit per change. The rootCollection and entityType changes are breaking (feat!). The next release is a major breaking change anyway, and everyone reindexes, so no migration or reindex tooling is needed.
User Stories
- As an API client, I want
rootCollectionon an entity to name the top-level collection it sits under, so that I can show where an entity belongs. - As an API client, I want a top-level collection's
rootCollectionto be empty, so that a collection never claims to be inside itself. - As an API client, I want the entities of a nested crate in a distributed collection to have the distributed collection's root as their
rootCollection, so that the whole collection is one unit. - As an Oni user, I want the "Collection" search facet to show a distributed collection once, so that I can filter to all of it in one click.
- As an Oni user, I want the "Record Type" facet to show readable labels for entity types, so that I can filter by collection, object, file and the other types.
- As an API client, I want the search field
entityTypeto hold the same class URI as the entity'sentityType, so that filtering search results and listing entities agree. - As an Oni user, I want a search's result count to match the results I can page through, so that I trust the numbers.
- As an admin, I want deleting a crate's indexes to remove all of its database rows and search documents, including the people, organisations and other entities it describes, so that nothing is left behind.
- As an admin, I want deleting or reindexing one crate to leave every other crate alone, including crates in the same distributed collection, so that I can work on one crate safely.
- As an admin, I want a crate's counts on the crates page to reflect its database rows and search documents, so that I can see whether indexing worked.
- As an admin, I want the crates page to label the two indexes "Database" and "Search", so that I know which store each state refers to.
- As an admin, I want indexing order not to matter, so that a fresh full index gives nested crates the right
rootCollectioneven if their parent crate is indexed after them. - As an operator, I want to run
pnpm crates:import <dir>to bring crates into the OCFL repository, so that the command says what it does. - As a developer, I want
pnpm bootstrapto keep importingtest-dataand indexing it, so that setting up locally is still one command. - As a signed-in user, I want to be offered the enrolment link when REMS can't be reached, rather than an error page, so that I can still apply for access.
- As an operator, I want a REMS outage to be logged, so that I can see why users are being refused.
- As a user, I want to open a crate's audio and video files directly in my browser and have them play, so that I can listen to and watch content.
- As an operator, I want crate files to be sandboxed by arocapi's own headers, so that a crate's HTML or SVG can't run script on the API's origin without Caddy having to duplicate that protection.
- As an operator, I want the README to let me deploy and run ldacapi without having to read arocapi's docs, so that deploying stays self-contained.
- As a developer, I want the README's "How it fits together" to describe each part (storage backend content and metadata halves, search backend, visibility, access) in one sentence with a deep link to arocapi's docs, so that I learn the mechanism from arocapi's docs and not from a copy.
- As an operator, I want the README's Licences section to say the licence check covers metadata as well as content, that the enrolment link comes back on the entity rather than on the error, and when a request gets 401 or 403, so that I can explain to users what they will see.
- As an operator, I want the README to describe the sandboxing and Caddy headers as they really are, so that I don't misconfigure the proxy.
- As a reader, I want references to ADRs to say they are ldacapi's, so that I don't confuse them with arocapi's ADRs of the same number.
- As a developer, I want
GLOSSARY.mdto define Crate, Import, Index, Database index and Search index, and to link to arocapi's glossary for Record, Metadata, Content and Row, so that we use one vocabulary with arocapi. - As a developer, I want "Import" defined by its outcome, bringing RO-Crates into the OCFL repository, so that the term still holds when the deposit API replaces
pnpm import. - As a developer, I want an ADR explaining that each entity's metadata is a virtual crate built at index time (the entity as root, plus everything it links to), and that it is only as fresh as the last index run, so that nobody "fixes" it into serving the whole crate.
- As a developer, I want ADR 0002's context to reflect that arocapi now sandboxes delivered bytes but leaves media out, so that the separate admin origin is justified correctly.
- As a developer, I want a comment saying why CORS is registered only on
/api, so that nobody moves it to the root as arocapi's docs suggest and leaks CORS headers onto/oauthand/admin. - As a developer upgrading arocapi, I want the upgrade note in
CLAUDE.mdto remind me to recheck the README's links to arocapi's docs, so that they don't go stale while arocapi's docs site is unversioned.
Implementation Decisions
rootCollectionfollows arocapi. It holds the top-level collection an entity sits under, and is empty on a top-level collection's own row. For a crate whose root has amemberOfpointing outside the crate, the indexer followsmemberOfupward by reading each parent crate's metadata from the OCFL repository, cached for the run, until it reaches a root with no parent. It doesn't read parents from the database, so the result is the same whatever order crates are indexed in. If a parent collection moves, its children keep the old value until they are reindexed. The README says so, and the code doesn't handle it.- The crate id moves to
meta.crateIdon each entity row (added to the shape ldacapi keeps inEntity.meta). The database index's deletes and counts find a crate's rows bymeta.crateId. The per-crate count currently counts by entity id, and becomes a count of the crate's rows. - The search index goes through arocapi's
SearchBackendcontract. Deleting a crate passes that crate's entity ids, looked up from the database bymeta.crateIdbefore its rows are deleted, toremove(ids). Refreshing uses the backend'srefreshoption rather than a manual refresh call. Deleting every crate may still drop the whole OpenSearch index directly. The search index's count for a crate becomes a count of its database rows. - Search documents only for entities with a row. The search indexer reads the crate's rows from the database by
meta.crateId(which also supplies each document'srootCollection) and skips any entity without one, logging each skip, or once per crate when it has no rows. Indexing runs database then search; deleting runs search then database and stops at the first failure, and deleting the database index also deletes the search index.pnpm indexdeletes every index, then indexes every crate. - The search field
typeis replaced byentityTypeandrecordType.entityTypeholds the entity row's single class URI (the same value asEntity.entityType) and is a filter only.recordTypeis the "Record Type" facet, holding a readable label for that class, because Oni can't label facet values. Oni's development and deploy configurations facet onrecordType. This changes the search index mapping, so existing indexes must be deleted and rebuilt (expected for this release). - "Structural index" becomes "database index":
DatabaseIndexer, the module renamed to match, state keydatabase, and admin label "Database". Index state is held in memory, so no data migration. The verb stays "index", which is arocapi's word for building the database and search index, including when it indexes deposits in future. pnpm seedbecomespnpm crates:import(pnpm importis a built-in pnpm command), with the same behaviour and arguments.pnpm bootstrap, the README and the deploy docs use the new name.- Entity metadata stays a virtual crate built at index time (the entity as root plus what it links to, with
Dataset,conformsToandlicenseadded). Recorded in a new ldacapi ADR. - REMS network failures are logged and treated as holding no licences, the same as a non-OK REMS response. This is the outage option arocapi documents for its licence check.
- Caddy drops its
Content-Security-Policyon/api/*and relies on arocapi's delivery headers. Before removing it, check that arocapi's CSP andnosniffreach the client through Caddy. Caddy's global security headers stay. - CORS stays registered in the
/apiscope, with a one-line comment on why. Mounting arocapi under our own prefixed plugin, rather than arocapi'sprefixoption, is unchanged. - Unchanged by choice: the per-request batching and raw SQL behind Oni's
counts, and the OpenSearch index template that supplies index settings. Both are commented already. Asking arocapi for hooks to replace them is being explored separately. - README: written for operators and developers. Operators can deploy without arocapi's docs, and developers are linked to them. "How it fits together" is rewritten in arocapi's terms, one sentence per part, with deep links to arocapi.crate-works.org (linked as it is, unversioned). Fix the Licences section (metadata is covered too, the enrolment link comes back on the entity's
access, 401 for anonymous and 403 for signed-in requests), the sandbox and Caddy wording, the admin-origin rationale, and the "ADR 0002" reference (make clear it is ldacapi's). Use the new names (pnpm import, database index,entityType) throughout, and note that a moved parent collection needs its children reindexed. GLOSSARY.md: add Crate (one RO-Crate stored as one OCFL object, the unit of import, indexing and admin actions; avoid "OCFL object", which means the storage, and "collection", which means the content), Import (bringing RO-Crates into the OCFL repository:pnpm importtoday, the deposit API in future), Index (building the database index and search index from the repository; rebuildable), Database index (the Entity and File rows arocapi reads, which arocapi calls Rows) and Search index (the search documents in arocapi's search backend). Link to arocapi's glossary for Record, Metadata, Content and Row.- ADR 0002: rewrite the context to say the decision as it now stands. arocapi sandboxes delivered bytes but not audio and video, and that still justifies the separate admin origin.
CLAUDE.md: the arocapi upgrade note also says to recheck the README's links to arocapi's docs.
Testing Decisions
- Good tests here exercise behaviour from the outside, through the HTTP API and the admin pages, against the real Postgres and OpenSearch that
pnpm testalready provides per run. They don't assert on indexer internals or on OpenSearch document layout. - The whole app, through the existing
buildTestAppharness: import thetest-datacrates, including the three distributed ones, into a per-run repository, index them, then useapp.injectagainst arocapi's routes to check:rootCollectionis empty on a top-level collection- the entities of
distributed_content_restrictedanddistributed_metadata_restrictedhavedistributed_rootas theirrootCollection, whatever order the crates were indexed in - the
recordTypefacet on/searchreturns labels, and filtering onentityTypematches/entities?entityType= - a search's
totalmatches its hits when an entity's row failed to be written /entity/:id/metadatareturns the virtual crate- Prior art:
src/app.test.ts,src/oni.test.ts,test/bootstrap.test.ts.
- The admin pages, through the existing tests in
src/admin/actions.test.tsandsrc/admin/crates.test.ts:- reindexing or deleting a crate removes all its rows and search documents, including people and organisations, and leaves other crates (including siblings in a distributed collection) untouched
- the crates page shows "Database" and "Search" states and counts
- The licence check, through the existing tests in
src/auth.test.ts: when the REMS endpoint can't be reached, a restricted entity is Denied with its enrolment link, and no 500. src/indexer/structural.test.tsfollows the rename and keeps covering the virtual crate's contents.- Not automated:
- The Caddy change gets a manual check in the PR: on a deploy, a crate's HTML file is sandboxed by arocapi's header, and an audio file opened directly plays in Chrome.
- The Oni facet labels are checked by looking at Oni in Docker Compose.
Out of Scope
- Changing arocapi. Drafting arocapi issues for a per-page transformer hook (to replace the
countsbatching) and asettingsoption onOpenSearchBackend(to replace the index template) is being handled in a separate session. - An arocapi hook for app-supplied row values when arocapi indexes deposits. Wait until arocapi starts on deposits.
- The deposit API itself, and replacing
pnpm crates:importwith a deposit client. - Reindex or migration tooling for existing deployments. Nobody is live, and the next release is breaking.
- Using arocapi's
prefixoption instead of our prefixed plugin. - Serving each crate's real
ro-crate-metadata.jsoninstead of the virtual crate.
Further Notes
- arocapi says it will index deposited RO-Crates itself, writing the database rows and search documents. Our database indexer is then temporary, so its rename stays mechanical and nothing more is built onto it than this issue needs. When arocapi takes over, it will need a way for ldacapi to supply
rootCollectionacross crates, the virtual crate andaccessControl. - arocapi's docs site is unversioned, and it currently describes v5 beta. Links may need updating when arocapi 5 is released.
Original request
Original title: Docs: compare the README with arocapi's
ldacapi's README describes how it mounts arocapi, its configuration, storage and deployment. arocapi's README and docs describe the library's options, storage backends and access model, and changed a lot in 5.0 (pluggable search and storage, row arguments for callbacks). Check ldacapi's README against arocapi's for anything that is duplicated, contradicts it, or is now out of date, and link to arocapi's docs instead of restating them where that's simpler.
From TODO.md (Medium):
- Compare against README in arocapi
Activity
[-]Docs: compare the README with arocapi's[/-][+]Align with arocapi 5 and rewrite the README in its terms[/+]johnf commentedon Nov 4, 2026
Implemented in #63 (20 signed commits, CI green). All eight tickets landed. #56 and #58 each have one manual check left. The PR closes the other six, and #52 stays open until those two checks are done.
Changes from the spec, made during the run (the body is updated to match):
pnpm crates:import, becausepnpm importis a built-in pnpm command.entityTypeis a filter-only class URI, and a newrecordTypefacet holds readable labels, because Oni can't label facet values.pnpm index: it deletes every index before rebuilding, so rows from before this change, which have nometa.crateId, don't block the rebuild.Review fixes beyond the spec:
https://schema.orgclasses get the same labels ashttp://schema.orgmemberOfloop between crates is loggedKnown and not fixed: listed in the PR's Evidence section. The main one is that lookups by
meta.crateIdhave no database index, because arocapi owns the Entity model.johnf commentedon Nov 4, 2026
All eight tickets are done: #63 merged, and the two manual checks passed (Oni's labels on #56, the deploy sandbox check on #58).