Skip to content

Align with arocapi 5 and rewrite the README in its terms #52

Closed
8 of 8 issues completed
Closed

Description

@johnf

Problem Statement

ldacapi mounts arocapi 5 (5.0.0-beta.9, matching arocapi's latest code) and mostly plugs into it the way arocapi documents. A comparison against arocapi's docs, README and example app found no hard pivot (no patching, route overrides or forked arocapi code, and no v4-era APIs), but it did find places where ldacapi has drifted from arocapi's meaning or contract, mostly without a recorded decision:

  • rootCollection holds the id of the crate an entity came from, even on the crate's own root. arocapi defines it as the top-level collection an entity sits under, and empty on a top-level collection. API clients see a top-level collection as its own collection. Oni's "Collection" facet splits a distributed collection into one entry per crate.
  • Admin deletes and counts on the search index call the OpenSearch client directly, filtering on rootCollection. That depends on how arocapi's OpenSearchBackend lays out its documents, instead of going through the backend's remove(ids). The index is also refreshed by hand instead of through the backend's refresh option.
  • If writing an entity's database row fails, its search document is still indexed. arocapi then drops the hit but still counts it in total and the facets, so a search says "20 results" and shows 19.
  • The search facet type holds the raw @type list, where arocapi's examples use entityType for the entity row's single class URI.
  • Caddy adds its own, weaker sandbox Content-Security-Policy to everything under /api/*. arocapi 5 now sandboxes delivered crate files itself, and deliberately leaves audio and video out because Chrome won't play media opened directly under a CSP. Caddy's header puts it back on media. ADR 0002 and the README still say crate files are served with no sandbox.
  • A REMS error response is treated as holding no licences, but a network failure reaching REMS becomes a 500.
  • "Structural index" is an ldacapi-only term that arocapi doesn't use and our glossary doesn't define. pnpm seed is used in production to bring real crates in, despite its name.

The README also repeats arocapi's docs in places, gets some of arocapi's behaviour wrong or leaves it out (the licence check also covers metadata, where the enrolment link appears, 401 vs 403), misquotes the Caddy header, and links to none of arocapi's docs.

Solution

Bring ldacapi back in line with arocapi wherever it has drifted, record the places where we differ on purpose, and rewrite the README so it tells operators and developers how ldacapi uses arocapi in arocapi's own terms, linking to arocapi's docs for how arocapi works. Everything ships in one PR, one commit per change. The rootCollection and entityType changes are breaking (feat!). The next release is a major breaking change anyway, and everyone reindexes, so no migration or reindex tooling is needed.

User Stories

  1. As an API client, I want rootCollection on an entity to name the top-level collection it sits under, so that I can show where an entity belongs.
  2. As an API client, I want a top-level collection's rootCollection to be empty, so that a collection never claims to be inside itself.
  3. As an API client, I want the entities of a nested crate in a distributed collection to have the distributed collection's root as their rootCollection, so that the whole collection is one unit.
  4. As an Oni user, I want the "Collection" search facet to show a distributed collection once, so that I can filter to all of it in one click.
  5. As an Oni user, I want the "Record Type" facet to show readable labels for entity types, so that I can filter by collection, object, file and the other types.
  6. As an API client, I want the search field entityType to hold the same class URI as the entity's entityType, so that filtering search results and listing entities agree.
  7. As an Oni user, I want a search's result count to match the results I can page through, so that I trust the numbers.
  8. As an admin, I want deleting a crate's indexes to remove all of its database rows and search documents, including the people, organisations and other entities it describes, so that nothing is left behind.
  9. As an admin, I want deleting or reindexing one crate to leave every other crate alone, including crates in the same distributed collection, so that I can work on one crate safely.
  10. As an admin, I want a crate's counts on the crates page to reflect its database rows and search documents, so that I can see whether indexing worked.
  11. As an admin, I want the crates page to label the two indexes "Database" and "Search", so that I know which store each state refers to.
  12. As an admin, I want indexing order not to matter, so that a fresh full index gives nested crates the right rootCollection even if their parent crate is indexed after them.
  13. As an operator, I want to run pnpm crates:import <dir> to bring crates into the OCFL repository, so that the command says what it does.
  14. As a developer, I want pnpm bootstrap to keep importing test-data and indexing it, so that setting up locally is still one command.
  15. As a signed-in user, I want to be offered the enrolment link when REMS can't be reached, rather than an error page, so that I can still apply for access.
  16. As an operator, I want a REMS outage to be logged, so that I can see why users are being refused.
  17. As a user, I want to open a crate's audio and video files directly in my browser and have them play, so that I can listen to and watch content.
  18. As an operator, I want crate files to be sandboxed by arocapi's own headers, so that a crate's HTML or SVG can't run script on the API's origin without Caddy having to duplicate that protection.
  19. As an operator, I want the README to let me deploy and run ldacapi without having to read arocapi's docs, so that deploying stays self-contained.
  20. As a developer, I want the README's "How it fits together" to describe each part (storage backend content and metadata halves, search backend, visibility, access) in one sentence with a deep link to arocapi's docs, so that I learn the mechanism from arocapi's docs and not from a copy.
  21. As an operator, I want the README's Licences section to say the licence check covers metadata as well as content, that the enrolment link comes back on the entity rather than on the error, and when a request gets 401 or 403, so that I can explain to users what they will see.
  22. As an operator, I want the README to describe the sandboxing and Caddy headers as they really are, so that I don't misconfigure the proxy.
  23. As a reader, I want references to ADRs to say they are ldacapi's, so that I don't confuse them with arocapi's ADRs of the same number.
  24. As a developer, I want GLOSSARY.md to define Crate, Import, Index, Database index and Search index, and to link to arocapi's glossary for Record, Metadata, Content and Row, so that we use one vocabulary with arocapi.
  25. As a developer, I want "Import" defined by its outcome, bringing RO-Crates into the OCFL repository, so that the term still holds when the deposit API replaces pnpm import.
  26. As a developer, I want an ADR explaining that each entity's metadata is a virtual crate built at index time (the entity as root, plus everything it links to), and that it is only as fresh as the last index run, so that nobody "fixes" it into serving the whole crate.
  27. As a developer, I want ADR 0002's context to reflect that arocapi now sandboxes delivered bytes but leaves media out, so that the separate admin origin is justified correctly.
  28. As a developer, I want a comment saying why CORS is registered only on /api, so that nobody moves it to the root as arocapi's docs suggest and leaks CORS headers onto /oauth and /admin.
  29. As a developer upgrading arocapi, I want the upgrade note in CLAUDE.md to remind me to recheck the README's links to arocapi's docs, so that they don't go stale while arocapi's docs site is unversioned.

Implementation Decisions

  • rootCollection follows arocapi. It holds the top-level collection an entity sits under, and is empty on a top-level collection's own row. For a crate whose root has a memberOf pointing outside the crate, the indexer follows memberOf upward by reading each parent crate's metadata from the OCFL repository, cached for the run, until it reaches a root with no parent. It doesn't read parents from the database, so the result is the same whatever order crates are indexed in. If a parent collection moves, its children keep the old value until they are reindexed. The README says so, and the code doesn't handle it.
  • The crate id moves to meta.crateId on each entity row (added to the shape ldacapi keeps in Entity.meta). The database index's deletes and counts find a crate's rows by meta.crateId. The per-crate count currently counts by entity id, and becomes a count of the crate's rows.
  • The search index goes through arocapi's SearchBackend contract. Deleting a crate passes that crate's entity ids, looked up from the database by meta.crateId before its rows are deleted, to remove(ids). Refreshing uses the backend's refresh option rather than a manual refresh call. Deleting every crate may still drop the whole OpenSearch index directly. The search index's count for a crate becomes a count of its database rows.
  • Search documents only for entities with a row. The search indexer reads the crate's rows from the database by meta.crateId (which also supplies each document's rootCollection) and skips any entity without one, logging each skip, or once per crate when it has no rows. Indexing runs database then search; deleting runs search then database and stops at the first failure, and deleting the database index also deletes the search index. pnpm index deletes every index, then indexes every crate.
  • The search field type is replaced by entityType and recordType. entityType holds the entity row's single class URI (the same value as Entity.entityType) and is a filter only. recordType is the "Record Type" facet, holding a readable label for that class, because Oni can't label facet values. Oni's development and deploy configurations facet on recordType. This changes the search index mapping, so existing indexes must be deleted and rebuilt (expected for this release).
  • "Structural index" becomes "database index": DatabaseIndexer, the module renamed to match, state key database, and admin label "Database". Index state is held in memory, so no data migration. The verb stays "index", which is arocapi's word for building the database and search index, including when it indexes deposits in future.
  • pnpm seed becomes pnpm crates:import (pnpm import is a built-in pnpm command), with the same behaviour and arguments. pnpm bootstrap, the README and the deploy docs use the new name.
  • Entity metadata stays a virtual crate built at index time (the entity as root plus what it links to, with Dataset, conformsTo and license added). Recorded in a new ldacapi ADR.
  • REMS network failures are logged and treated as holding no licences, the same as a non-OK REMS response. This is the outage option arocapi documents for its licence check.
  • Caddy drops its Content-Security-Policy on /api/* and relies on arocapi's delivery headers. Before removing it, check that arocapi's CSP and nosniff reach the client through Caddy. Caddy's global security headers stay.
  • CORS stays registered in the /api scope, with a one-line comment on why. Mounting arocapi under our own prefixed plugin, rather than arocapi's prefix option, is unchanged.
  • Unchanged by choice: the per-request batching and raw SQL behind Oni's counts, and the OpenSearch index template that supplies index settings. Both are commented already. Asking arocapi for hooks to replace them is being explored separately.
  • README: written for operators and developers. Operators can deploy without arocapi's docs, and developers are linked to them. "How it fits together" is rewritten in arocapi's terms, one sentence per part, with deep links to arocapi.crate-works.org (linked as it is, unversioned). Fix the Licences section (metadata is covered too, the enrolment link comes back on the entity's access, 401 for anonymous and 403 for signed-in requests), the sandbox and Caddy wording, the admin-origin rationale, and the "ADR 0002" reference (make clear it is ldacapi's). Use the new names (pnpm import, database index, entityType) throughout, and note that a moved parent collection needs its children reindexed.
  • GLOSSARY.md: add Crate (one RO-Crate stored as one OCFL object, the unit of import, indexing and admin actions; avoid "OCFL object", which means the storage, and "collection", which means the content), Import (bringing RO-Crates into the OCFL repository: pnpm import today, the deposit API in future), Index (building the database index and search index from the repository; rebuildable), Database index (the Entity and File rows arocapi reads, which arocapi calls Rows) and Search index (the search documents in arocapi's search backend). Link to arocapi's glossary for Record, Metadata, Content and Row.
  • ADR 0002: rewrite the context to say the decision as it now stands. arocapi sandboxes delivered bytes but not audio and video, and that still justifies the separate admin origin.
  • CLAUDE.md: the arocapi upgrade note also says to recheck the README's links to arocapi's docs.

Testing Decisions

  • Good tests here exercise behaviour from the outside, through the HTTP API and the admin pages, against the real Postgres and OpenSearch that pnpm test already provides per run. They don't assert on indexer internals or on OpenSearch document layout.
  • The whole app, through the existing buildTestApp harness: import the test-data crates, including the three distributed ones, into a per-run repository, index them, then use app.inject against arocapi's routes to check:
    • rootCollection is empty on a top-level collection
    • the entities of distributed_content_restricted and distributed_metadata_restricted have distributed_root as their rootCollection, whatever order the crates were indexed in
    • the recordType facet on /search returns labels, and filtering on entityType matches /entities?entityType=
    • a search's total matches its hits when an entity's row failed to be written
    • /entity/:id/metadata returns the virtual crate
    • Prior art: src/app.test.ts, src/oni.test.ts, test/bootstrap.test.ts.
  • The admin pages, through the existing tests in src/admin/actions.test.ts and src/admin/crates.test.ts:
    • reindexing or deleting a crate removes all its rows and search documents, including people and organisations, and leaves other crates (including siblings in a distributed collection) untouched
    • the crates page shows "Database" and "Search" states and counts
  • The licence check, through the existing tests in src/auth.test.ts: when the REMS endpoint can't be reached, a restricted entity is Denied with its enrolment link, and no 500.
  • src/indexer/structural.test.ts follows the rename and keeps covering the virtual crate's contents.
  • Not automated:
    • The Caddy change gets a manual check in the PR: on a deploy, a crate's HTML file is sandboxed by arocapi's header, and an audio file opened directly plays in Chrome.
    • The Oni facet labels are checked by looking at Oni in Docker Compose.

Out of Scope

  • Changing arocapi. Drafting arocapi issues for a per-page transformer hook (to replace the counts batching) and a settings option on OpenSearchBackend (to replace the index template) is being handled in a separate session.
  • An arocapi hook for app-supplied row values when arocapi indexes deposits. Wait until arocapi starts on deposits.
  • The deposit API itself, and replacing pnpm crates:import with a deposit client.
  • Reindex or migration tooling for existing deployments. Nobody is live, and the next release is breaking.
  • Using arocapi's prefix option instead of our prefixed plugin.
  • Serving each crate's real ro-crate-metadata.json instead of the virtual crate.

Further Notes

  • arocapi says it will index deposited RO-Crates itself, writing the database rows and search documents. Our database indexer is then temporary, so its rename stays mechanical and nothing more is built onto it than this issue needs. When arocapi takes over, it will need a way for ldacapi to supply rootCollection across crates, the virtual crate and accessControl.
  • arocapi's docs site is unversioned, and it currently describes v5 beta. Links may need updating when arocapi 5 is released.

Original request

Original title: Docs: compare the README with arocapi's

ldacapi's README describes how it mounts arocapi, its configuration, storage and deployment. arocapi's README and docs describe the library's options, storage backends and access model, and changed a lot in 5.0 (pluggable search and storage, row arguments for callbacks). Check ldacapi's README against arocapi's for anything that is duplicated, contradicts it, or is now out of date, and link to arocapi's docs instead of restating them where that's simpler.

From TODO.md (Medium):

  • Compare against README in arocapi

Activity

  1. added
    needs-triageMaintainer needs to evaluate this issue
    ready-for-agentFully specified, ready for an AFK agent
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Nov 4, 2026
  2. changed the title [-]Docs: compare the README with arocapi's[/-] [+]Align with arocapi 5 and rewrite the README in its terms[/+] on Nov 4, 2026
  3. johnf commented on Nov 4, 2026

    @johnf
    CollaboratorAuthor

    Implemented in #63 (20 signed commits, CI green). All eight tickets landed. #56 and #58 each have one manual check left. The PR closes the other six, and #52 stays open until those two checks are done.

    Changes from the spec, made during the run (the body is updated to match):

    • Import script: it's pnpm crates:import, because pnpm import is a built-in pnpm command.
    • Search fields: entityType is a filter-only class URI, and a new recordType facet holds readable labels, because Oni can't label facet values.
    • How search finds rows: the search indexer reads each crate's rows itself, rather than being handed the ids the database indexer wrote.
    • Order: indexing and deleting each follow a fixed order, deletes stop at the first failure, and deleting the database index also deletes the search index.
    • pnpm index: it deletes every index before rebuilding, so rows from before this change, which have no meta.crateId, don't block the rebuild.

    Review fixes beyond the spec:

    • a garbled REMS reply is treated as holding no licences
    • search counts and deletes go in batches of 10,000
    • the search index is refreshed once per crate
    • https://schema.org classes get the same labels as http://schema.org
    • a memberOf loop between crates is logged

    Known and not fixed: listed in the PR's Evidence section. The main one is that lookups by meta.crateId have no database index, because arocapi owns the Entity model.

  4. johnf commented on Nov 4, 2026

    @johnf
    CollaboratorAuthor

    All eight tickets are done: #63 merged, and the two manual checks passed (Oni's labels on #56, the deploy sandbox check on #58).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions