Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
655a1b9
fix: treat a REMS network failure as holding no licences (#57)
johnf Nov 4, 2026
d327e69
feat: rely on arocapi's sandbox headers instead of Caddy's CSP (#58)
johnf Nov 4, 2026
7b1b1fa
refactor: rename the structural index to the database index (#55)
johnf Nov 4, 2026
de1010e
feat!: facet search on entityType, the entity's class URI (#56)
johnf Nov 4, 2026
1e01032
fix: find a crate's rows by meta.crateId, and remove its search docum…
johnf Nov 4, 2026
44d7184
feat!: rootCollection is the top-level collection, across distributed…
johnf Nov 4, 2026
b647465
fix: index no search document for an entity without a database row (#61)
johnf Nov 4, 2026
e6d0ffe
refactor: rename pnpm seed to pnpm crates:import (#55)
johnf Nov 4, 2026
095acdb
feat: facet search on recordType, a readable label for the entity's c…
johnf Nov 4, 2026
3b0609e
docs: rewrite the README, glossary and ADRs in arocapi's terms (#62)
johnf Nov 4, 2026
db74df4
refactor: label record types from the Oni class constants (#56)
johnf Nov 4, 2026
19d387b
refactor: search reads each crate's rows itself, and counts its docum…
johnf Nov 4, 2026
d2c39d2
test: share the OCFL testbed, log capture and row helpers
johnf Nov 4, 2026
75f4ba5
fix: pnpm index deletes every index before rebuilding them
johnf Nov 4, 2026
7abfd0a
refactor: one list of index types, one way to read a crate's raw root…
johnf Nov 4, 2026
7625dcd
fix: count and remove a crate's search documents in batches of 10,000
johnf Nov 4, 2026
b80c361
fix: refresh the search index once after each crate's search delete a…
johnf Nov 4, 2026
69ec0f2
fix: label https://schema.org classes as their http://schema.org ones
johnf Nov 4, 2026
41d8991
fix: log each crate's indexed search documents, and its skip once whe…
johnf Nov 4, 2026
a08be87
fix: warn when collections are members of each other, naming them
johnf Nov 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,7 @@ Single-context: `GLOSSARY.md` and `docs/adr/` at the repo root. See `docs/agents
## Database migrations

A migration that creates or changes arocapi's models (`prisma/arocapi`, symlinked from the arocapi package) is named after the arocapi version: after an arocapi upgrade that changes the models, run `pnpm db:migrate --name arocapi-<version>`. Name migrations for our own models (`prisma/models`) after what they change.

## Upgrading arocapi

After any arocapi upgrade, recheck the README's links to arocapi's docs (arocapi.crate-works.org is unversioned): each page and anchor should still exist and still say what the README relies on.
24 changes: 24 additions & 0 deletions GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,30 @@ Serves RO-Crates from OCFL storage through the arocapi API, and controls access

## Language

arocapi's terms, such as **Record**, **Metadata**, **Content** and **Row**, mean what [arocapi's glossary](https://github.com/crate-works/arocapi/blob/main/GLOSSARY.md) says.

### Crates and indexes

**Crate**:
One RO-Crate, stored as one OCFL object in the repository. The unit of import, indexing and admin actions.
_Avoid_: OCFL object (the storage), collection (what a crate may describe)

**Import**:
Bringing RO-Crates into the OCFL repository: `pnpm crates:import` today, the deposit API in future.
_Avoid_: Seed

**Index**:
Building the database index and the search index from the repository, for one crate or all of them. Both can be deleted and rebuilt at any time.
_Avoid_: Import

**Database index**:
The Entity and File rows that arocapi reads, written by the indexer (arocapi's Rows).
_Avoid_: Structural index

**Search index**:
The search documents in arocapi's search backend, one per entity that has a row.
_Avoid_: OpenSearch index (one backend's storage)

### Authentication

**Upstream provider**:
Expand Down
171 changes: 120 additions & 51 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,26 +7,59 @@ RO-Crate API. It serves RO-Crates over the

## How it fits together

The core API (entities, files, search, RO-Crate metadata) comes from
[arocapi](https://github.com/crate-works/arocapi), which this
app mounts as a Fastify plugin. ldacapi adds what LDaCA needs on top:

- **OCFL storage**: crates live in an [OCFL](https://ocfl.io/) repository on
disk (`src/ocfl.ts`).
- **Indexing**: crates are read from OCFL into the database (the structural
index, used by arocapi's entity and file routes) and, as one search
document per entity, into arocapi's OpenSearch search backend
(`src/indexer/`). The search fields are declared in `src/indexer/search.ts`.
The API's routes (entities, files, search, RO-Crate metadata) come from
[arocapi](https://github.com/crate-works/arocapi), which ldacapi mounts as a
Fastify plugin under `/api` (`src/app.ts`). Its
[docs](https://arocapi.crate-works.org/) explain each part; ldacapi
configures them as follows:

- **Storage backend, Content half**: arocapi's
[`OcflStorage`](https://arocapi.crate-works.org/configuration/storage/#ocflstorage)
serves crate files from the OCFL repository.
- **Storage backend, Metadata half**: arocapi's
[`EntityMetaMetadata`](https://arocapi.crate-works.org/configuration/storage/#entitymetametadata)
serves each entity's virtual crate, built at index time and held in its
database row ([ADR 0004](docs/adr/0004-entity-metadata-is-a-virtual-crate.md)).
- **Search backend**: arocapi's
[`OpenSearchBackend`](https://arocapi.crate-works.org/configuration/search/#opensearchbackend),
with the [fields](https://arocapi.crate-works.org/configuration/search/#fields)
declared in `src/indexer/search.ts`, where `entityType` filters on an
entity's class URI and the "Record Type" facet, `recordType`, shows a
readable label for it.
- **Visibility**: every entity is visible to everyone, through arocapi's
all-visible [visibility filters](https://arocapi.crate-works.org/configuration/visibility/)
([ADR 0003](docs/adr/0003-every-entity-is-visible.md)).
- **Access**: arocapi's
[`LicenceAccess`](https://arocapi.crate-works.org/configuration/access/#access),
with the licences a user holds looked up in REMS
([licence checks against an external authority](https://arocapi.crate-works.org/strategies/access/#licence-checks-against-an-external-authority));
see [Licences](#licences) (`src/auth.ts`).

ldacapi adds what LDaCA needs on top (terms as in [GLOSSARY.md](GLOSSARY.md)):

- **OCFL repository**: each crate is one [OCFL](https://ocfl.io/) object on
disk, imported with `pnpm crates:import` (`src/ocfl.ts`).
- **Indexing**: `pnpm index` and the admin pages build the database index (the
Entity and File rows arocapi reads) and the search index (one search
document per entity) from the repository (`src/indexer/`). Both can be
rebuilt at any time.
- **Collections**: an entity's `rootCollection` is the top-level collection
it sits under, and is empty on a top-level collection
([membership](https://arocapi.crate-works.org/concepts/entities/#membership)).
Indexing finds it by following `memberOf` up through the parent crates in
the repository, so a crate in a distributed collection gets the
distributed collection's root whatever order the crates are indexed in. If
a parent collection moves, reindex the crates under it: they keep the old
`rootCollection` until then.
- **Sign-in**: the API is its own OIDC provider, signing users in at an
upstream provider such as CILogon (`src/oidc/`,
[ADR 0001](docs/adr/0001-own-oidc-provider.md)).
- **Admin pages**: server-rendered pages at `/admin/` on their own origin,
where admins, who sign in through our own OIDC provider, see and rebuild or
delete indexes (`src/admin/`).
- **Sign-in**: the API is its own OIDC provider, signing users in at an
upstream provider such as CILogon (`src/oidc/`).
- **Access control**: every entity is visible, and what a request may read is
decided by licence, checked against REMS (`src/auth.ts`).
- **Files and metadata**: arocapi serves crate files from the OCFL repository
through its OCFL storage backend, and each entity's RO-Crate metadata from
the structural index (`src/app.ts`).
delete indexes (`src/admin/`,
[ADR 0002](docs/adr/0002-server-rendered-admin-pages.md)).

The ADRs linked here are ldacapi's own, in `docs/adr/`, not arocapi's.

## Getting started

Expand All @@ -35,7 +68,7 @@ You need Node.js 26 (26.13 or later), [pnpm](https://pnpm.io/) and Docker.
```sh
pnpm install
docker compose up -d --wait # Postgres, OpenSearch, Oni and a fake upstream provider
pnpm bootstrap # migrate the database, seed OCFL from test-data, index
pnpm bootstrap # migrate the database, import test-data into OCFL, index
pnpm dev
```

Expand All @@ -54,25 +87,25 @@ override a default, set it in `.env`.

## Scripts

| Script | Does |
| -------------------- | -------------------------------------------------------------------------------- |
| `pnpm dev` | Run the API, restarting on changes, and rebuild the admin pages' stylesheet |
| `pnpm build` | Build the admin pages' stylesheet (run before deploying) |
| `pnpm test` | Run the tests (needs `docker compose up`) |
| `pnpm lint` | Biome, Knip and TypeScript |
| `pnpm bootstrap` | `db:deploy`, `seed` and `index` in one go |
| `pnpm ocfl:init` | Create an empty OCFL repository at `OCFL_ROOT`, unless one exists already |
| `pnpm seed [dir]` | Import each crate folder in `dir` (default `test-data`) into the OCFL repository |
| `pnpm index` | Rebuild both indexes from the OCFL repository |
| `pnpm generate-keys` | Print new `OIDC_SIGNING_KEYS` and `OIDC_COOKIE_KEYS` for `.env` |
| `pnpm db:migrate` | Create and apply a migration after a schema change |
| `pnpm db:deploy` | Apply pending migrations |
| `pnpm db:reset` | Drop and recreate the dev database |
| `pnpm db:console` | Open `psql` on the dev database |
| Script | Does |
| -------------------------- | -------------------------------------------------------------------------------- |
| `pnpm dev` | Run the API, restarting on changes, and rebuild the admin pages' stylesheet |
| `pnpm build` | Build the admin pages' stylesheet (run before deploying) |
| `pnpm test` | Run the tests (needs `docker compose up`) |
| `pnpm lint` | Biome, Knip and TypeScript |
| `pnpm bootstrap` | `db:deploy`, `crates:import` and `index` in one go |
| `pnpm ocfl:init` | Create an empty OCFL repository at `OCFL_ROOT`, unless one exists already |
| `pnpm crates:import [dir]` | Import each crate folder in `dir` (default `test-data`) into the OCFL repository |
| `pnpm index` | Rebuild both indexes from the OCFL repository |
| `pnpm generate-keys` | Print new `OIDC_SIGNING_KEYS` and `OIDC_COOKIE_KEYS` for `.env` |
| `pnpm db:migrate` | Create and apply a migration after a schema change |
| `pnpm db:deploy` | Apply pending migrations |
| `pnpm db:reset` | Drop and recreate the dev database |
| `pnpm db:console` | Open `psql` on the dev database |

## Test data

Each folder in `test-data/` is one RO-Crate. `pnpm seed` imports them into the
Each folder in `test-data/` is one RO-Crate. `pnpm crates:import` imports them into the
OCFL repository at `storage/ocfl` (gitignored), and re-running it only adds a
new OCFL version for crates that changed. Run `pnpm index` afterwards.

Expand All @@ -83,7 +116,7 @@ dataset) unless `DEFAULT_METADATA_LICENSE` or `DEFAULT_LICENSE` is set.
## Configuration

Configuration comes only from environment variables, validated at startup
(`src/configuration.ts`). `pnpm dev`, `pnpm seed`, `pnpm index` and Prisma
(`src/configuration.ts`). `pnpm dev`, `pnpm crates:import`, `pnpm index` and Prisma
also read `.env` if it exists, without overriding variables that are already
set.

Expand Down Expand Up @@ -114,10 +147,28 @@ have no defaults and must be set.

### Licences

Content under a licence that isn't open needs an entitlement to it in REMS,
looked up at `REMS_ENDPOINT` with `REMS_USER` and `REMS_KEY`. A user without
one is offered an enrolment link: `ENROLLMENT_URL` with every `{licenseId}`
replaced by the URL-encoded licence id, e.g.
Every entity has a metadata licence, which guards its RO-Crate metadata
(`/entity/:id/metadata`), and a content licence, which guards a file's bytes
(`/file/:id`)
([two licences per entity](https://arocapi.crate-works.org/concepts/licences/#two-licences-per-entity)).
The indexer takes the metadata licence from the `license` of
`ro-crate-metadata.json`, and the content licence from the entity's own
`license` or else the root dataset's.

A licence that isn't one of the open licences in `src/configuration.ts` needs
an entitlement to it in REMS, looked up at `REMS_ENDPOINT` with `REMS_USER`
and `REMS_KEY`, once per request. If REMS can't be reached or returns an
error, the API logs it and treats the user as holding no licences, so
restricted entities are refused until it recovers.

A refused request gets 401 if it is anonymous, so the client can prompt a
sign-in, and 403 if it is signed in
([Denied is 401 or 403](https://arocapi.crate-works.org/concepts/visibility-vs-access/#denied-is-401-or-403-never-404)).
The enrolment link isn't on the error: every entity the API returns carries
an `access` object, and for each licence the user doesn't hold it has a
`metadataAuthorizationUrl` or `contentAuthorizationUrl`. That link is
`ENROLLMENT_URL` with every `{licenseId}` replaced by the URL-encoded licence
id, e.g.

```dotenv
ENROLLMENT_URL=https://rems.example.org/apply-for?resource={licenseId}
Expand Down Expand Up @@ -199,9 +250,11 @@ username to `ADMIN_USERS` in `.env`. Anyone else sees "Not an admin".

The admin pages are at `<ADMIN_ORIGIN>/admin/`. `ADMIN_ORIGIN` is an origin,
with no path, such as `https://admin.example.org`, and must differ from
`OIDC_ISSUER`: crate files are served on the API's origin with their own
content type, so HTML or SVG in a crate could run script there, and a separate
origin keeps that away from admin. It defaults to `http://admin.localhost:8080`
`OIDC_ISSUER`. Crate files are served on the API's origin with their own
content type. arocapi sandboxes them, except audio and video, but that rests
on each response's headers, so a separate origin keeps any script a crate file
does run away from admin (ldacapi's
[ADR 0002](docs/adr/0002-server-rendered-admin-pages.md)). It defaults to `http://admin.localhost:8080`
outside production, which browsers resolve to this machine; in production it
must be https. The admin pages are on only when sign-in is and `ADMIN_ORIGIN`
is set; otherwise startup logs a warning and `/admin/` is not found. On the
Expand Down Expand Up @@ -287,7 +340,7 @@ docker run --rm --env-file ldacapi.env -v /srv/ldacapi:/storage ldacapi pnpm db:
```

`pnpm ocfl:init` (which creates an empty OCFL repository; the API won't start
without one), `pnpm seed [dir]` (mount `dir` into the container) and
without one), `pnpm crates:import [dir]` (mount `dir` into the container) and
`pnpm index` run the same way, offline. Give them and `pnpm db:deploy` the API's environment file:
Prisma and the scripts load the whole [configuration](#configuration).
`pnpm generate-keys` reads no configuration, so it needs no environment file:
Expand All @@ -302,7 +355,7 @@ The image has no default `OCFL_ROOT`. Mount your storage and set `OCFL_ROOT`
(and `OCFL_SCRATCH`, if you need it elsewhere) to paths inside the container
(above, `OCFL_ROOT=/storage/ocfl`). The API runs as the
image's `node` user, uid 1000, so the OCFL repository must be readable by uid
1000 and the scratch directory writable by it. `pnpm seed` writes to the
1000 and the scratch directory writable by it. `pnpm crates:import` writes to the
repository, so give it write access too when you run it.

## Production with Docker Compose
Expand All @@ -319,11 +372,15 @@ from Let's Encrypt on its own. It serves two hostnames:
| `DOMAIN` | everything else | Oni |
| `ADMIN_DOMAIN` | `/admin/*` | The API's admin pages |

The admin pages need their own hostname, so that script in a crate file can
never run on their origin (ADR 0002). Caddy also serves everything under
`/api/` with `Content-Security-Policy: sandbox`, which gives any HTML or SVG in
a crate file an opaque origin, so its scripts can't reach Oni's sign-in on
`DOMAIN`.
The admin pages need their own hostname (see [Admin pages](#admin-pages)).
Caddy sets no Content-Security-Policy on the API's responses: arocapi sends
crate files with `nosniff` and, except audio and video, a `sandbox`
Content-Security-Policy, which gives any HTML or SVG in a crate an opaque
origin, so its scripts can't reach Oni's sign-in on `DOMAIN`
([protective headers](https://arocapi.crate-works.org/configuration/storage/#protective-headers)).
Audio and video are left out so they play when opened directly. On both
hostnames Caddy adds HSTS, `nosniff` and a `strict-origin-when-cross-origin`
referrer policy, and Oni's pages also get `X-Frame-Options: SAMEORIGIN`.

No other container publishes a port, on the host or on localhost, so Postgres,
OpenSearch, the API and Oni are reachable only from within the stack.
Expand Down Expand Up @@ -389,7 +446,7 @@ Import crates and index them with the API's image, mounting the directory of
crates read-only:

```sh
docker compose run --rm -v /path/to/crates:/import:ro api pnpm seed /import
docker compose run --rm -v /path/to/crates:/import:ro api pnpm crates:import /import
docker compose run --rm api pnpm index
```

Expand All @@ -404,6 +461,18 @@ docker compose run --rm api pnpm db:deploy
docker compose up -d
```

`pnpm index` deletes both indexes and rebuilds them, so run it when an upgrade
changes what they hold. If an upgrade changes the search fields, the API and
`pnpm index` stop with an error naming the changed field, as they check the
search index's mapping before anything else. Delete the search index (named by
`ENTITY_INDEX`, `entities` by default) first, then rebuild and start again:

```sh
docker compose exec opensearch curl -sX DELETE localhost:9200/entities
docker compose run --rm api pnpm index
docker compose up -d
```

To run a branch's image (see [Docker image](#docker-image)) on a dev
deployment, change the API's `image` tag in `compose.yaml` from `main` to the
branch's tag and follow the upgrade steps above. Migrations the branch adds
Expand Down
9 changes: 1 addition & 8 deletions deploy/Caddyfile
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,7 @@
file_server
}

# Crate files are served inline with their own content type. The sandbox gives
# any HTML or SVG among them an opaque origin, so its scripts can't reach Oni's sign-in.
handle /api/* {
header Content-Security-Policy "sandbox allow-scripts allow-popups allow-downloads"
reverse_proxy api:8080
}

@provider path /api /oauth/* /.well-known/openid-configuration
@provider path /api /api/* /oauth/* /.well-known/openid-configuration
handle @provider {
reverse_proxy api:8080
}
Expand Down
2 changes: 1 addition & 1 deletion deploy/oni/configuration.json
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@
},
{
"display": "Record Type",
"name": "type"
"name": "recordType"
},
{
"display": "Language",
Expand Down
2 changes: 1 addition & 1 deletion docker/oni/configuration.json
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@
},
{
"display": "Record Type",
"name": "type"
"name": "recordType"
},
{
"display": "Language",
Expand Down
Loading
Loading