diff --git a/.env.example b/.env.example index 5fe9408..dafa1b5 100644 --- a/.env.example +++ b/.env.example @@ -2,6 +2,10 @@ PORT=3000 HOST=127.0.0.1 PUBLIC_ORIGIN=http://localhost:3000 ALLOWED_HOSTS=localhost,127.0.0.1 +# Production registry storage. When set, PostgreSQL is used and W3BS_DATA_DIR is ignored. +# Railway: reference the Postgres service, e.g. ${{Postgres.DATABASE_URL}}. +DATABASE_URL= +# Local development and tests fall back to SQLite in this directory. W3BS_DATA_DIR=.local/data # Set a random secret of at least 32 characters to enable publishing/revocation. # Keep this in your deployment secret manager, never in the repository. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6b6c762..96867b4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,6 +5,22 @@ permissions: jobs: validate: runs-on: ubuntu-latest + services: + postgres: + image: postgres:18-alpine + env: + POSTGRES_PASSWORD: w3bs + POSTGRES_DB: w3bs + ports: + - 5432:5432 + options: >- + --health-cmd "pg_isready -U postgres" + --health-interval 5s + --health-timeout 5s + --health-retries 10 + env: + # Registry tests run against SQLite and, with this set, PostgreSQL as well. + W3BS_TEST_DATABASE_URL: postgres://postgres:w3bs@127.0.0.1:5432/w3bs steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 diff --git a/README.md b/README.md index 4586bee..cc85fee 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ W3BS_API=http://localhost:3000 node native/client.mjs w3bs://prompt/w3bs/researc - Three implemented protocol drafts and five initial proposals, governance and contribution process. - Strict URI and manifest validation, RFC 8785 canonicalization, Ed25519 signatures and namespace-bound pinned public keys. -- Persistent SQLite registry with ten signed prompt examples, immutable publication and resource revocation. +- Persistent registry with ten signed prompt examples, immutable publication and resource revocation. PostgreSQL in production (`DATABASE_URL`), SQLite for local development and tests. - Shared search, resolve, inspect, verify, publish, run, revoke and conformance operations over CLI, HTTP and MCP, including stdio and Streamable HTTP. - Responsive web inspector, prompt search, manifest downloads, explicit template-rendering consent and a PWA offline state. - An experimental native URI handler with an independent parser and signature verifier. @@ -46,9 +46,9 @@ Conformance compares the actual API, CLI process, HTML inspection envelope, HTTP ## Configuration and operation -See `.env.example`, `docs/developers.md` and `docs/deployment.md`. Production requires a persistent `/data` volume, an explicit public origin and host allowlist. Administrative writes are disabled without a publisher token. Example private signing material stays in ignored `.local/keys` and is not required by the server. +See `.env.example`, `docs/developers.md` and `docs/deployment.md`. Production requires a PostgreSQL `DATABASE_URL`, an explicit public origin and host allowlist. Administrative writes are disabled without a publisher token. Example private signing material stays in ignored `.local/keys` and is not required by the server. -The Docker image runs one Node service as a non-root user. Use a single replica with SQLite. Source, governance proposals, issue templates and conformance fixtures are intended for public release. +The Docker image runs one Node service as a non-root user against the self-hosted PostgreSQL service in the same Railway project. Source, governance proposals, issue templates and conformance fixtures are intended for public release. ## Layout diff --git a/docs/deployment.md b/docs/deployment.md index d5e5e21..b23543a 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -10,18 +10,18 @@ The service is deployment-ready source. Production launch requires publishing th | `PORT` | Supplied by the host | | `PUBLIC_ORIGIN` | `https://w3bs.org` | | `ALLOWED_HOSTS` | `w3bs.org,www.w3bs.org,specs.w3bs.org,browse.w3bs.org,prompt.w3bs.org` plus the exact generated preview hostname if used | -| `W3BS_DATA_DIR` | `/data` on a persistent volume | -| `RAILWAY_RUN_UID` | `0` for Railway's root-owned volume; the bootstrap drops to uid/gid 1000 before starting HTTP | +| `DATABASE_URL` | PostgreSQL connection string. On Railway, `${{Postgres.DATABASE_URL}}` from the self-hosted Postgres service | +| `W3BS_DATA_DIR` | Only used without `DATABASE_URL`: SQLite directory for local development | | `W3BS_PUBLISH_TOKEN` | Random administrative secret, at least 32 characters; absent means read-only | | `W3BS_TRUST_FILE` | Optional path to a mounted operator-managed trust file; otherwise bundled example public keys | -Never deploy `.local/keys` or upload publisher private keys. The container needs public keys and pre-signed examples only. Store any administrative token in the provider's secret manager. Back up the SQLite database and trust configuration. Use one replica; horizontal replication requires replacing the local SQLite adapter. +Never deploy `.local/keys` or upload publisher private keys. The container needs public keys and pre-signed examples only. Store any administrative token in the provider's secret manager. Back up the PostgreSQL database and trust configuration. The schema is created on first start; the ten bundled examples are seeded once and never re-inserted. ## Railway -Create a service from the reviewed repository or local Docker context. The repository includes `Dockerfile` and `railway.json`. Attach a persistent volume at `/data`, provide the settings above and deploy. Health is `/healthz`. +The production project is `w3bs` in the Profullstack workspace: a self-hosted PostgreSQL service (Railway's `postgres-ssl` image, one replica, its own volume) and the `w3bs` web service built from this repository's `Dockerfile` on every push to `main`. `railway.json` sets the health check to `/healthz`, which reports `"storage": "postgres"` when the database is in use. -The health endpoint accepts Railway's `healthcheck.railway.app` hostname; that exception does not permit access to other routes. Railway volumes mount as root, so the container bootstrap initializes only the data directory and registry database files, then drops privileges before importing the server. It also works when launched directly as the image's default `node` user with an already writable volume. Provider references: https://docs.railway.com/deployments/healthchecks and https://docs.railway.com/volumes +The health endpoint accepts Railway's `healthcheck.railway.app` hostname; that exception does not permit access to other routes. No application volume is needed. If a database URL carries `sslmode=require` (Railway's public proxy uses a self-signed certificate), the connection uses TLS without certificate verification; the internal URL has no `sslmode` and runs in plain TCP inside the private network. Provider references: https://docs.railway.com/deployments/healthchecks and https://docs.railway.com/guides/postgresql Add `w3bs.org`, `specs.w3bs.org`, `browse.w3bs.org` and `prompt.w3bs.org` as custom service domains. Read each exact DNS target and verification record from the hosting provider; do not guess them. Update the registrar records, replacing the parking records only for those requested website names. Wait for domain validation and certificate issuance. diff --git a/fixtures/conformance-report.json b/fixtures/conformance-report.json index de8761b..befbe7a 100644 --- a/fixtures/conformance-report.json +++ b/fixtures/conformance-report.json @@ -1,8 +1,8 @@ { "status": "passed", - "testedAt": "2026-09-14T18:11:16.415Z", + "testedAt": "2026-09-16T17:31:35.889Z", "implementationVersion": "0.1.0", - "sourceFingerprint": "05135f148e24f47f458afe99be87690460c0f32888299fed6a16b6e6486efbbe", + "sourceFingerprint": "5ef70588c02e655abee500bd3de87b75f3085e914fd02d2705143818d96558c7", "nodeVersion": "v24.18.1", "environment": "isolated loopback registry, not production", "resource": "w3bs://prompt/w3bs/research@1", diff --git a/package-lock.json b/package-lock.json index 5172542..84acd3b 100644 --- a/package-lock.json +++ b/package-lock.json @@ -13,6 +13,7 @@ "canonicalize": "2.1.0", "express": "5.1.0", "marked": "16.4.2", + "pg": "^8.23.0", "zod": "4.3.6" }, "bin": { @@ -942,6 +943,95 @@ "url": "https://opencollective.com/express" } }, + "node_modules/pg": { + "version": "8.23.0", + "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz", + "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==", + "license": "MIT", + "dependencies": { + "pg-connection-string": "^2.14.0", + "pg-pool": "^3.14.0", + "pg-protocol": "^1.16.0", + "pg-types": "2.2.0", + "pgpass": "1.0.5" + }, + "engines": { + "node": ">= 16.0.0" + }, + "optionalDependencies": { + "pg-cloudflare": "^1.4.0" + }, + "peerDependencies": { + "pg-native": ">=3.0.1" + }, + "peerDependenciesMeta": { + "pg-native": { + "optional": true + } + } + }, + "node_modules/pg-cloudflare": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz", + "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==", + "license": "MIT", + "optional": true + }, + "node_modules/pg-connection-string": { + "version": "2.14.0", + "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz", + "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==", + "license": "MIT" + }, + "node_modules/pg-int8": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz", + "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==", + "license": "ISC", + "engines": { + "node": ">=4.0.0" + } + }, + "node_modules/pg-pool": { + "version": "3.14.0", + "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz", + "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==", + "license": "MIT", + "peerDependencies": { + "pg": ">=8.0" + } + }, + "node_modules/pg-protocol": { + "version": "1.16.0", + "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz", + "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==", + "license": "MIT" + }, + "node_modules/pg-types": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz", + "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==", + "license": "MIT", + "dependencies": { + "pg-int8": "1.0.1", + "postgres-array": "~2.0.0", + "postgres-bytea": "~1.0.0", + "postgres-date": "~1.0.4", + "postgres-interval": "^1.1.0" + }, + "engines": { + "node": ">=4" + } + }, + "node_modules/pgpass": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz", + "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==", + "license": "MIT", + "dependencies": { + "split2": "^4.1.0" + } + }, "node_modules/pkce-challenge": { "version": "5.0.1", "resolved": "https://registry.npmjs.org/pkce-challenge/-/pkce-challenge-5.0.1.tgz", @@ -980,6 +1070,45 @@ "node": ">=20" } }, + "node_modules/postgres-array": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz", + "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==", + "license": "MIT", + "engines": { + "node": ">=4" + } + }, + "node_modules/postgres-bytea": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz", + "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/postgres-date": { + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz", + "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/postgres-interval": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz", + "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==", + "license": "MIT", + "dependencies": { + "xtend": "^4.0.0" + }, + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/prettier": { "version": "3.9.6", "resolved": "https://registry.npmjs.org/prettier/-/prettier-3.9.6.tgz", @@ -1228,6 +1357,15 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/split2": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz", + "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==", + "license": "ISC", + "engines": { + "node": ">= 10.x" + } + }, "node_modules/statuses": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", @@ -1316,6 +1454,15 @@ "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", "license": "ISC" }, + "node_modules/xtend": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz", + "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==", + "license": "MIT", + "engines": { + "node": ">=0.4" + } + }, "node_modules/zod": { "version": "4.3.6", "resolved": "https://registry.npmjs.org/zod/-/zod-4.3.6.tgz", diff --git a/package.json b/package.json index 6220bd4..63176fa 100644 --- a/package.json +++ b/package.json @@ -28,6 +28,7 @@ "canonicalize": "2.1.0", "express": "5.1.0", "marked": "16.4.2", + "pg": "^8.23.0", "zod": "4.3.6" }, "devDependencies": { diff --git a/scripts/conformance.mjs b/scripts/conformance.mjs index 38f37dc..fb86236 100644 --- a/scripts/conformance.mjs +++ b/scripts/conformance.mjs @@ -25,7 +25,7 @@ function sourceFingerprint() { } export async function conformance() { const instance = await startServer({ - store: openStore({ path: ':memory:' }), + store: await openStore({ path: ':memory:' }), port: 0, host: '127.0.0.1', }); diff --git a/src/operations.mjs b/src/operations.mjs index 93accd9..35387b0 100644 --- a/src/operations.mjs +++ b/src/operations.mjs @@ -63,7 +63,7 @@ export function authorize(token, expected = process.env.W3BS_PUBLISH_TOKEN) { if (actual.length !== target.length || !timingSafeEqual(actual, target)) throw new W3bsError('UNAUTHORIZED', 'A valid registry publisher token is required.', 401); } -export function dispatch(store, operation, args, { token, publishToken } = {}) { +export async function dispatch(store, operation, args, { token, publishToken } = {}) { const definition = operations[operation]; if (!definition) throw new W3bsError('UNKNOWN_OPERATION', 'Unknown W3BS operation.', 404); const parsed = definition.schema.safeParse(args); @@ -74,10 +74,10 @@ export function dispatch(store, operation, args, { token, publishToken } = {}) { ); if (definition.write) authorize(token, publishToken); const input = parsed.data; - if (operation === 'search') return { resources: store.search(input.query, input.type) }; + if (operation === 'search') return { resources: await store.search(input.query, input.type) }; if (operation === 'publish') return store.publish(input.manifest); if (operation === 'revoke') return store.revoke(input.uri, input.reason); - const result = store.inspect(input.uri); + const result = await store.inspect(input.uri); if (operation === 'verify') return { canonicalUri: result.canonicalUri, verification: result.verification }; if (operation === 'run') { diff --git a/src/server.mjs b/src/server.mjs index f19d5fb..9b126e9 100644 --- a/src/server.mjs +++ b/src/server.mjs @@ -12,11 +12,12 @@ import { createMcp } from './mcp.mjs'; import { renderPage } from './site.mjs'; export function createApp({ - store = openStore(), + store, publishToken = process.env.W3BS_PUBLISH_TOKEN, publicOrigin = process.env.PUBLIC_ORIGIN || 'http://localhost:3000', allowedHosts = (process.env.ALLOWED_HOSTS || 'localhost,127.0.0.1').split(','), } = {}) { + if (!store) throw new Error('createApp requires an opened store; use startServer or openStore.'); const app = express(); const origin = new URL(publicOrigin).origin; const hosts = new Set(allowedHosts.map((host) => host.trim().toLowerCase())); @@ -55,8 +56,13 @@ export function createApp({ const token = (req) => /^Bearer (.+)$/.exec(req.headers.authorization || '')?.[1]; const invoke = (req, name, args) => dispatch(store, name, args, { token: token(req), publishToken }); - app.get('/healthz', (_req, res) => - res.json({ status: 'ok', version: '0.1.0', resources: store.search().length }), + app.get('/healthz', async (_req, res) => + res.json({ + status: 'ok', + version: '0.1.0', + storage: store.dialect, + resources: (await store.search()).length, + }), ); app.get('/.well-known/w3bs.json', (_req, res) => res.json({ @@ -108,13 +114,15 @@ export function createApp({ ); res.type('json').send(readFileSync(path, 'utf8')); }); - app.get('/api/manifest', (req, res) => res.json(store.inspect(req.query.uri).manifest)); + app.get('/api/manifest', async (req, res) => + res.json((await store.inspect(req.query.uri)).manifest), + ); for (const name of Object.keys(operations)) { - app.post(`/api/${name}`, (req, res) => res.json(invoke(req, name, req.body))); + app.post(`/api/${name}`, async (req, res) => res.json(await invoke(req, name, req.body))); if (['search', 'resolve', 'inspect', 'verify', 'conformance'].includes(name)) { - app.get(`/api/${name}`, (req, res) => + app.get(`/api/${name}`, async (req, res) => res.json( - invoke( + await invoke( req, name, name === 'search' @@ -177,11 +185,13 @@ export function createApp({ if (!existsSync(path)) throw new W3bsError('NOT_FOUND', 'Document not found.', 404); res.type('text/markdown').send(readFileSync(path, 'utf8')); }); - app.get(/.*/, (req, res, next) => { + app.get(/.*/, async (req, res, next) => { try { res .type('html') - .send(renderPage({ path: req.path, host: req.hostname, query: req.query, store, origin })); + .send( + await renderPage({ path: req.path, host: req.hostname, query: req.query, store, origin }), + ); } catch (error) { next(error); } @@ -208,16 +218,14 @@ export function createApp({ ? 'NOT_FOUND' : 'INTERNAL_ERROR'); if (status === 500) process.stderr.write(`W3BS ${req.method} ${req.path}: ${error.message}\n`); - res - .status(status) - .json({ - error: { code, message: status === 500 ? 'Internal server error.' : error.message }, - }); + res.status(status).json({ + error: { code, message: status === 500 ? 'Internal server error.' : error.message }, + }); }); return { app, store }; } export async function startServer(options = {}) { - const { app, store } = createApp(options); + const { app, store } = createApp({ ...options, store: options.store ?? (await openStore()) }); const server = createServer({ requestTimeout: 30000, headersTimeout: 10000 }, app); await new Promise((resolve, reject) => { server.once('error', reject); @@ -233,7 +241,7 @@ export async function startServer(options = {}) { close: async () => { server.closeAllConnections(); await new Promise((resolve) => server.close(resolve)); - store.close(); + await store.close(); }, }; } diff --git a/src/site.mjs b/src/site.mjs index 4ddb900..7650471 100644 --- a/src/site.mjs +++ b/src/site.mjs @@ -80,8 +80,8 @@ function header(active) { function footer() { return ``; } -function home(store) { - const count = store.search().length; +async function home(store) { + const count = (await store.search()).length; return `
AN OPEN STANDARDS INITIATIVE

The open Web.
For all of us.

Humans, agents & devices.
One interoperable foundation.

A Web where every participant can discover, communicate and collaborate. Open protocols. Portable resources. Authority you can inspect.

Explore the open Web Read the standards
Founding stage. Built in the open.
ONE IDENTITY. EVERY INTERFACE.+
Humans Agents Devicesw3bs://
CANONICAL RESOURCE ···
w3bs://prompt/w3bs/research@1

Same identity. Same manifest. Same meaning.

CLIAPIMCPWEBNATIVE
OPEN BY DEFAULTVENDOR NEUTRALVERIFIABLE BY DESIGNBUILT TO INTEROPERATE
A SHARED FOUNDATION

The Web is getting
new participants.

Pages became applications. Applications are becoming collaborators. The next chapter needs common ground that belongs to everyone.

01 / DISCOVER

A common address.

A w3bs:// URI identifies the resource. The interface, registry and transport can change without changing what it means.

Explore identifiers
02 / UNDERSTAND

Nothing hidden.

Inspect who published it, what it contains, which permissions it needs and whether its signature checks out.

Inside a manifest
03 / COLLABORATE

Authority, on purpose.

Content is data by default. A signature proves provenance; running instructions still requires explicit authorization.

Understand trust
@@ -95,17 +95,17 @@ function home(store) { .join('')}
OPEN IS SOMETHING WE BUILD TOGETHER

Help shape what comes next.

Implement a draft. Question an assumption. Bring a different perspective.
The next Web needs more than one voice.

Join the work
`; } -function promptCards(store, query) { - const resources = store.search(query.q || '', query.type || ''); +async function promptCards(store, query) { + const resources = await store.search(query.q || '', query.type || ''); const catalog = JSON.parse(readFileSync(resolve(ROOT, 'fixtures/catalog.json'), 'utf8')); const categories = new Map(catalog.map((item) => [item.id, item.category])); return `
THE OPEN PROMPT REGISTRY

Good instructions
should travel.

Portable, versioned, signed artifacts. Inspect their source, understand their permissions and make them your own.

${resources.length} ${resources.length === 1 ? 'resource' : 'resources'}Open examples · MIT licensed · v1.0.0
${resources.map((item) => `
${pill(categories.get(item.id) || item.type)}${item.verification.valid ? '✓ Signed' : 'Verification failed'}

${e(item.name)}

${e(item.description)}

${e(item.id.replace('w3bs://', ''))}
${e(item.publisher.namespace)} / ${e(item.version)}
`).join('') || '

No matching resources.

Try another phrase or view all prompts.

'}
`; } -function browse(store, query) { +async function browse(store, query) { let result, error; if (query.uri) { try { - result = store.inspect(query.uri); + result = await store.inspect(query.uri); } catch (failure) { error = failure.message; } @@ -133,7 +133,7 @@ function document(file, title) { if (!existsSync(filename)) throw new W3bsError('NOT_FOUND', 'Specification not found.', 404); return `
← Standards & documentationRaw Markdown ↗
${marked.parse(readFileSync(filename, 'utf8'))}
`; } -export function renderPage({ path, host, query, store, origin }) { +export async function renderPage({ path, host, query, store, origin }) { if (path === '/') path = host.startsWith('browse.') ? '/browse' @@ -147,13 +147,13 @@ export function renderPage({ path, host, query, store, origin }) { active = path; if (path === '/mission') { title = 'The Open Web for Humans, Agents & Devices'; - content = home(store); + content = await home(store); } else if (path === '/prompts') { title = 'Open Prompt Registry'; - content = promptCards(store, query); + content = await promptCards(store, query); } else if (path === '/browse') { title = 'Resource Browser'; - content = browse(store, query); + content = await browse(store, query); } else if (path === '/specs') { title = 'Open Standards'; content = specIndex(); diff --git a/src/store.mjs b/src/store.mjs index 0c06307..a2395fa 100644 --- a/src/store.mjs +++ b/src/store.mjs @@ -1,36 +1,120 @@ -import { DatabaseSync } from 'node:sqlite'; import { mkdirSync, readFileSync, readdirSync } from 'node:fs'; import { dirname, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; import { W3bsError, parseUri, validateManifest, verifyManifest, compareVersions } from './core.mjs'; export const ROOT = fileURLToPath(new URL('../', import.meta.url)); -export function openStore({ + +// Registry storage. PostgreSQL is the production backend (DATABASE_URL); SQLite +// remains for local development and tests. Both expose the same async adapter: +// all/get/run take SQL with `?` placeholders, and transaction(fn) runs fn against +// a connection that holds one transaction open. +const SCHEMA = { + sqlite: `PRAGMA journal_mode=WAL; PRAGMA busy_timeout=5000; + CREATE TABLE IF NOT EXISTS resources (id TEXT PRIMARY KEY, base TEXT NOT NULL, version TEXT NOT NULL, manifest TEXT NOT NULL, published_at TEXT NOT NULL); + CREATE INDEX IF NOT EXISTS resources_base ON resources(base); + CREATE TABLE IF NOT EXISTS revocations (id TEXT PRIMARY KEY REFERENCES resources(id), reason TEXT NOT NULL, revoked_at TEXT NOT NULL); + CREATE TABLE IF NOT EXISTS audit (sequence INTEGER PRIMARY KEY AUTOINCREMENT, operation TEXT NOT NULL, resource_id TEXT NOT NULL, at TEXT NOT NULL);`, + postgres: `CREATE TABLE IF NOT EXISTS resources (id TEXT PRIMARY KEY, base TEXT NOT NULL, version TEXT NOT NULL, manifest TEXT NOT NULL, published_at TEXT NOT NULL); + CREATE INDEX IF NOT EXISTS resources_base ON resources(base); + CREATE TABLE IF NOT EXISTS revocations (id TEXT PRIMARY KEY REFERENCES resources(id), reason TEXT NOT NULL, revoked_at TEXT NOT NULL); + CREATE TABLE IF NOT EXISTS audit (sequence BIGSERIAL PRIMARY KEY, operation TEXT NOT NULL, resource_id TEXT NOT NULL, at TEXT NOT NULL);`, +}; + +async function openSqlite(path) { + const { DatabaseSync } = await import('node:sqlite'); + if (path !== ':memory:') mkdirSync(dirname(path), { recursive: true }); + const db = new DatabaseSync(path); + db.exec(SCHEMA.sqlite); + const bind = (target) => ({ + all: async (sql, params = []) => target.prepare(sql).all(...params), + get: async (sql, params = []) => target.prepare(sql).get(...params), + run: async (sql, params = []) => { + target.prepare(sql).run(...params); + }, + }); + return { + ...bind(db), + dialect: 'sqlite', + async transaction(fn) { + db.exec('BEGIN IMMEDIATE'); + try { + const result = await fn(bind(db)); + db.exec('COMMIT'); + return result; + } catch (error) { + db.exec('ROLLBACK'); + throw error; + } + }, + close: async () => db.close(), + }; +} + +async function openPostgres(connectionString) { + const { default: pg } = await import('pg'); + const sslmode = new URL(connectionString).searchParams.get('sslmode'); + const pool = new pg.Pool({ + connectionString, + max: 8, + // Railway's PostgreSQL presents a self-signed certificate on its public + // proxy and runs internal traffic without TLS. Honour sslmode without + // certificate verification; the internal URL carries no sslmode. + ...(sslmode && sslmode !== 'disable' ? { ssl: { rejectUnauthorized: false } } : {}), + }); + const positional = (sql) => { + let index = 0; + return sql.replaceAll('?', () => `$${++index}`); + }; + const bind = (target) => ({ + all: async (sql, params = []) => (await target.query(positional(sql), params)).rows, + get: async (sql, params = []) => (await target.query(positional(sql), params)).rows[0], + run: async (sql, params = []) => { + await target.query(positional(sql), params); + }, + }); + await pool.query(SCHEMA.postgres); + return { + ...bind(pool), + dialect: 'postgres', + async transaction(fn) { + const client = await pool.connect(); + try { + await client.query('BEGIN'); + const result = await fn(bind(client)); + await client.query('COMMIT'); + return result; + } catch (error) { + await client.query('ROLLBACK').catch(() => {}); + throw error; + } finally { + client.release(); + } + }, + close: () => pool.end(), + }; +} + +export async function openStore({ + databaseUrl = process.env.DATABASE_URL, path = resolve(process.env.W3BS_DATA_DIR || resolve(ROOT, '.local/data'), 'registry.sqlite'), trustStore, seed = true, } = {}) { - if (path !== ':memory:') mkdirSync(dirname(path), { recursive: true }); const trust = trustStore ?? JSON.parse( readFileSync(process.env.W3BS_TRUST_FILE || resolve(ROOT, 'fixtures/trust.json'), 'utf8'), ); - const db = new DatabaseSync(path); - db.exec(`PRAGMA journal_mode=WAL; PRAGMA busy_timeout=5000; - CREATE TABLE IF NOT EXISTS resources (id TEXT PRIMARY KEY, base TEXT NOT NULL, version TEXT NOT NULL, manifest TEXT NOT NULL, published_at TEXT NOT NULL); - CREATE INDEX IF NOT EXISTS resources_base ON resources(base); - CREATE TABLE IF NOT EXISTS revocations (id TEXT PRIMARY KEY REFERENCES resources(id), reason TEXT NOT NULL, revoked_at TEXT NOT NULL); - CREATE TABLE IF NOT EXISTS audit (sequence INTEGER PRIMARY KEY AUTOINCREMENT, operation TEXT NOT NULL, resource_id TEXT NOT NULL, at TEXT NOT NULL);`); - function revoked(id) { - return Boolean(db.prepare('SELECT id FROM revocations WHERE id = ?').get(id)); + const db = databaseUrl ? await openPostgres(databaseUrl) : await openSqlite(path); + async function revoked(id, tx = db) { + return Boolean(await tx.get('SELECT id FROM revocations WHERE id = ?', [id])); } - function inspect(uri) { + async function inspect(uri, tx = db) { const parsed = parseUri(uri); - let candidates = db - .prepare('SELECT manifest FROM resources WHERE base = ?') - .all(parsed.base) - .map((row) => JSON.parse(row.manifest)); + let candidates = ( + await tx.all('SELECT manifest FROM resources WHERE base = ?', [parsed.base]) + ).map((row) => JSON.parse(row.manifest)); if (parsed.version) candidates = candidates.filter((item) => parsed.version.includes('.') @@ -46,10 +130,10 @@ export function openStore({ requestedUri: uri, canonicalUri: manifest.id, manifest, - verification: verifyManifest(manifest, trust, revoked(manifest.id)), + verification: verifyManifest(manifest, trust, await revoked(manifest.id, tx)), }; } - function publish(input) { + async function publish(input) { const manifest = validateManifest(input); const verification = verifyManifest(manifest, trust); if (!verification.valid) @@ -58,48 +142,43 @@ export function openStore({ 'Signature must verify against an operator-pinned key for this publisher namespace.', 403, ); - const existing = db.prepare('SELECT manifest FROM resources WHERE id = ?').get(manifest.id); - if (existing) { - if (JSON.parse(existing.manifest).proof.digest === manifest.proof.digest) - return inspect(manifest.id); - throw new W3bsError( - 'IMMUTABLE_VERSION', - 'This version already exists. Publish a new version.', - 409, - ); - } const now = new Date().toISOString(); - db.exec('BEGIN IMMEDIATE'); - try { - db.prepare('INSERT INTO resources VALUES (?, ?, ?, ?, ?)').run( + await db.transaction(async (tx) => { + const existing = await tx.get('SELECT manifest FROM resources WHERE id = ?', [manifest.id]); + if (existing) { + if (JSON.parse(existing.manifest).proof.digest === manifest.proof.digest) return; + throw new W3bsError( + 'IMMUTABLE_VERSION', + 'This version already exists. Publish a new version.', + 409, + ); + } + await tx.run('INSERT INTO resources VALUES (?, ?, ?, ?, ?)', [ manifest.id, parseUri(manifest.id).base, manifest.version, JSON.stringify(manifest), now, - ); - db.prepare('INSERT INTO audit(operation, resource_id, at) VALUES (?, ?, ?)').run( + ]); + await tx.run('INSERT INTO audit(operation, resource_id, at) VALUES (?, ?, ?)', [ 'publish', manifest.id, now, - ); - db.exec('COMMIT'); - } catch (error) { - db.exec('ROLLBACK'); - throw error; - } + ]); + }); return inspect(manifest.id); } - function search(query = '', type = '') { + async function search(query = '', type = '') { if (typeof query !== 'string' || query.length > 200) throw new W3bsError('INVALID_QUERY', 'Search is limited to 200 characters.'); const latest = new Map(); - for (const row of db.prepare('SELECT manifest FROM resources').all()) { + for (const row of await db.all('SELECT manifest FROM resources')) { const manifest = JSON.parse(row.manifest), base = parseUri(manifest.id).base; if (!latest.has(base) || compareVersions(manifest.version, latest.get(base).version) > 0) latest.set(base, manifest); } + const revokedIds = new Set((await db.all('SELECT id FROM revocations')).map((row) => row.id)); return [...latest.values()] .filter( (item) => @@ -116,33 +195,28 @@ export function openStore({ version: manifest.version, publisher: manifest.publisher, permissions: manifest.permissions, - verification: verifyManifest(manifest, trust, revoked(manifest.id)), + verification: verifyManifest(manifest, trust, revokedIds.has(manifest.id)), })); } - function revoke(uri, reason) { + async function revoke(uri, reason) { if (!parseUri(uri).version?.includes('.')) throw new W3bsError('EXACT_VERSION_REQUIRED', 'Revocation requires a fully versioned URI.'); - const { canonicalUri } = inspect(uri); + const { canonicalUri } = await inspect(uri); if (typeof reason !== 'string' || reason.trim().length < 3 || reason.length > 1000) throw new W3bsError('INVALID_REASON', 'Provide a revocation reason of 3–1000 characters.'); const now = new Date().toISOString(); - db.exec('BEGIN IMMEDIATE'); - try { - db.prepare('INSERT OR IGNORE INTO revocations VALUES (?, ?, ?)').run( + await db.transaction(async (tx) => { + await tx.run('INSERT INTO revocations VALUES (?, ?, ?) ON CONFLICT (id) DO NOTHING', [ canonicalUri, reason, now, - ); - db.prepare('INSERT INTO audit(operation, resource_id, at) VALUES (?, ?, ?)').run( + ]); + await tx.run('INSERT INTO audit(operation, resource_id, at) VALUES (?, ?, ?)', [ 'revoke', canonicalUri, now, - ); - db.exec('COMMIT'); - } catch (error) { - db.exec('ROLLBACK'); - throw error; - } + ]); + }); return inspect(canonicalUri); } if (seed) { @@ -152,8 +226,17 @@ export function openStore({ const manifest = JSON.parse( readFileSync(resolve(ROOT, 'fixtures/resources', filename), 'utf8'), ); - if (!db.prepare('SELECT id FROM resources WHERE id = ?').get(manifest.id)) publish(manifest); + if (!(await db.get('SELECT id FROM resources WHERE id = ?', [manifest.id]))) + await publish(manifest); } } - return { inspect, publish, search, revoke, trust, close: () => db.close() }; + return { + inspect, + publish, + search, + revoke, + trust, + dialect: db.dialect, + close: () => db.close(), + }; } diff --git a/test/cli.test.mjs b/test/cli.test.mjs index dc844e3..6f83953 100644 --- a/test/cli.test.mjs +++ b/test/cli.test.mjs @@ -32,7 +32,7 @@ test( writeFileSync(trustPath, JSON.stringify(trust)); const token = 'c'.repeat(32); instance = await startServer({ - store: openStore({ path: ':memory:', trustStore: trust }), + store: await openStore({ path: ':memory:', trustStore: trust }), port: 0, publishToken: token, }); diff --git a/test/core.test.mjs b/test/core.test.mjs index 6b015fb..ed53a63 100644 --- a/test/core.test.mjs +++ b/test/core.test.mjs @@ -140,81 +140,142 @@ test('template execution requires consent and treats inserted template syntax as code: 'UNSUPPORTED_CAPABILITY', }); }); -test('numeric aliases, immutable publication and revoked latest versions are preserved', () => { - const p = publisher(), - store = openStore({ path: ':memory:', trustStore: p.trust, seed: false }); - try { - store.publish(signed(p, '1.2.0')); - store.publish(signed(p, '1.10.0')); - store.publish(signed(p, '2.0.0')); - assert.equal(store.inspect('w3bs://prompt/example/research@1').manifest.version, '1.10.0'); - assert.equal(store.inspect('w3bs://prompt/example/research').manifest.version, '2.0.0'); - assert.throws( - () => - store.publish( - signed(p, '1.2.0', (m) => { - m.name = 'Changed'; +// The registry contract is exercised against SQLite always, and against +// PostgreSQL as well when W3BS_TEST_DATABASE_URL points at a disposable database. +const backends = [{ name: 'sqlite', options: () => ({ path: ':memory:' }) }]; +if (process.env.W3BS_TEST_DATABASE_URL) + backends.push({ + name: 'postgres', + options: () => ({ databaseUrl: process.env.W3BS_TEST_DATABASE_URL }), + }); +async function resetPostgres() { + if (!process.env.W3BS_TEST_DATABASE_URL) return; + const store = await openStore({ databaseUrl: process.env.W3BS_TEST_DATABASE_URL, seed: false }); + await store.close(); + const { default: pg } = await import('pg'); + const client = new pg.Client({ connectionString: process.env.W3BS_TEST_DATABASE_URL }); + await client.connect(); + await client.query('DROP TABLE IF EXISTS audit, revocations, resources'); + await client.end(); +} +for (const backend of backends) { + test(`[${backend.name}] numeric aliases, immutable publication and revoked latest versions are preserved`, async () => { + await resetPostgres(); + const p = publisher(), + store = await openStore({ ...backend.options(), trustStore: p.trust, seed: false }); + try { + assert.equal(store.dialect, backend.name); + await store.publish(signed(p, '1.2.0')); + await store.publish(signed(p, '1.10.0')); + await store.publish(signed(p, '2.0.0')); + assert.equal( + (await store.inspect('w3bs://prompt/example/research@1')).manifest.version, + '1.10.0', + ); + assert.equal( + (await store.inspect('w3bs://prompt/example/research')).manifest.version, + '2.0.0', + ); + await assert.rejects( + () => + store.publish( + signed(p, '1.2.0', (m) => { + m.name = 'Changed'; + }), + ), + { code: 'IMMUTABLE_VERSION' }, + ); + // Republishing the identical signed artifact is idempotent, not a conflict. + assert.equal((await store.publish(signed(p, '1.2.0'))).manifest.version, '1.2.0'); + await store.revoke('w3bs://prompt/example/research@1.10.0', 'Withdraw unsafe example'); + await store.revoke('w3bs://prompt/example/research@1.10.0', 'Revoking twice is harmless'); + const result = await store.inspect('w3bs://prompt/example/research@1'); + assert.equal(result.manifest.version, '1.10.0'); + assert.equal(result.verification.revoked, true); + // Search lists only the newest version per resource, with its revocation state. + const latest = () => + store.search().then((items) => items.find((item) => item.id.includes('example/research'))); + assert.equal((await latest()).version, '2.0.0'); + assert.equal((await latest()).verification.revoked, false); + await store.revoke('w3bs://prompt/example/research@2.0.0', 'Withdraw latest'); + assert.equal((await latest()).version, '2.0.0'); + assert.equal((await latest()).verification.revoked, true); + await assert.rejects( + () => + dispatch(store, 'run', { + uri: result.canonicalUri, + inputs: { topic: 'x' }, + consent: true, }), - ), - { code: 'IMMUTABLE_VERSION' }, - ); - store.revoke('w3bs://prompt/example/research@1.10.0', 'Withdraw unsafe example'); - const result = store.inspect('w3bs://prompt/example/research@1'); - assert.equal(result.manifest.version, '1.10.0'); - assert.equal(result.verification.revoked, true); - assert.throws( - () => - dispatch(store, 'run', { uri: result.canonicalUri, inputs: { topic: 'x' }, consent: true }), - { code: 'VERIFICATION_FAILED' }, - ); - } finally { - store.close(); - } -}); -test('publication and revocation survive reopening a persistent registry', () => { - const directory = mkdtempSync(resolve(tmpdir(), 'w3bs-store-')), - p = publisher(); - const config = { path: resolve(directory, 'test.sqlite'), trustStore: p.trust, seed: false }; - let store = openStore(config); - try { - store.publish(signed(p)); - store.revoke('w3bs://prompt/example/research@1.0.0', 'Withdrawn'); - store.close(); - store = openStore(config); - assert.equal(store.inspect('w3bs://prompt/example/research@1').verification.revoked, true); - } finally { - store.close(); - rmSync(directory, { recursive: true }); - } -}); -test('administrative operations fail closed when authorization is missing', () => { - const store = openStore({ path: ':memory:' }); - try { - assert.equal(store.search().length, 10); - for (const result of store.search()) assert.equal(result.verification.valid, true); - assert.throws(() => dispatch(store, 'publish', { manifest: fixture }, { publishToken: '' }), { - code: 'PUBLISHING_DISABLED', - }); - assert.throws( - () => - dispatch( - store, - 'publish', - { manifest: fixture }, - { publishToken: 'a'.repeat(32), token: 'wrong' }, - ), - { code: 'UNAUTHORIZED' }, - ); - assert.equal( - dispatch( - store, - 'publish', - { manifest: fixture }, - { publishToken: 'a'.repeat(32), token: 'a'.repeat(32) }, - ).verification.valid, - true, - ); - } finally { - store.close(); - } -}); + { code: 'VERIFICATION_FAILED' }, + ); + } finally { + await store.close(); + } + }); + test(`[${backend.name}] publication and revocation survive reopening a persistent registry`, async () => { + await resetPostgres(); + const directory = mkdtempSync(resolve(tmpdir(), 'w3bs-store-')), + p = publisher(); + const config = { + ...(backend.name === 'sqlite' + ? { path: resolve(directory, 'test.sqlite') } + : backend.options()), + trustStore: p.trust, + seed: false, + }; + let store = await openStore(config); + try { + await store.publish(signed(p)); + await store.revoke('w3bs://prompt/example/research@1.0.0', 'Withdrawn'); + await store.close(); + store = await openStore(config); + assert.equal( + (await store.inspect('w3bs://prompt/example/research@1')).verification.revoked, + true, + ); + } finally { + await store.close(); + rmSync(directory, { recursive: true }); + } + }); + test(`[${backend.name}] administrative operations fail closed when authorization is missing`, async () => { + await resetPostgres(); + const store = await openStore(backend.options()); + try { + assert.equal((await store.search()).length, 10); + for (const result of await store.search()) assert.equal(result.verification.valid, true); + // Seeding is idempotent: a second open against the same database adds nothing. + const again = await openStore(backend.options()); + assert.equal((await again.search()).length, 10); + await again.close(); + await assert.rejects( + () => dispatch(store, 'publish', { manifest: fixture }, { publishToken: '' }), + { code: 'PUBLISHING_DISABLED' }, + ); + await assert.rejects( + () => + dispatch( + store, + 'publish', + { manifest: fixture }, + { publishToken: 'a'.repeat(32), token: 'wrong' }, + ), + { code: 'UNAUTHORIZED' }, + ); + assert.equal( + ( + await dispatch( + store, + 'publish', + { manifest: fixture }, + { publishToken: 'a'.repeat(32), token: 'a'.repeat(32) }, + ) + ).verification.valid, + true, + ); + } finally { + await store.close(); + } + }); +} diff --git a/test/integration.test.mjs b/test/integration.test.mjs index ef95f3e..7631441 100644 --- a/test/integration.test.mjs +++ b/test/integration.test.mjs @@ -21,7 +21,7 @@ test( ); test('HTTP rejects hostile origins, arbitrary hosts, invalid bodies and unauthenticated writes', async () => { const instance = await startServer({ - store: openStore({ path: ':memory:' }), + store: await openStore({ path: ':memory:' }), port: 0, publicOrigin: 'http://localhost:3000', publishToken: 'a'.repeat(32), @@ -157,7 +157,7 @@ test('a new publisher can publish, render, revoke and inspect through HTTP', asy const manifest = signManifest(fixture, privateKey), token = 'b'.repeat(32); const instance = await startServer({ - store: openStore({ path: ':memory:', trustStore: trust, seed: false }), + store: await openStore({ path: ':memory:', trustStore: trust, seed: false }), port: 0, publishToken: token, }); @@ -188,8 +188,8 @@ test('a new publisher can publish, render, revoke and inspect through HTTP', asy } }); test('independent clients reject a valid signed artifact returned for the wrong URI', async () => { - const store = openStore({ path: ':memory:' }), - wrong = store.inspect('w3bs://prompt/w3bs/code-review@1'); + const store = await openStore({ path: ':memory:' }), + wrong = await store.inspect('w3bs://prompt/w3bs/code-review@1'); wrong.requestedUri = 'w3bs://prompt/w3bs/research@1'; const server = createServer((_req, res) => { res.setHeader('Content-Type', 'application/json'); @@ -205,6 +205,6 @@ test('independent clients reject a valid signed artifact returned for the wrong } finally { server.closeAllConnections(); await new Promise((resolve) => server.close(resolve)); - store.close(); + await store.close(); } });