diff --git a/.changeset/typed-multipart-json-documents.md b/.changeset/typed-multipart-json-documents.md new file mode 100644 index 00000000..712a0a28 --- /dev/null +++ b/.changeset/typed-multipart-json-documents.md @@ -0,0 +1,20 @@ +--- +"@cleverbrush/server": minor +"@cleverbrush/client": minor +"@cleverbrush/server-openapi": minor +"@cleverbrush/knex-schema": minor +"@cleverbrush/orm": minor +--- + +Add schema-based single and multiple file upload contracts, typed multipart client +serialization, and matching OpenAPI schemas. Enforce multipart body, file, field, +and part limits, reject truncated or duplicate singleton uploads, and support +file-only endpoints. Existing options-only uploads retain their single-file +shape and explicit MIME rejection reporting. + +Add lossless JSONB object reads and writes using native object schemas with +`.acceptUnknownProps().jsonb()`. Preserve nested extension data through returning +rows and projections, validate JSON extensions in the database layer, align +nullable object column DDL with reads, and track nested edits independently in +the ORM. Fix the PostgreSQL +upsert returning path exercised by document round trips. diff --git a/docs/framework-feature-candidates.md b/docs/framework-feature-candidates.md new file mode 100644 index 00000000..ff51e844 --- /dev/null +++ b/docs/framework-feature-candidates.md @@ -0,0 +1,275 @@ +# Framework feature candidates + +Status: F01–F03 implemented on the feature branch for PR review; F04–F07 remain proposed. +Assessment date: 2026-10-02. + +This is an unprioritized list of reusable Framework capabilities and correctness +fixes. Numbering identifies candidates; it does not indicate implementation order. +Priority fields are intentionally blank. There are no estimates or release +commitments, and proposed interfaces are not current supported APIs. + +## Review overview + +| ID | Candidate | Main packages | Priority | +| --- | --- | --- | --- | +| F01 | Reliable multiple-file multipart uploads | `server` | | +| F02 | Typed upload contracts, client support, and OpenAPI | `server`, `client`, `server-openapi` | | +| F03 | Lossless JSONB document storage | `knex-schema`, `orm` | | +| F04 | Provider-independent object storage | Proposed `storage` package | | +| F05 | S3-compatible storage adapter | Proposed `storage-s3` package | | +| F06 | CORS preflight support | `server` | | +| F07 | Consistent polymorphic ORM writes | `orm`, `knex-schema` | | + +Package names in the table omit the `@cleverbrush/` scope. New package names are +proposals for review. + +## F01 — Reliable multiple-file multipart uploads + +**Priority:** + +**Original assessment.** `.upload()` parses multipart requests and exposes +buffered files. The [parser](../libs/server/src/Server.ts) stores one file per field +name and does not handle all parser limit signals. Read-only HTTP probes found: + +| Request | Observed result | +| --- | --- | +| Two files under the same `files` field | Only the second file reached the handler | +| Eight-byte file with a four-byte limit | Success with four truncated bytes | +| Three files with a two-file limit | Third file silently discarded | +| Multipart body exceeding `maxBodySize` | Request accepted | +| Upload endpoint without a body schema | Uploaded files not populated | + +**Proposed capability.** Preserve multiple files per field and their order. Enforce +file, request, field, and part limits; never present truncated files as successful +uploads. Provide explicit rejection information, terminate interrupted parsing, +and release resources. File-only requests must work without an unrelated text +body schema. Bounded buffering is sufficient for the initial capability; streaming +or temporary-file upload modes can be considered separately. + +**Public API implications.** Extend upload options and handler file collections +without breaking existing single-file usage. Coordinate collection types and +rejection behavior with F02. + +**Acceptance criteria.** + +- Repeated fields preserve every accepted file, filename, and byte sequence. +- Size/count limits, including total multipart size, produce explicit failures or + documented rejections without silent data loss. +- File-only requests, malformed bodies, disallowed types, truncated fields, and + disconnected requests have coverage. +- Existing single-file endpoints continue to work. + +**Implementation:** Added schema-based `file()` / `array(file())` uploads, +resource-limit enforcement, typed client serialization and matching OpenAPI. +See the [server upload guide](../libs/server/README.md#file-upload). + +**Review notes:** + +## F02 — Typed upload contracts, client support, and OpenAPI + +**Priority:** + +**Original assessment.** The typed client accepts one `FilePart` or `Blob` per +field. Its [request builder](../libs/client/src/client.ts) only creates multipart +bodies when a body argument is supplied. The +[OpenAPI generator](../libs/server-openapi/src/generateOpenApiSpec.ts) describes +multipart text fields but does not describe the uploaded file fields. + +**Proposed capability.** Declare file field names, requiredness, and cardinality +in endpoint contracts. Infer matching handler and browser-client types. Serialize +file collections as repeated multipart fields, including file-only requests, and +generate equivalent OpenAPI binary-file schemas. + +**Public API implications.** Extend endpoint upload metadata, inferred handler and +client argument types, and OpenAPI generation together. Preserve legacy single-file +calls. Keep shared contracts browser-safe and avoid duplicated application DTOs or +manual `FormData` construction in ordinary client calls. + +**Acceptance criteria.** + +- Type tests cover required/optional file fields and single/multiple cardinality. +- A real typed-client request reaches the server with all files intact. +- File-only requests work without a dummy body argument. +- OpenAPI describes the same file fields and requirements as the runtime contract. +- Existing client calls and browser builds remain compatible. + +**Review notes:** + +## F03 — Lossless JSONB document storage + +**Priority:** + +**Original assessment.** JSONB DDL already exists through `.jsonb()`. However, +the [read-schema compiler](../libs/knex-schema/src/read-schema.ts) reconstructs +objects from declared properties. A decoder probe using a stored document with +`type`, `scenes`, and extension data returned only `type` when that was the only +declared property. Nested `any` schemas and record-based JSONB columns were also +rejected. This probe exercised decoding, not a complete PostgreSQL round trip. + +**Proposed capability.** Use native `object({...}).jsonb()` schemas for document +columns. Add `.acceptUnknownProps()` at each object node whose undeclared JSON +fields must survive reads and write-returning results. Document roots are objects; +nested values can include arrays, scalars and nulls. Reject non-JSON extension +values before persistence, while retaining declared-field serialization and +strict object projection behavior. + +**Public API implications.** Reuse existing object schemas and database extensions. +Keep JSON storage validation, serialization and decoding in `knex-schema` and ORM +tracking in `orm`. Schema and JSON Schema packages remain database-agnostic. +Declared fields retain normal type inference; unknown fields are preserved at +runtime. Identifiers, ownership and revision metadata remain relational columns. + +**Acceptance criteria.** + +- Real PostgreSQL insert, update, select, projection, and ORM round trips preserve + nested objects, arrays, nulls, optional fields, and undeclared extension keys. +- Write-returning results and subsequent reads have the same document shape. +- Invalid values such as functions, cycles, and non-finite numbers are rejected. +- Type inference and mapping preserve the JSON-document contract without `any`. +- Equality is structural; JSON object key ordering is not a storage guarantee. +- Existing relational projection and strict-schema behavior remains unchanged. + +**Implementation:** Added native open-object preservation, database-local JSON +validation, PostgreSQL round-trip coverage and document-aware ORM tracking. +See the [JSONB guide](../libs/knex-schema/README.md#lossless-jsonb-documents). + +**Review notes:** + +## F04 — Provider-independent object storage + +**Priority:** + +**Existing support and gap.** HTTP file/stream results exist, but the Framework has +no reusable object-storage contract or implementation. Applications must currently +own provider access and object lifecycle plumbing themselves. + +**Proposed capability.** Introduce a small server-side storage abstraction for +writing, streaming reads, metadata lookup, copying, and deletion. Identify objects +by stable keys and carry content type, size, and relevant metadata. Support +cancellation and explicit resource ownership for streams. Allow applications to +configure stable public asset URLs independently of provider endpoints. + +**Public API implications.** Add a proposed `@cleverbrush/storage` package with +provider-neutral interfaces, results, and errors suitable for dependency injection. +The core package must not require an S3 SDK. Application ownership checks, +reference tracking, rendering, and database/filesystem consistency policies remain +application responsibilities. + +**Acceptance criteria.** + +- A shared adapter contract suite exercises read/write/stat/copy/delete behavior. +- Missing objects, failed writes, cancellation, and stream cleanup are explicit. +- Content metadata survives storage and retrieval. +- Public URL construction handles object keys correctly and does not expose + credentials or depend on temporary signed URLs. + +**Review notes:** + +## F05 — S3-compatible storage adapter + +**Priority:** + +**Existing support and gap.** There is no S3 adapter in the Framework. This +candidate supplies the first production implementation of F04. + +**Proposed capability.** Add an adapter using the modular AWS SDK, configurable +with endpoint, region, bucket, credentials, path-style addressing, and key prefix. +Support streaming transfers and metadata, plus a separately configured public +asset base URL. All durable assets can use object storage while processing tools +materialize temporary local inputs when needed. + +**Public API implications.** Add a proposed `@cleverbrush/storage-s3` package +implementing F04. Keep SDK types and credentials out of browser contracts. Preserve +public reads at stable URLs; bucket/CDN/proxy provisioning remains deployment +configuration. Direct browser uploads, private signed URLs, and provider-specific +features are separate future candidates. + +**Acceptance criteria.** + +- Run the storage contract suite against an S3-compatible test service. +- Exercise custom endpoints, path-style addressing, prefixes, public URL mapping, + metadata, streaming, copies, and deletion. +- Cover missing objects, failed/interrupted transfers, repeated operations, and + provider errors without leaking credentials. +- Verify object contents with an independent checksum or byte comparison rather + than assuming an ETag always represents a content checksum. + +**Review notes:** + +## F06 — CORS preflight support + +**Priority:** + +**Existing support and gap.** Applications can set response headers in middleware, +but [route matching](../libs/server/src/Server.ts) occurs first. An HTTP probe sent +an OPTIONS preflight to a POST route and received `405`; ordinary middleware never +ran. Application middleware alone therefore cannot handle that preflight. + +**Proposed capability.** Provide opt-in CORS configuration that handles preflight +before route rejection, while retaining normal authentication and authorization +for the actual request. Apply appropriate CORS headers to successful and error +responses. Support configured origins, methods, allowed/exposed headers, and +credential behavior. + +**Public API implications.** Add server configuration or a first-class CORS helper +with documented execution order. Avoid silently changing the execution order of +existing ordinary middleware. + +**Acceptance criteria.** + +- Preflight for a registered route succeeds when origin/method/headers are allowed. +- Disallowed origins, methods, and headers are not inadvertently authorized. +- Relevant validation, authentication, not-found, and method errors have consistent + CORS behavior. +- Actual protected requests still require authentication. +- Same-origin applications without CORS configuration remain unaffected. + +**Review notes:** + +## F07 — Consistent polymorphic ORM writes + +**Priority:** + +**Existing support and gap.** Ordinary entities support soft deletion and lifecycle +hooks. Source review of [variant writes](../libs/orm/src/variant-write.ts) found +direct Knex updates/deletes: variant deletion physically removes rows and these +paths bypass the ordinary write pipeline. This finding needs PostgreSQL +integration coverage before its complete behavior and compatibility impact are +considered verified. + +**Proposed capability.** Make polymorphic mutations honor applicable soft-delete, +timestamp, and lifecycle-hook metadata. Keep an explicit permanent-deletion path. +Define consistent behavior across single-table and class-table variants, including +transactional updates to related base/variant rows. + +**Public API implications.** Review variant update/delete/restore/permanent-delete +operations alongside ordinary DbSet behavior. Changing physical deletion to soft +deletion may affect existing consumers and requires an explicit compatibility and +release decision. Ordinary entity relationships remain a viable alternative while +this candidate is pending. + +**Acceptance criteria.** + +- PostgreSQL tests cover soft deletion, restoration, and permanent deletion for + both polymorphic storage strategies. +- Hooks and timestamps follow the documented ordinary-entity contract. +- Failures roll back base/variant writes together and respect query predicates. +- Type tests describe the supported mutation surface accurately. + +**Review notes:** + +## Dependencies and review boundaries + +- F01 and F02 form one coordinated upload capability; priorities remain open. +- F05 depends on F04. Its integration tests should reuse the storage contract suite. +- Existing bearer-token authentication, typed API calls, durable jobs, dependency + injection, logging, and tracing can be reused. Additional authentication methods + are not part of this list. +- Rendering engines, document revision rules, asset ownership, shared-asset cleanup, + and data-import tools belong to applications, rather than Framework packages. +- Each accepted candidate needs focused tests, documentation, and a changeset for + published-package changes. Required repository gates remain `npm run lint`, + `npm run build`, and `npm run test`; database/storage features also need their + relevant integration suites. + +**Overall review notes:** diff --git a/libs/client/README.md b/libs/client/README.md index 66a6723d..4eb317ae 100644 --- a/libs/client/README.md +++ b/libs/client/README.md @@ -31,6 +31,34 @@ standard. Business messages and network exceptions are never guessed into fields See the [multi-file action/form example](../react-form/README.md#server-validation-issues) for serialization boundaries and the form issue lifecycle. +## Typed file uploads + +An endpoint's upload schema determines its `files` argument. Single fields accept +`File`, `Blob`, or `FilePart`; array fields accept arrays of those values. + +```ts +import { createClient } from '@cleverbrush/client'; +import { defineApi, endpoint, file } from '@cleverbrush/server/contract'; +import { array, object } from '@cleverbrush/schema'; + +const api = defineApi({ assets: { + upload: endpoint.post('/assets').upload(object({ + images: array(file()).minLength(1), + cover: file().optional() + })) +} }); +const client = createClient(api); +await client.assets.upload({ files: { + images: [new File(['first'], 'first.txt'), new File(['second'], 'second.txt')] +} }); +``` + +File-only calls need no `body` argument. The client serializes arrays as repeated +multipart fields in order, omits undefined optional fields, and lets `FormData` +set the content-type boundary. Use `File` or `FilePart` to supply a filename; +a plain `Blob` uses the platform's default filename. Text fields remain in the +endpoint's separate `body` argument. + ## Overview `@cleverbrush/client` provides a Proxy-based HTTP client that infers all endpoint types (params, body, query, headers, responses) from an API contract defined with `defineApi()` from `@cleverbrush/server/contract`. No code generation or manual type annotations are needed. diff --git a/libs/client/src/client.ts b/libs/client/src/client.ts index 12021fd4..674aac32 100644 --- a/libs/client/src/client.ts +++ b/libs/client/src/client.ts @@ -203,41 +203,49 @@ export function createClient( // -- Body -- let body: string | FormData | undefined; - if (args?.body !== undefined && hasBody(method)) { + if (hasBody(method) && (meta.fileUpload || args?.body !== undefined)) { if (meta.fileUpload) { // Build FormData for multipart uploads const fd = new FormData(); if ( - args.body && + args?.body && typeof args.body === 'object' && !(args.body instanceof Blob) ) { for (const [key, val] of Object.entries(args.body)) { - fd.append(key, String(val)); + if (val !== undefined) fd.append(key, String(val)); } } // Append file fields from args.files - if (args.files) { + if (args?.files) { for (const [key, value] of Object.entries( args.files as Record )) { - if (value instanceof Blob) { - fd.append(key, value); - } else { - const fp = value as FilePart; - fd.append( - key, - new Blob([fp.buffer], { - type: fp.mimeType - }), - fp.filename - ); + for (const part of Array.isArray(value) + ? value + : [value]) { + if (part === undefined) continue; + if (part instanceof Blob) { + fd.append(key, part); + } else { + const fp = part as FilePart; + fd.append( + key, + new Blob([fp.buffer], { + type: fp.mimeType + }), + fp.filename + ); + } } } } body = fd; // Let the browser set Content-Type with boundary - delete reqHeaders['Content-Type']; + for (const name of Object.keys(reqHeaders)) { + if (name.toLowerCase() === 'content-type') + delete reqHeaders[name]; + } } else { reqHeaders['Content-Type'] = JSON_CONTENT_TYPE; body = JSON.stringify(args.body); diff --git a/libs/client/src/types.ts b/libs/client/src/types.ts index 22624f7f..807acf30 100644 --- a/libs/client/src/types.ts +++ b/libs/client/src/types.ts @@ -12,7 +12,9 @@ import type { EndpointBuilder, FilePart, ApiContract as ServerApiContract, - SubscriptionBuilder + SubscriptionBuilder, + UploadContract, + UploadFiles } from '@cleverbrush/server/contract'; // Re-export types shared between server and client @@ -52,6 +54,13 @@ type HasKeys = keyof T extends never ? false : true; type InferSchema = T extends SchemaBuilder ? InferType : T; +type ClientFile = T extends FilePart + ? FilePart | Blob + : T extends readonly (infer E)[] + ? ClientFile[] + : T; +type ClientFiles = { [K in keyof T]: ClientFile }; + /** * Assembles the parts of the request argument object conditionally. * Only keys that carry data are included. @@ -61,12 +70,12 @@ type CallArgsParts< TBody, TQuery, THeaders, - TUpload extends boolean + TUpload extends UploadContract > = (HasKeys extends true ? { params: TParams } : {}) & (TBody extends undefined ? {} : { body: InferSchema }) & (HasKeys extends true ? { query: TQuery } : {}) & (HasKeys extends true ? { headers: THeaders } : {}) & - (TUpload extends true ? { files: Record } : {}); + (TUpload extends false ? {} : { files: ClientFiles> }); /** * Extracts the typed request argument shape from an `EndpointBuilder`. diff --git a/libs/client/src/upload.test-d.ts b/libs/client/src/upload.test-d.ts new file mode 100644 index 00000000..60766899 --- /dev/null +++ b/libs/client/src/upload.test-d.ts @@ -0,0 +1,51 @@ +import { array, object } from '@cleverbrush/schema'; +import { + type ActionContext, + defineApi, + endpoint, + type FilePart, + file +} from '@cleverbrush/server/contract'; +import { expectTypeOf, it } from 'vitest'; +import { createClient } from './client.js'; +import type { EndpointCallArgs } from './types.js'; + +it('infers upload fields on server and client', () => { + const upload = endpoint + .post('/files') + .upload( + object({ + images: array(file()).minLength(1), + cover: file().optional() + }) + ) + .summary('Files'); + const legacy = endpoint.post('/legacy').upload(); + const api = defineApi({ assets: { upload, legacy } }); + const client = createClient(api); + expectTypeOf< + ActionContext['files']['images'] + >().toEqualTypeOf(); + expectTypeOf< + ActionContext['files']['cover'] + >().toEqualTypeOf(); + expectTypeOf< + EndpointCallArgs['files']['images'] + >().toEqualTypeOf<(FilePart | Blob)[]>(); + expectTypeOf< + EndpointCallArgs['files']['cover'] + >().toEqualTypeOf(); + expectTypeOf['files']>().toEqualTypeOf< + Record + >(); + client.assets.upload({ files: { images: [new Blob(['one'])] } }); + // @ts-expect-error required field missing + client.assets.upload({ files: {} }); + // @ts-expect-error multiple field requires an array + client.assets.upload({ files: { images: new Blob() } }); + // @ts-expect-error no dummy body is part of a files-only contract + client.assets.upload({ files: { images: [] }, body: {} }); + // @ts-expect-error undeclared file field + client.assets.upload({ files: { images: [], extra: new Blob() } }); + client.assets.legacy({ files: { image: new Blob() } }); +}); diff --git a/libs/client/tsconfig.build.json b/libs/client/tsconfig.build.json index 1e54764b..7f9fb763 100644 --- a/libs/client/tsconfig.build.json +++ b/libs/client/tsconfig.build.json @@ -18,6 +18,7 @@ }, "include": ["src/**/*.ts", "src/**/*.tsx"], "exclude": [ + "src/**/*.test-d.ts", "src/**/*.test.ts", "src/**/*.test.tsx", "src/**/*.spec.ts", diff --git a/libs/client/tsconfig.typecheck.json b/libs/client/tsconfig.typecheck.json new file mode 100644 index 00000000..c24a9fd6 --- /dev/null +++ b/libs/client/tsconfig.typecheck.json @@ -0,0 +1,6 @@ +{ + "extends": "./tsconfig.build.json", + "compilerOptions": { "noEmit": true, "strict": true }, + "include": ["src/**/*.test-d.ts"], + "exclude": [] +} diff --git a/libs/client/vitest.config.mts b/libs/client/vitest.config.mts index 7db16b24..eb2b867b 100644 --- a/libs/client/vitest.config.mts +++ b/libs/client/vitest.config.mts @@ -4,6 +4,11 @@ import { defineConfig } from 'vitest/config'; export default defineConfig({ test: { environment: 'jsdom', + typecheck: { + enabled: true, + include: ['src/**/*.test-d.ts'], + tsconfig: './tsconfig.typecheck.json' + }, include: ['src/**/*.{test,spec}.{ts,tsx}'] } }); diff --git a/libs/knex-schema/README.md b/libs/knex-schema/README.md index 26fa1779..419619a1 100644 --- a/libs/knex-schema/README.md +++ b/libs/knex-schema/README.md @@ -963,3 +963,47 @@ Optionally pass a `baseQuery` (e.g. a scoped `knex('users').where('deleted_at', | Execution | `.execute()`, `.first()`, `await builder` (thenable) | | Debugging | `.toQuery()`, `.toString()` | | Escape hatch | `.apply(fn)` | + + +## Lossless JSONB documents + +Use ordinary object schemas with `.jsonb()` for document columns. Enable +`.acceptUnknownProps()` on each object that must preserve extension data. + +```ts +import { array, number, object, string } from '@cleverbrush/knex-schema'; + +const Document = object({ + id: number().primaryKey(), + content: object({ + title: string(), + tags: array(string()), + metadata: object({ revision: number().optional() }).acceptUnknownProps() + }).acceptUnknownProps().jsonb(), + settings: object({}).acceptUnknownProps().jsonb().nullable() +}).hasTableName('documents'); +``` + +Open objects preserve undeclared JSON keys through inserts, updates, returning +results, reads, projections and mapping. This includes nested objects, arrays, +scalars and nulls inside the document. Document column roots are objects. +Declared properties retain their inferred types and existing read decoding rules; +undeclared properties do not acquire an inferred type. Strict objects project +only their declared properties, and relational projections remain unchanged. + +Optional or nullable document columns accept SQL NULL, exposed as JavaScript +`null` on reads. Omitting an optional column follows existing SQL default/null +rules. Required, non-nullable document columns reject null. DDL and generated +migrations use the same nullability rules. + +Database validation rejects non-JSON extension values before persistence, +including functions, cycles, non-finite numbers, undefined, accessors and +non-JSON instances. Declared fields retain existing serialization, including +schema-declared dates and omitted optional fields. Input preprocessors and +defaults are not replayed during persistence or database reads. + +Object key ordering is not a storage guarantee; compare documents structurally. +JSON numbers use JavaScript precision; use strings for exact decimals or large +integers. JSONB storage behavior belongs to `knex-schema` and `orm`; normal +object schemas describe API responses and JSON Schema without database-specific +builders. diff --git a/libs/knex-schema/integration/json-documents.test.ts b/libs/knex-schema/integration/json-documents.test.ts new file mode 100644 index 00000000..44260a17 --- /dev/null +++ b/libs/knex-schema/integration/json-documents.test.ts @@ -0,0 +1,237 @@ +import { randomUUID } from 'node:crypto'; +import { mapper } from '@cleverbrush/mapper'; +import { + array, + createDb, + date, + defineEntity, + generateCreateTable, + number, + object, + query, + string +} from '@cleverbrush/orm'; +import Knex from 'knex'; +import { afterAll, beforeAll, expect, it } from 'vitest'; + +const connection = process.env.QUERY_TEST_DATABASE_URL; +if (!connection) throw new Error('QUERY_TEST_DATABASE_URL is required'); +const knex = Knex({ client: 'pg', connection }); +const table = `json_documents_${randomUUID().replaceAll('-', '')}`; +const open = object({ + type: string(), + nested: object({ label: string().optional() }).acceptUnknownProps() +}) + .acceptUnknownProps() + .jsonb(); +const content = object({ + metadata: object({ tags: array(string()) }).acceptUnknownProps() +}) + .acceptUnknownProps() + .jsonb(); +const Document = object({ + id: number().primaryKey(), + document: content.hasColumnName('document_data'), + dates: object({ seen: date(), absent: string().optional() }).jsonb(), + open, + strict: object({ name: string() }).jsonb(), + optional: object({}).acceptUnknownProps().jsonb().optional(), + nullable: object({}).acceptUnknownProps().jsonb().nullable() +}).hasTableName(table); +const entity = defineEntity(Document); +const data = () => ({ + document: { + scenes: [ + { + objects: [{ kind: 'path', points: [1, 2.5, null] }], + extension: { a: true } + } + ], + metadata: { tags: [] } + }, + dates: { seen: new Date('2026-01-01T00:00:00Z'), absent: undefined }, + open: { + type: 'drawing', + nested: { extra: { enabled: true } }, + extension: [1, { a: null }] + }, + nullable: null, + strict: { name: 'known', ignored: true } +}); + +beforeAll(async () => { + await generateCreateTable(Document)(knex); +}); +afterAll(async () => { + await knex.schema.dropTableIfExists(table); + await knex.destroy(); +}); + +it('preserves complete documents on insert, select, update and returning', async () => { + const input = { id: 1, ...data() }; + const inserted = await query(knex, Document).insert(input); + expect(inserted).toEqual({ + ...input, + strict: { name: 'known' }, + dates: { seen: input.dates.seen }, + optional: null + }); + expect( + await query(knex, Document) + .where(t => t.id, 1) + .first() + ).toEqual(inserted); + const next = { + metadata: { tags: ['updated'] }, + nodes: [{ custom: { a: [null, false, 'ok'] } }] + }; + const updated = await query(knex, Document) + .where(t => t.id, 1) + .update({ document: next }); + expect(updated[0].document).toEqual(next); + const selected = await query(knex, Document) + .where(t => t.id, 1) + .select(t => ({ + payload: t.document, + permissive: t.open + })) + .first(); + expect(selected).toEqual({ + payload: next, + permissive: input.open + }); +}); + +it('stores optional and nullable object columns as SQL NULL', async () => { + await query(knex, Document).insert({ id: 10, ...data() }); + const stored = await knex(table) + .where('id', 10) + .select( + knex.raw( + 'optional IS NULL as omitted, nullable IS NULL as explicit, jsonb_typeof(document_data) as kind' + ) + ) + .first(); + expect(stored).toEqual({ omitted: true, explicit: true, kind: 'object' }); +}); + +it('preserves documents on insertMany, bulk inserts, bulk updates and upserts', async () => { + await query(knex, Document).insertMany([ + { id: 20, ...data() }, + { id: 21, ...data() } + ]); + await query(knex, Document).bulkInsert([{ id: 22, ...data() }]); + const result = await query(knex, Document).upsert( + { id: 20, ...data(), optional: { extra: [true] } }, + { conflictColumns: [t => t.id] } + ); + expect(result.optional).toEqual({ extra: [true] }); + await query(knex, Document).bulkUpdate([ + { where: { id: 21 }, set: { optional: { items: ['updated'] } } } + ]); + expect( + ( + await query(knex, Document) + .where(t => t.id, 21) + .first() + )?.optional + ).toEqual({ items: ['updated'] }); +}); + +it('keeps mapping and derived row validation lossless', async () => { + const read = query(knex, Document) + .where(t => t.id, 20) + .select(t => ({ document: t.document })); + const target = object({ document: content }); + const convert = mapper() + .configure(read.rowSchema, target, m => m) + .getSyncMapper(read.rowSchema, target); + const row = (await read.first())!; + expect(read.rowSchema.validate(row).valid).toBe(true); + expect(convert(row)).toEqual(row); +}); + +it('tracks nested changes, structural equality, reset and repeated saves independently', async () => { + await query(knex, Document).insert({ id: 30, ...data() }); + const db = createDb(knex, { documents: entity }, { tracking: true }); + const row = await db.documents.findOrFail(30); + const original = structuredClone(row.document); + row.document.metadata.tags.push('edited'); + expect(db.entry(row).isModified('document')).toBe(true); + expect((await db.saveChanges()).updated).toBe(1); + expect( + ( + await query(knex, Document) + .where(t => t.id, 30) + .first() + )?.document + ).toEqual(row.document); + expect((await db.saveChanges()).updated).toBe(0); + row.document = Object.fromEntries( + Object.entries(row.document).reverse() + ) as typeof row.document; + expect(db.entry(row).isModified()).toBe(false); + row.document.metadata.tags.push('discard'); + db.entry(row).reset(); + expect(row.document.metadata.tags).toEqual(['edited']); + row.document.metadata.tags.push('another'); + db.discardChanges(); + expect(row.document.metadata.tags).toEqual(['edited']); + expect(original).not.toEqual(row.document); + row.optional = { items: [null, { replaced: true }] }; + await db.saveChanges(); + expect( + ( + await query(knex, Document) + .where(t => t.id, 30) + .first() + )?.optional + ).toEqual(row.optional); +}); + +it('supports ORM save and rejects non-JSON document writes before issuing SQL', async () => { + const db = createDb(knex, { documents: entity }); + const saved = await db.documents.save({ id: 40, ...data() }); + expect(saved.document).toEqual(data().document); + const cycle: any = {}; + cycle.self = cycle; + for (const value of [ + null, + [], + 'text', + false, + 1, + { bad: undefined }, + { bad: () => 1 }, + { bad: Infinity }, + cycle + ]) { + const sql: unknown[] = []; + const listener = (statement: unknown) => sql.push(statement); + knex.on('query', listener); + try { + await expect( + query(knex, Document).insert({ + id: 90, + ...data(), + document: value as any + }) + ).rejects.toThrow(); + await expect( + query(knex, Document) + .where(t => t.id, 40) + .update({ document: value as any }) + ).rejects.toThrow(); + expect(sql).toHaveLength(0); + } finally { + knex.off('query', listener); + } + } + await expect( + query(knex, Document) + .where(t => t.id, 40) + .update({ + open: { type: 'drawing', nested: {}, bad: new Date() } + } as any) + ).rejects.toThrow(/JSON/); +}); diff --git a/libs/knex-schema/src/SchemaQueryBuilder.ts b/libs/knex-schema/src/SchemaQueryBuilder.ts index 197c057a..6c100f42 100644 --- a/libs/knex-schema/src/SchemaQueryBuilder.ts +++ b/libs/knex-schema/src/SchemaQueryBuilder.ts @@ -86,7 +86,7 @@ export interface ReadQueryShape { let readAliasSequence = 0; /** A typed SQL column; selectors receive descriptions, not row values. */ export interface ReadColumn - extends AliasedColumn> { + extends AliasedColumn, S> { /** @internal Decoding and projection metadata shared by the query compiler. */ readonly [READ_COLUMN]: ReadNode; /** @internal Captured JSON path beneath a storage column. */ diff --git a/libs/knex-schema/src/ddl.ts b/libs/knex-schema/src/ddl.ts index 6cd43e59..d9b92694 100644 --- a/libs/knex-schema/src/ddl.ts +++ b/libs/knex-schema/src/ddl.ts @@ -1,3 +1,4 @@ +import { isStorageNullable } from './json-storage.js'; // @cleverbrush/knex-schema — DDL generation from schema introspection import type { ObjectSchemaBuilder, SchemaBuilder } from '@cleverbrush/schema'; @@ -199,11 +200,11 @@ export function generateCreateTable( // Nullability if ( - propIntrospected.isRequired && + !isStorageNullable(propIntrospected) && !ext.primaryKey?.autoIncrement ) { column = column.notNullable(); - } else if (!propIntrospected.isRequired) { + } else if (isStorageNullable(propIntrospected)) { column = column.nullable(); } @@ -390,9 +391,12 @@ export function generateCreateTableSource( } // Nullability - if (propIntrospected.isRequired && !ext.primaryKey?.autoIncrement) { + if ( + !isStorageNullable(propIntrospected) && + !ext.primaryKey?.autoIncrement + ) { line += '.notNullable()'; - } else if (!propIntrospected.isRequired) { + } else if (isStorageNullable(propIntrospected)) { line += '.nullable()'; } diff --git a/libs/knex-schema/src/extension.ts b/libs/knex-schema/src/extension.ts index 5f75d00d..f60295a6 100644 --- a/libs/knex-schema/src/extension.ts +++ b/libs/knex-schema/src/extension.ts @@ -250,6 +250,13 @@ export const dbExtension = defineExtension({ } }, object: { + /** Override the SQL column name when this object is stored as JSON. */ + hasColumnName( + this: ObjectSchemaBuilder, + name: string + ) { + return hasColumnName.call(this, name); + }, /** * Set the SQL table name for this object schema. * diff --git a/libs/knex-schema/src/index.ts b/libs/knex-schema/src/index.ts index e727ed58..5cdc59b4 100644 --- a/libs/knex-schema/src/index.ts +++ b/libs/knex-schema/src/index.ts @@ -76,6 +76,7 @@ export { string, union } from './extension.js'; +export { encodeJsonColumn, isJsonColumn } from './json-storage.js'; // Mappers (from knex-eager) export { clearRow, MAPPERS, mapObject, mapValue } from './mappers.js'; // Migration generation diff --git a/libs/knex-schema/src/json-documents.test-d.ts b/libs/knex-schema/src/json-documents.test-d.ts new file mode 100644 index 00000000..54e1e628 --- /dev/null +++ b/libs/knex-schema/src/json-documents.test-d.ts @@ -0,0 +1,32 @@ +import type { InferType } from '@cleverbrush/schema'; +import Knex from 'knex'; +import { expectTypeOf, it } from 'vitest'; +import { array, number, object, query, string } from './index.js'; + +it('preserves declared document types through database reads and projections', () => { + const content = object({ + title: string(), + tags: array(string()), + metadata: object({ revision: number().optional() }).acceptUnknownProps() + }) + .acceptUnknownProps() + .jsonb(); + const schema = object({ + id: number().primaryKey(), + document: content, + optional: content.optional(), + nullable: content.nullable() + }).hasTableName('documents'); + type Content = InferType; + const read = query(Knex({ client: 'pg' }), schema); + type Row = InferType; + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + const projected = read.select(t => ({ payload: t.document })); + type Projected = InferType; + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + // @ts-expect-error Undeclared fields are preserved, not inferred as any. + const _arbitrary: string = ({} as Row).document.extension; +}); diff --git a/libs/knex-schema/src/json-storage.test.ts b/libs/knex-schema/src/json-storage.test.ts new file mode 100644 index 00000000..255b86c1 --- /dev/null +++ b/libs/knex-schema/src/json-storage.test.ts @@ -0,0 +1,116 @@ +import Knex from 'knex'; +import { expect, it, vi } from 'vitest'; +import { generateCreateTable, generateCreateTableSource } from './ddl.js'; +import { array, date, number, object, string } from './extension.js'; +import { encodeJsonColumn } from './json-storage.js'; +import { entitySchemaToTableState } from './migration.js'; +import { compileReadSchema } from './read-schema.js'; + +it('preserves open objects only where declared, including array elements', () => { + const node = compileReadSchema( + object({ + closed: object({ type: string() }), + open: array(object({ type: string() }).acceptUnknownProps()) + }) + .acceptUnknownProps() + .jsonb() + ); + const value = { + closed: { type: 'x', extra: 1 }, + open: [{ type: 'y', extra: [null, { a: 2 }] }], + unknown: { nested: true } + }; + const decoded = node.decode(value, 'document'); + expect(decoded).toEqual({ ...value, closed: { type: 'x' } }); + expect(node.schema.validate(decoded).valid).toBe(true); +}); + +it('keeps native object metadata and nullability through DDL, migrations and reads', () => { + const document = object({}).acceptUnknownProps().jsonb(); + const schema = object({ + id: number().primaryKey(), + payload: document.hasColumnName('data'), + optional: document.optional(), + nullable: document.nullable() + }).hasTableName('documents'); + const knex = Knex({ client: 'pg' }); + const ddl = generateCreateTable(schema)(knex).toQuery(); + expect(ddl).toContain('"data" jsonb not null'); + expect(ddl).toContain('"nullable" jsonb null'); + const source = generateCreateTableSource(schema).up; + expect(source).toContain( + "table.specificType('data', 'jsonb').notNullable()" + ); + expect(source).toContain( + "table.specificType('nullable', 'jsonb').nullable()" + ); + const state = entitySchemaToTableState(schema).columns; + expect(state.data).toMatchObject({ type: 'jsonb', nullable: false }); + expect(state.optional.nullable).toBe(true); + expect(state.nullable.nullable).toBe(true); + expect( + compileReadSchema(document.optional()).decode(null, 'payload') + ).toBeNull(); + expect( + compileReadSchema(document.nullable()).decode(null, 'payload') + ).toBeNull(); + expect(() => compileReadSchema(document).decode(null, 'payload')).toThrow(); +}); + +it('requires object roots and rejects invalid open data before encoding', () => { + const schema = object({}).acceptUnknownProps().jsonb(); + for (const value of [ + null, + undefined, + [], + 'text', + false, + 1, + new Date(), + { bad: NaN } + ]) + expect(() => encodeJsonColumn(schema, value)).toThrow(); + expect(encodeJsonColumn(schema, { nested: [null, true, 2, 'text'] })).toBe( + '{"nested":[null,true,2,"text"]}' + ); + expect(encodeJsonColumn(schema.optional(), undefined)).toBeUndefined(); + expect(encodeJsonColumn(schema.optional(), null)).toBeNull(); + expect(encodeJsonColumn(schema.nullable(), null)).toBeNull(); + expect(() => encodeJsonColumn(schema.nullable(), undefined)).toThrow(); +}); + +it('preserves declared date and optional field behavior without replaying input transforms', () => { + const preprocess = vi.fn(value => value); + const schema = object({ + seen: date(), + absent: string().optional(), + label: string().addPreprocessor(preprocess).default('default') + }) + .acceptUnknownProps() + .jsonb(); + const seen = new Date('2026-01-01T00:00:00Z'); + const encoded = encodeJsonColumn(schema, { + seen, + absent: undefined, + extra: { enabled: true } + }); + expect(JSON.parse(encoded as string)).toEqual({ + seen: seen.toISOString(), + extra: { enabled: true } + }); + expect(preprocess).not.toHaveBeenCalled(); + expect(() => encodeJsonColumn(schema, { seen, extra: seen })).toThrow( + /JSON/ + ); +}); + +it('keeps special property names as own properties in open document reads', () => { + const node = compileReadSchema(object({}).acceptUnknownProps().jsonb()); + const value = JSON.parse( + '{"__proto__":{"safe":true},"constructor":{"x":1}}' + ); + const result = node.decode(value, 'document'); + expect(result).toEqual(value); + expect(Object.hasOwn(result, '__proto__')).toBe(true); + expect(({} as any).safe).toBeUndefined(); +}); diff --git a/libs/knex-schema/src/json-storage.ts b/libs/knex-schema/src/json-storage.ts new file mode 100644 index 00000000..d719c10f --- /dev/null +++ b/libs/knex-schema/src/json-storage.ts @@ -0,0 +1,94 @@ +import type { SchemaBuilder } from '@cleverbrush/schema'; +import { assertJsonValue } from './json-validation.js'; + +type Schema = SchemaBuilder; + +/** @internal Whether an explicit object column stores a JSON document. */ +export function isJsonColumn(schema: Schema | undefined): boolean { + const info = schema?.introspect(); + return ( + info?.type === 'object' && + /^jsonb?$/i.test(String(info.extensions?.columnType)) + ); +} + +function validateExtras(schema: Schema, value: unknown): void { + if (value === null || value === undefined) return; + const info = schema.introspect() as any; + if ( + info.type === 'object' && + typeof value === 'object' && + !Array.isArray(value) + ) { + for (const key of Reflect.ownKeys(value)) { + const descriptor = Object.getOwnPropertyDescriptor(value, key)!; + const child = + typeof key === 'string' && Object.hasOwn(info.properties, key) + ? info.properties[key] + : undefined; + if (child) { + if (!('value' in descriptor)) + throw new TypeError( + 'JSON properties must not be accessors' + ); + validateExtras(child, descriptor.value); + } else if (info.acceptUnknownProps) { + if ( + typeof key !== 'string' || + !descriptor.enumerable || + !('value' in descriptor) + ) + throw new TypeError( + 'JSON extension properties must be enumerable string data properties' + ); + assertJsonValue(descriptor.value); + } + } + } else if ( + info.type === 'array' && + Array.isArray(value) && + info.elementSchema + ) { + for (const item of value) validateExtras(info.elementSchema, item); + } +} + +/** + * @internal Validate object storage and extension data before binding JSON text. + * Declared fields retain their existing serialization, including dates. Input + * preprocessors and defaults are not replayed at the persistence boundary. + */ +export function encodeJsonColumn( + schema: Schema | undefined, + value: unknown +): unknown { + if (!schema || !isJsonColumn(schema)) return value; + const info = schema.introspect(); + if (value === undefined && !info.isRequired) return undefined; + if (value === null && (info.isNullable || !info.isRequired)) return null; + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + (Object.getPrototypeOf(value) !== Object.prototype && + Object.getPrototypeOf(value) !== null) + ) + throw new TypeError('Expected a JSON object document'); + validateExtras(schema, value); + return JSON.stringify(value); +} + +/** @internal Optional and nullable JSON object columns accept SQL NULL. */ +export function isStorageNullable(info: { + type: string; + isRequired: boolean; + isNullable: boolean; + extensions?: Record; +}): boolean { + return ( + !info.isRequired || + (info.type === 'object' && + /^jsonb?$/i.test(String(info.extensions?.columnType)) && + info.isNullable) + ); +} diff --git a/libs/knex-schema/src/json-validation.test.ts b/libs/knex-schema/src/json-validation.test.ts new file mode 100644 index 00000000..7fb8b21c --- /dev/null +++ b/libs/knex-schema/src/json-validation.test.ts @@ -0,0 +1,43 @@ +import { expect, it, vi } from 'vitest'; +import { assertJsonValue } from './json-validation.js'; + +it('accepts nested JSON, shared references and null-prototype objects', () => { + const shared = { tags: [null, true, 2.5, 'text'] }; + const value = Object.assign( + Object.create(null), + JSON.parse('{"__proto__":{"safe":true}}'), + { a: shared, b: shared } + ); + expect(() => assertJsonValue(value)).not.toThrow(); + expect(Object.hasOwn(value, '__proto__')).toBe(true); + expect(({} as any).safe).toBeUndefined(); +}); + +it('rejects lossy extension values without invoking getters or serialization hooks', () => { + const cycle: any = {}; + cycle.self = cycle; + const getter = vi.fn(() => 1); + const toJSON = vi.fn(() => ({})); + const invalid = [ + undefined, + NaN, + Infinity, + 1n, + () => 1, + new Date(), + new Map(), + cycle, + new Array(2), + [undefined], + { a: undefined }, + Object.defineProperty({}, 'value', { get: getter, enumerable: true }), + Object.defineProperty({}, 'hidden', { value: 1 }), + { [Symbol('key')]: 1 }, + Object.assign([1], { extra: 2 }), + { toJSON } + ]; + for (const value of invalid) + expect(() => assertJsonValue(value)).toThrow(TypeError); + expect(getter).not.toHaveBeenCalled(); + expect(toJSON).not.toHaveBeenCalled(); +}); diff --git a/libs/knex-schema/src/json-validation.ts b/libs/knex-schema/src/json-validation.ts new file mode 100644 index 00000000..01d039d5 --- /dev/null +++ b/libs/knex-schema/src/json-validation.ts @@ -0,0 +1,64 @@ +/** + * Assert strict JSON without invoking getters or serialization hooks. + * Shared references are allowed; cycles and lossy JavaScript values are not. + */ +export function assertJsonValue(value: unknown): void { + const ancestors = new Set(); + const pending: { value: unknown; path: string; leave?: boolean }[] = [ + { value, path: '$' } + ]; + while (pending.length) { + const item = pending.pop()!; + const current = item.value; + if (item.leave) { + ancestors.delete(current as object); + continue; + } + if ( + current === null || + typeof current === 'string' || + typeof current === 'boolean' || + (typeof current === 'number' && Number.isFinite(current)) + ) + continue; + const invalid = (reason: string): never => { + throw new TypeError(`${item.path}: ${reason}`); + }; + if (typeof current !== 'object' || current === null) + invalid('Expected a JSON value'); + const object = current as object; + if (ancestors.has(object)) invalid('Cyclic JSON value'); + const array = Array.isArray(object); + if ( + !array && + Object.getPrototypeOf(object) !== Object.prototype && + Object.getPrototypeOf(object) !== null + ) + invalid('Expected a plain JSON object'); + ancestors.add(object); + pending.push({ value: object, path: item.path, leave: true }); + const keys = Reflect.ownKeys(object); + if (array && keys.length !== (object as unknown[]).length + 1) + invalid('Sparse arrays or extra array properties are not JSON'); + for (const key of keys) { + if (array && key === 'length') continue; + const descriptor = Object.getOwnPropertyDescriptor(object, key)!; + if ( + typeof key !== 'string' || + !descriptor.enumerable || + !('value' in descriptor) + ) + invalid('Only enumerable string data properties are JSON'); + if ( + array && + (!/^(0|[1-9]\d*)$/.test(String(key)) || + Number(key) >= (object as unknown[]).length) + ) + invalid('Extra array properties are not JSON'); + pending.push({ + value: descriptor.value, + path: `${item.path}[${JSON.stringify(key)}]` + }); + } + } +} diff --git a/libs/knex-schema/src/migration.ts b/libs/knex-schema/src/migration.ts index fc923d5f..e006b38f 100644 --- a/libs/knex-schema/src/migration.ts +++ b/libs/knex-schema/src/migration.ts @@ -1,3 +1,4 @@ +import { isStorageNullable } from './json-storage.js'; // @cleverbrush/knex-schema — Schema diff & migration generation import type { ObjectSchemaBuilder, SchemaBuilder } from '@cleverbrush/schema'; @@ -229,7 +230,7 @@ export function entitySchemaToTableState( columns[col] = { name: col, type: schemaTypeToDbType(propIntrospected, ext), - nullable: !propIntrospected.isRequired, + nullable: isStorageNullable(propIntrospected), defaultValue: ext.defaultTo ?? null, maxLength: ext.maxLength ?? propIntrospected.maxLength ?? null, numericPrecision: null @@ -388,7 +389,7 @@ export function diffSchema( addColumns.push({ name: col, type: schemaTypeToDbType(propIntrospected, ext), - nullable: !propIntrospected.isRequired, + nullable: isStorageNullable(propIntrospected), defaultValue: ext.defaultTo, references: ext.references, onDelete: ext.onDelete, @@ -398,7 +399,7 @@ export function diffSchema( // Column exists in both → check for alterations const changes: Record = {}; - const expectedNullable = !propIntrospected.isRequired; + const expectedNullable = isStorageNullable(propIntrospected); if (dbCol.nullable !== expectedNullable) { changes.nullable = { from: dbCol.nullable, diff --git a/libs/knex-schema/src/operations/helpers.ts b/libs/knex-schema/src/operations/helpers.ts index a39c1c01..034edd39 100644 --- a/libs/knex-schema/src/operations/helpers.ts +++ b/libs/knex-schema/src/operations/helpers.ts @@ -1,3 +1,4 @@ +import { encodeJsonColumn } from '../json-storage.js'; // @cleverbrush/knex-schema — Extracted helper functions from QuerySource import type { InferType } from '@cleverbrush/schema'; @@ -911,11 +912,12 @@ export function mapObjectToColumns( const state = getState(builder); const { propToCol } = buildColumnMap(state.localSchema); const result: Record = {}; + const properties = state.localSchema.introspect().properties; for (const [key, value] of Object.entries(obj)) { const colName = propToCol.get(key); if (colName) { - result[colName] = value; + result[colName] = encodeJsonColumn(properties[key], value); } else { result[key] = value; } diff --git a/libs/knex-schema/src/operations/insert.ts b/libs/knex-schema/src/operations/insert.ts index 8595f497..60b27c3f 100644 --- a/libs/knex-schema/src/operations/insert.ts +++ b/libs/knex-schema/src/operations/insert.ts @@ -388,16 +388,16 @@ export async function upsertImpl( const qb = state .knex(state.tableName) - .insert(mapObjectToColumns(builder, data as Record)) - .onConflict(cols); + .insert(mapObjectToColumns(builder, data as Record)); + const conflict = qb.onConflict(cols); if (opts.updateColumns && opts.updateColumns.length > 0) { const updateCols = opts.updateColumns.map( c => resolveColumn(builder, c, 'upsert') as string ); - (qb as any).merge(updateCols); + conflict.merge(updateCols); } else { - (qb as any).merge(); + conflict.merge(); } const [row] = await (qb as any).returning( diff --git a/libs/knex-schema/src/read-schema.ts b/libs/knex-schema/src/read-schema.ts index 16e8399f..68666b36 100644 --- a/libs/knex-schema/src/read-schema.ts +++ b/libs/knex-schema/src/read-schema.ts @@ -14,6 +14,7 @@ import { } from '@cleverbrush/schema'; import type { Knex } from 'knex'; import { buildColumnMap } from './columns.js'; +import { assertJsonValue } from './json-validation.js'; /** A schema builder accepted by the database-read schema compiler. */ export type ReadSchema = SchemaBuilder; @@ -207,7 +208,14 @@ export function compileReadSchema(source: ReadSchema, column = true): ReadNode { ]) ) ); - convert = (value, path) => decodeObject(children, value, path); + if (info.acceptUnknownProps) schema = schema.acceptUnknownProps(); + convert = (value, path) => + decodeObject( + children, + value, + path, + info.acceptUnknownProps === true + ); break; } case 'array': { @@ -277,11 +285,28 @@ export function compileReadSchema(source: ReadSchema, column = true): ReadNode { export function decodeObject( children: Record, value: any, - path: string + path: string, + preserveUnknown = false ): Record { if (!value || typeof value !== 'object' || Array.isArray(value)) throw new ReadSchemaError(`${path}: expected a database object`); const result: Record = {}; + if (preserveUnknown) { + for (const [key, extra] of Object.entries(value)) { + if (Object.hasOwn(children, key)) continue; + try { + assertJsonValue(extra); + } catch (error) { + throw new ReadSchemaError(`${path}.${key}: ${String(error)}`); + } + Object.defineProperty(result, key, { + value: extra, + enumerable: true, + configurable: true, + writable: true + }); + } + } for (const [key, child] of Object.entries(children)) { const decoded = child.decode(value[key], `${path}.${key}`); if (decoded !== undefined) diff --git a/libs/orm/README.md b/libs/orm/README.md index aebbc37c..52a65622 100644 --- a/libs/orm/README.md +++ b/libs/orm/README.md @@ -415,3 +415,16 @@ precision policy, grouped aggregates, cursor restrictions, and examples. query builder - [`@cleverbrush/orm-cli`](../orm-cli) — migration CLI tool - [API reference](https://cleverbrush.github.io/framework/api-docs/latest) + + +## JSON document columns + +Declare document columns with `object({...}).jsonb()` using the ORM schema +factories. Add `.acceptUnknownProps()` to preserve undeclared JSON fields. See [JSONB document contracts](../knex-schema/README.md#lossless-jsonb-documents) +for declarations, null behavior and permissive object schemas. + +Tracked JSON columns use independent document snapshots and structural comparison. +Editing a nested object or array marks the column modified; replacing it with an +equivalent document does not. `entry(entity).reset()` and `discardChanges()` restore +independent copies, so subsequent edits cannot mutate the saved snapshot. JSON +validation and encoding also apply to ORM saves and tracked updates. diff --git a/libs/orm/src/change-tracker.ts b/libs/orm/src/change-tracker.ts index edbd7657..f9e003ad 100644 --- a/libs/orm/src/change-tracker.ts +++ b/libs/orm/src/change-tracker.ts @@ -14,9 +14,11 @@ import { buildColumnMap, + encodeJsonColumn, getPrimaryKeyColumns, getRowVersionColumn, getVariants, + isJsonColumn, query as schemaQuery } from '@cleverbrush/knex-schema'; import type { Knex } from 'knex'; @@ -207,37 +209,107 @@ function extractPkValues( return pkInfo.propertyKeys.map(k => entity[k]); } -/** Take a shallow snapshot of the entity's persisted (non-relation) columns. */ +// Document columns need independent snapshots; relational values keep their +// existing identity semantics. Metadata stays outside the public snapshot. +const documentKeys = new WeakMap>(); + +function cloneDocument(value: any): any { + if (value === null || typeof value !== 'object') return value; + if (value instanceof Date) return new Date(value.getTime()); + if (Array.isArray(value)) return value.map(cloneDocument); + const result = Object.create(Object.getPrototypeOf(value)); + for (const key of Object.keys(value)) + Object.defineProperty(result, key, { + value: cloneDocument(value[key]), + enumerable: true, + writable: true, + configurable: true + }); + return result; +} + +function documentsEqual(a: any, b: any): boolean { + if (Object.is(a, b)) return true; + if (!a || !b || typeof a !== 'object' || typeof b !== 'object') + return false; + if (a instanceof Date || b instanceof Date) + return ( + a instanceof Date && + b instanceof Date && + a.getTime() === b.getTime() + ); + if (Array.isArray(a) !== Array.isArray(b)) return false; + const prototype = Object.getPrototypeOf(b); + if ( + !Array.isArray(b) && + prototype !== Object.prototype && + prototype !== null + ) + return false; + const keys = Reflect.ownKeys(a); + if (keys.length !== Reflect.ownKeys(b).length) return false; + return keys.every(key => { + const descriptor = Object.getOwnPropertyDescriptor(b, key); + return ( + descriptor && + descriptor.enumerable === + Object.getOwnPropertyDescriptor(a, key)!.enumerable && + 'value' in descriptor && + documentsEqual(a[key], descriptor.value) + ); + }); +} + +function sameColumn( + snapshot: Record, + key: string, + current: unknown +): boolean { + return documentKeys.get(snapshot)?.has(key) + ? documentsEqual(snapshot[key], current) + : Object.is(snapshot[key], current); +} + +function restoreColumn( + snapshot: Record, + key: string +): unknown { + return documentKeys.get(snapshot)?.has(key) + ? cloneDocument(snapshot[key]) + : snapshot[key]; +} + +/** Snapshot document contents independently, preserving other column semantics. */ function snapshotEntity(entity: object, schema: any): Record { - const introspected = (schema as any).introspect?.() as { - properties?: Record; - }; - const propKeys = new Set(Object.keys(introspected?.properties ?? {})); + const properties = { ...schema.introspect().properties }; const variants = getVariants(schema); const variant = variants?.variants[(entity as any)[variants.discriminatorKey]]; if (variant) - for (const key of Object.keys(variant.schema.introspect().properties)) - propKeys.add(key); + Object.assign(properties, variant.schema.introspect().properties); const snap: Record = {}; - for (const [k, v] of Object.entries(entity as Record)) { - if (propKeys.has(k)) snap[k] = v; + const documents = new Set(); + for (const [k, v] of Object.entries(entity)) { + if (!Object.hasOwn(properties, k)) continue; + if (isJsonColumn(properties[k])) { + // Validate before snapshotting, including invalid nested mutations. + encodeJsonColumn(properties[k], v); + documents.add(k); + snap[k] = cloneDocument(v); + } else snap[k] = v; } + documentKeys.set(snap, documents); return Object.freeze(snap); } -/** Check if two values differ (shallow). */ function isDirty( original: Record, current: Record ): boolean { - for (const key of Object.keys(original)) { - if (!Object.is(original[key], current[key])) return true; - } - // Check for new keys on current that weren't in original - for (const key of Object.keys(current)) { + for (const key of Object.keys(original)) + if (!sameColumn(original, key, current[key])) return true; + for (const key of Object.keys(current)) if (!(key in original) && current[key] !== undefined) return true; - } return false; } @@ -438,8 +510,9 @@ export class ChangeTracker { isModified(field?: keyof T): boolean { const current = entity as Record; if (field !== undefined) { - return !Object.is( - rawEntry.originalSnapshot[field as string], + return !sameColumn( + rawEntry.originalSnapshot, + field as string, current[field as string] ); } @@ -448,10 +521,8 @@ export class ChangeTracker { reset(): void { // Restore current values from snapshot const current = entity as Record; - for (const [k, v] of Object.entries( - rawEntry.originalSnapshot - )) { - current[k] = v; + for (const k of Object.keys(rawEntry.originalSnapshot)) { + current[k] = restoreColumn(rawEntry.originalSnapshot, k); } rawEntry.state = rawEntry.pkKey ? 'Unchanged' : 'Added'; } @@ -484,8 +555,8 @@ export class ChangeTracker { ) { // Restore values from snapshot const current = entry.entity as Record; - for (const [k, v] of Object.entries(entry.originalSnapshot)) { - current[k] = v; + for (const k of Object.keys(entry.originalSnapshot)) { + current[k] = restoreColumn(entry.originalSnapshot, k); } entry.state = 'Unchanged'; } @@ -730,6 +801,10 @@ export class ChangeTracker { : new Map(); const variantData: Record = {}; const put = (key: string, value: unknown) => { + const property = + config.schema.introspect().properties[key] ?? + variant?.schema.introspect().properties[key]; + value = encodeJsonColumn(property, value); const variantColumn = !propToCol.has(key) && variantColumns.get(key); if (variantColumn && variant?.storage === 'cti') { @@ -742,8 +817,9 @@ export class ChangeTracker { for (const propKey of Object.keys(entry.originalSnapshot)) { if (pkPropSet.has(propKey)) continue; if ( - !Object.is( - entry.originalSnapshot[propKey], + !sameColumn( + entry.originalSnapshot, + propKey, current[propKey] ) ) { diff --git a/libs/orm/src/variant-write.ts b/libs/orm/src/variant-write.ts index 0f4eb8e6..4240009c 100644 --- a/libs/orm/src/variant-write.ts +++ b/libs/orm/src/variant-write.ts @@ -7,6 +7,7 @@ import { buildColumnMap, + encodeJsonColumn, getPrimaryKeyColumns, getVariants, object, @@ -183,7 +184,10 @@ export async function insertVariant( const variantRow: Record = { [fkCol]: pkValue }; for (const [propKey, val] of Object.entries(variantPayload)) { const colName = varPropToCol.get(propKey) ?? propKey; - variantRow[colName] = val; + variantRow[colName] = encodeJsonColumn( + variantSchema.introspect().properties[propKey], + val + ); } const discColInVariant = varPropToCol.get(discKey); if (discColInVariant) { @@ -259,7 +263,11 @@ export async function updateVariant( const updateData: Record = {}; for (const [propKey, val] of Object.entries(set)) { - updateData[propToCol.get(propKey) ?? propKey] = val; + updateData[propToCol.get(propKey) ?? propKey] = encodeJsonColumn( + spec.schema.introspect().properties[propKey] ?? + schema.introspect().properties[propKey], + val + ); } await db(baseTable) @@ -279,7 +287,10 @@ export async function updateVariant( for (const [propKey, val] of Object.entries(set)) { // Don't allow updating the FK (it's the join column) if (propKey === fkCol) continue; - updateData[varPropToCol.get(propKey) ?? propKey] = val; + updateData[varPropToCol.get(propKey) ?? propKey] = encodeJsonColumn( + variantSchema.introspect().properties[propKey], + val + ); } if (Object.keys(updateData).length === 0) return; diff --git a/libs/server-integration-tests/tests/uploads.test.ts b/libs/server-integration-tests/tests/uploads.test.ts new file mode 100644 index 00000000..4cf50efe --- /dev/null +++ b/libs/server-integration-tests/tests/uploads.test.ts @@ -0,0 +1,290 @@ +import http from 'node:http'; +import { createClient } from '@cleverbrush/client'; +import { array, object, string } from '@cleverbrush/schema'; +import { + createServer, + defineApi, + endpoint, + file, + type Server, + type UploadOptions +} from '@cleverbrush/server'; +import { afterEach, expect, it, vi } from 'vitest'; + +let server: Server | undefined; +afterEach(async () => { + await server?.close(); +}); +async function setup(options: UploadOptions = {}, maxBodySize = 4096) { + const api = defineApi({ + assets: { + upload: endpoint.post('/files').upload( + object({ + images: array(file()).minLength(1).maxLength(3), + cover: file().optional() + }), + options + ), + mixed: endpoint + .post('/mixed') + .body(object({ title: string(), note: string().optional() })) + .upload(object({ image: file() }), options), + legacy: endpoint.post('/legacy').upload(options), + empty: endpoint + .post('/empty') + .upload( + object({ images: array(file()), cover: file().optional() }), + options + ) + } + }); + const handler = vi.fn(({ files, body, rejectedFiles }) => ({ + files: Object.fromEntries( + Object.entries(files).map(([name, value]) => [ + name, + Array.isArray(value) ? value.map(info) : info(value) + ]) + ), + body, + rejectedFiles + })); + server = await createServer({ maxBodySize }) + .handle(api.assets.upload, handler) + .handle(api.assets.mixed, handler) + .handle(api.assets.legacy, handler) + .handle(api.assets.empty, handler) + .listen(0); + const base = `http://127.0.0.1:${server.address!.port}`; + return { + base, + handler, + client: createClient(api, { + baseUrl: base, + headers: { 'content-type': 'application/json' } + }) + }; +} +function info(value: any) { + return { + filename: value.filename, + mimeType: value.mimeType, + text: value.buffer.toString(), + size: value.size + }; +} +function form(entries: [string, string | Blob, string?][]) { + const data = new FormData(); + for (const [key, value, filename] of entries) { + if (typeof value === 'string') data.append(key, value); + else data.append(key, value, filename); + } + return data; +} +function image(text = 'abc') { + return new Blob([text], { type: 'image/png' }); +} + +it('round trips typed client arrays, optional files, filenames and file-only requests', async () => { + const { client, handler } = await setup(); + const result: any = await client.assets.upload({ + files: { + images: [ + new File(['one'], 'first.png', { type: 'image/png' }), + { + filename: 'second.png', + mimeType: 'image/png', + buffer: Buffer.from('two'), + size: 3 + } + ] + } + }); + expect(result.files.images).toEqual([ + { filename: 'first.png', mimeType: 'image/png', text: 'one', size: 3 }, + { filename: 'second.png', mimeType: 'image/png', text: 'two', size: 3 } + ]); + expect(result.files.cover).toBeUndefined(); + expect(handler).toHaveBeenCalledOnce(); + const mixed: any = await client.assets.mixed({ + body: { title: 'Title', note: undefined }, + files: { image: image() } + }); + expect(mixed.body).toEqual({ title: 'Title' }); + const empty: any = await client.assets.empty({ files: { images: [] } }); + expect(empty.files).toEqual({ images: [] }); + const legacy: any = await client.assets.legacy({ + files: { avatar: image() } + }); + expect(legacy.files.avatar.text).toBe('abc'); +}); + +it.each([ + ['missing required files', [], 400, {}], + ['unknown field', [['extra', image(), 'a.png']], 400, {}], + [ + 'too many array elements', + Array.from({ length: 4 }, (_, i) => ['images', image(), `${i}.png`]), + 400, + {} + ], + [ + 'duplicate singleton', + [ + ['images', image(), 'a'], + ['cover', image(), 'b'], + ['cover', image(), 'c'] + ], + 400, + {} + ], + [ + 'MIME mismatch', + [['images', image(), 'a']], + 400, + { allowedMimeTypes: ['text/plain'] } + ], + ['file size', [['images', image('12345'), 'a']], 413, { maxFileSize: 4 }], + [ + 'file count', + [ + ['images', image(), 'a'], + ['images', image(), 'b'] + ], + 413, + { maxFileCount: 1 } + ], + [ + 'field size', + [ + ['note', '12345'], + ['images', image(), 'a'] + ], + 413, + { maxFieldSize: 4 } + ], + [ + 'field count', + [ + ['a', '1'], + ['b', '2'] + ], + 413, + { maxFieldCount: 1 } + ], + [ + 'part count', + [ + ['a', '1'], + ['images', image(), 'a'] + ], + 413, + { maxPartCount: 1 } + ], + [ + 'field name size', + [['images', image(), 'a']], + 413, + { maxFieldNameSize: 5 } + ], + ['text encoded as file name', [['images', 'not a file']], 400, {}] +] as const)('rejects %s without running the handler', async (_name, entries, status, limits) => { + const { base, handler } = await setup(limits); + const response = await fetch(`${base}/files`, { + method: 'POST', + body: form(entries as any) + }); + expect(response.status).toBe(status); + expect(response.headers.get('content-type')).toContain( + 'application/problem+json' + ); + expect(await response.json()).toMatchObject({ + status, + errors: expect.any(Array) + }); + expect(handler).not.toHaveBeenCalled(); +}); + +it('allows exact file, field and count limits without truncation', async () => { + const { base } = await setup({ + maxFileSize: 4, + maxFileCount: 1, + maxFieldSize: 4, + maxFieldCount: 1, + maxPartCount: 2 + }); + const response = await fetch(`${base}/mixed`, { + method: 'POST', + body: form([ + ['image', image('1234'), 'a'], + ['title', '1234'] + ]) + }); + expect(response.status).toBe(200); + expect(await response.json()).toMatchObject({ + files: { image: { text: '1234', size: 4 } }, + body: { title: '1234' } + }); +}); + +it('keeps legacy MIME rejections explicit and rejects duplicate legacy files', async () => { + const { base, handler } = await setup({ allowedMimeTypes: ['image/*'] }); + const response = await fetch(`${base}/legacy`, { + method: 'POST', + body: form([ + ['avatar', new Blob(['x'], { type: 'text/plain' }), 'bad.txt'] + ]) + }); + expect(response.status).toBe(200); + expect(await response.json()).toMatchObject({ + rejectedFiles: [{ fieldName: 'avatar', filename: 'bad.txt' }] + }); + handler.mockClear(); + const duplicate = await fetch(`${base}/legacy`, { + method: 'POST', + body: form([ + ['image', image(), 'a'], + ['image', image(), 'b'] + ]) + }); + expect(duplicate.status).toBe(400); + expect(handler).not.toHaveBeenCalled(); +}); + +it('rejects malformed boundaries and total size with and without content-length', async () => { + const { base, handler } = await setup({}, 512); + const malformed = await fetch(`${base}/files`, { + method: 'POST', + headers: { 'content-type': 'multipart/form-data; boundary=missing' }, + body: 'bad' + }); + expect(malformed.status).toBe(400); + const payload = + '--boundary\r\nContent-Disposition: form-data; name="images"; filename="a"\r\nContent-Type: image/png\r\n\r\n' + + 'x'.repeat(600) + + '\r\n--boundary--\r\n'; + for (const sized of [true, false]) { + const status = await new Promise(resolve => { + const req = http.request( + `${base}/files`, + { + method: 'POST', + headers: { + 'content-type': + 'multipart/form-data; boundary=boundary', + ...(sized + ? { 'content-length': Buffer.byteLength(payload) } + : {}) + } + }, + res => { + res.resume(); + res.on('end', () => resolve(res.statusCode!)); + } + ); + req.write(payload.slice(0, 150)); + req.end(payload.slice(150)); + }); + expect(status).toBe(413); + } + expect(handler).not.toHaveBeenCalled(); +}); diff --git a/libs/server-openapi/README.md b/libs/server-openapi/README.md index 8290de49..9fcf13ed 100644 --- a/libs/server-openapi/README.md +++ b/libs/server-openapi/README.md @@ -486,3 +486,16 @@ with recursive uses referencing that component. ## License BSD-3-Clause — see [LICENSE](../../LICENSE). + + +## Multipart upload contracts + +`.upload(object({ images: array(file()).minLength(1), cover: file().optional() }))` +emits a multipart object with a binary `images` array and an optional binary +`cover` property. Required fields and array bounds come from the same schema used +by the server and typed client. File-only endpoints emit a request body even +without `.body()`. When a text-body schema exists, its properties are combined +with the file properties in the multipart schema. + +Object response schemas explicitly using `.acceptUnknownProps()` remain open +in the generated schema, including responses containing JSON document fields. diff --git a/libs/server-openapi/src/generateOpenApiSpec.test.ts b/libs/server-openapi/src/generateOpenApiSpec.test.ts index 7882593d..5f2418f1 100644 --- a/libs/server-openapi/src/generateOpenApiSpec.test.ts +++ b/libs/server-openapi/src/generateOpenApiSpec.test.ts @@ -12,7 +12,7 @@ import type { EndpointMetadata, EndpointRegistration } from '@cleverbrush/server'; -import { endpoint } from '@cleverbrush/server'; +import { endpoint, file } from '@cleverbrush/server'; import { describe, expect, it } from 'vitest'; import { generateOpenApiSpec, @@ -1838,3 +1838,37 @@ describe('responsesSchemas', () => { expect(op.responses['200']).toBeUndefined(); }); }); + +it('describes file-only and mixed upload contracts with binary arrays and required fields', () => { + const files = object({ + images: array(file()).minLength(1).maxLength(3), + cover: file().optional() + }); + for (const text of [ + null, + object({ title: string() }).schemaName('UploadText') + ]) { + let ep = endpoint.post('/assets').upload(files); + if (text) ep = ep.body(text) as any; + const spec: any = generateOpenApiSpec( + makeOptions([{ endpoint: ep.introspect(), handler: () => {} }]) + ); + const request = spec.paths['/assets'].post.requestBody; + const schema = request.content['multipart/form-data'].schema; + expect(request.required).toBe(true); + expect(schema.properties.images).toEqual({ + type: 'array', + items: { type: 'string', format: 'binary' }, + minItems: 1, + maxItems: 3 + }); + expect(schema.properties.cover).toEqual({ + type: 'string', + format: 'binary' + }); + expect(schema.required).toEqual( + text ? ['title', 'images'] : ['images'] + ); + expect(schema.additionalProperties).toBe(false); + } +}); diff --git a/libs/server-openapi/src/generateOpenApiSpec.ts b/libs/server-openapi/src/generateOpenApiSpec.ts index f566eb3e..eb3f6f9a 100644 --- a/libs/server-openapi/src/generateOpenApiSpec.ts +++ b/libs/server-openapi/src/generateOpenApiSpec.ts @@ -7,7 +7,7 @@ import type { AuthenticationConfig, EndpointMetadata, EndpointRegistration, - UploadOptions, + UploadConfiguration, WebhookDefinition } from '@cleverbrush/server'; import { resolvePath } from './pathUtils.js'; @@ -162,24 +162,62 @@ function buildParameterObject( } function buildRequestBody( - bodySchema: SchemaBuilder, + bodySchema: SchemaBuilder | null, registry: SchemaRegistry, example?: unknown | null, examples?: Record< string, { summary?: string; description?: string; value: unknown } > | null, - fileUpload?: UploadOptions | null + fileUpload?: UploadConfiguration | null ): Record { - const bodyInfo = bodySchema.introspect() as any; + const bodyInfo = (bodySchema?.introspect() as any) ?? {}; const body: Record = { required: bodyInfo.isRequired !== false }; // When file uploads are enabled, emit multipart/form-data if (fileUpload) { - const jsonSchema = convertSchema(bodySchema, registry); - const mediaType: Record = { schema: jsonSchema }; + // Inline the outer text object so additionalProperties does not reject + // file fields; keep references for its nested property schemas. + const jsonSchema = convertSchema(bodySchema, node => + node === bodySchema ? null : registry.getName(node) + ); + const properties: Record = { + ...((jsonSchema.properties as Record) ?? {}) + }; + const required = [...((jsonSchema.required as string[]) ?? [])]; + for (const [name, field] of Object.entries( + fileUpload.schema?.introspect().properties ?? {} + )) { + const info = (field as SchemaBuilder).introspect() as any; + const binary = { type: 'string', format: 'binary' }; + const property: Record = + info.type === 'array' + ? { type: 'array', items: binary } + : binary; + if (info.description) property.description = info.description; + if (info.minLength !== undefined) + property.minItems = info.minLength; + if (info.maxLength !== undefined) + property.maxItems = info.maxLength; + properties[name] = property; + if (info.isRequired) required.push(name); + } + body.required = required.length > 0; + const mediaType: Record = { + schema: { + ...jsonSchema, + type: 'object', + properties, + ...(required.length ? { required } : {}), + additionalProperties: fileUpload.schema + ? bodyInfo.acceptUnknownProps === true + ? { type: 'string' } + : false + : { type: 'string', format: 'binary' } + } + }; body['content'] = { 'multipart/form-data': mediaType }; @@ -529,7 +567,7 @@ function buildOperation( if (parameters.length > 0) operation['parameters'] = parameters; // Request body - if (meta.bodySchema) { + if (meta.bodySchema || meta.fileUpload) { operation['requestBody'] = buildRequestBody( meta.bodySchema, registry, diff --git a/libs/server/README.md b/libs/server/README.md index 1dff6f41..9c2970d9 100644 --- a/libs/server/README.md +++ b/libs/server/README.md @@ -467,34 +467,54 @@ validated against the endpoint body schema. Repeated fields become arrays: ## File Upload -Accept file uploads via `multipart/form-data` by chaining `.upload()` on an endpoint: +Declare uploaded fields with schema builders. Import contracts through the +browser-safe entry point when sharing them with a client: ```ts -import { endpoint } from '@cleverbrush/server'; -import { object, string } from '@cleverbrush/schema'; +import { endpoint, file } from '@cleverbrush/server/contract'; +import { array, object, string } from '@cleverbrush/schema'; -const UploadAvatar = endpoint - .post('/api/avatar') - .upload({ maxFileSize: 2 * 1024 * 1024, allowedMimeTypes: ['image/*'] }) - .body(object({ description: string().optional() })) - .authorize(UserPrincipal); - -const handler: Handler = async ({ body, files }) => { - const avatar = files['avatar']; - // avatar: FilePart { filename, mimeType, buffer, size } - return ActionResult.created({ name: avatar.filename }); -}; +export const UploadAssets = endpoint.post('/api/assets') + .upload(object({ + images: array(file()).minLength(1).maxLength(3), + cover: file().optional() + }), { allowedMimeTypes: ['image/*'] }) + .body(object({ description: string().optional() })); ``` -The `files` object on the handler context contains one `FilePart` entry per uploaded file field. Non-file form fields are validated against the body schema and available via `body`. +Handlers receive `files.images: FilePart[]` in request order and an optional +`files.cover: FilePart`. Text fields are validated through `.body()`. Omit +`.body()` for a file-only endpoint. Text and file field names must be distinct. +A missing required array becomes `[]`; use `.minLength(1)` to require a file. +Absent optional file fields are omitted. + +Typed upload endpoints reject malformed or invalid requests with `400` Problem +Details before calling the handler. Resource limits return `413`, including +oversized files, text fields, field names, request bodies and part counts. +Truncated content is never passed to handlers. Files are buffered in memory +within these limits; authorization runs before parsing. MIME allowlists compare +the declared multipart MIME type; they do not inspect file contents. + +The options-only `.upload(options?)` overload accepts one `FilePart` per field. +It reports MIME exclusions through `rejectedFiles` (including `fieldName`) and +rejects duplicate file fields and limit violations. ### Options -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `maxFileSize` | `number` | 10 MB | Maximum file size per file in bytes | -| `allowedMimeTypes` | `string[]` | all | MIME type allowlist (supports `image/*` glob) | -| `maxFileCount` | `number` | 10 | Maximum number of files per request | +| Option | Default | Meaning | +| --- | --- | --- | +| `maxFileSize` | 10 MiB | Bytes per file | +| `maxFileCount` | 10 | Files per request, including rejected files | +| `allowedMimeTypes` | All | Declared MIME types; supports patterns such as `image/*` | +| `maxFieldSize` | 1 MiB | Bytes per text field | +| `maxFieldCount` | 100 | Text fields per request | +| `maxFieldNameSize` | 100 | UTF-8 bytes per field name | +| `maxPartCount` | File-count limit + field-count limit | Total file and text parts | + +Limits must be positive safe integers. The server's `maxBodySize` (default 5 MiB) +also limits the **entire multipart request**, including boundaries and headers. +Configure it large enough for the permitted file collection. It applies even +when the request has no `Content-Length`. ### FilePart type diff --git a/libs/server/src/Endpoint.ts b/libs/server/src/Endpoint.ts index 4edc86ef..bf9bf01f 100644 --- a/libs/server/src/Endpoint.ts +++ b/libs/server/src/Endpoint.ts @@ -33,6 +33,13 @@ import type { RejectedFile, UploadOptions } from './types.js'; +import { + type UploadConfiguration, + type UploadContract, + type UploadFiles, + type UploadSchema, + validateUploadConfiguration +} from './upload.js'; // --------------------------------------------------------------------------- // Simplify — flattens intersection types for clean IDE tooltips @@ -52,7 +59,7 @@ type ActionContextParts< TQuery, THeaders, TPrincipal, - TUpload extends boolean + TUpload extends UploadContract > = { context: RequestContext; } & (HasKeys extends true ? { params: TParams } : {}) & @@ -66,9 +73,11 @@ type ActionContextParts< (HasKeys extends true ? { query: TQuery } : {}) & (HasKeys extends true ? { headers: THeaders } : {}) & (TPrincipal extends undefined ? {} : { principal: TPrincipal }) & - (TUpload extends true - ? { files: Record; rejectedFiles?: RejectedFile[] } - : {}); + (TUpload extends false + ? {} + : TUpload extends true + ? { files: Record; rejectedFiles?: RejectedFile[] } + : { files: UploadFiles }); /** * The fully-typed argument object passed to endpoint handlers. @@ -582,7 +591,7 @@ export interface EndpointMetadata { * The configuration controls max file size, allowed MIME types, etc. * @see `EndpointBuilder.upload()` */ - readonly fileUpload: UploadOptions | null; + readonly fileUpload: UploadConfiguration | null; /** * Cache tags declared via `.clearsCacheTag()`, providing tag-based cache * key computation for the client middleware. @@ -685,7 +694,7 @@ export class EndpointBuilder< TRoles extends string = string, TResponse = any, TResponses extends Record = {}, - TUpload extends boolean = false + TUpload extends UploadContract = false > { readonly #method: string; readonly #basePath: string; @@ -749,7 +758,7 @@ export class EndpointBuilder< readonly #externalDocs: { url: string; description?: string } | null; readonly #links: Record | null; readonly #callbacks: Record | null; - readonly #fileUpload: UploadOptions | null; + readonly #fileUpload: UploadConfiguration | null; readonly #cacheTags: readonly CacheTagDefinition[]; constructor( @@ -815,9 +824,10 @@ export class EndpointBuilder< externalDocs: { url: string; description?: string } | null = null, links: Record | null = null, callbacks: Record | null = null, - fileUpload: UploadOptions | null = null, + fileUpload: UploadConfiguration | null = null, cacheTags: readonly CacheTagDefinition[] = [] ) { + validateUploadConfiguration(fileUpload, bodySchema); this.#method = method; this.#basePath = basePath; this.#pathTemplate = pathTemplate; @@ -1694,9 +1704,11 @@ export class EndpointBuilder< * * When set, the server parses the request body with a streaming multipart * parser instead of the default JSON deserializer. File fields are made - * available to the handler via `arg.files` (a `Record`), - * while non-file form fields are validated against the body schema and - * available via `arg.body`. + * available via `arg.files`. A schema of file() / array(file()) fields + * infers their names, cardinality and optionality. Without a schema, + * files remain a Record. Non-file fields use arg.body. + * File-only endpoints do not require a body schema. Typed uploads fail + * before the handler if their contract is invalid; limits produce 413. * * @param options - Upload configuration (max file size, allowed MIME types, etc.). * @@ -1714,6 +1726,21 @@ export class EndpointBuilder< * }; * ``` */ + upload( + schema: S, + options?: UploadOptions + ): EndpointBuilder< + TParams, + TBody, + TQuery, + THeaders, + TServices, + TPrincipal, + TRoles, + TResponse, + TResponses, + S + >; upload( options?: UploadOptions ): EndpointBuilder< @@ -1727,7 +1754,29 @@ export class EndpointBuilder< TResponse, TResponses, true + >; + upload( + schemaOrOptions?: UploadSchema | UploadOptions, + options?: UploadOptions + ): EndpointBuilder< + TParams, + TBody, + TQuery, + THeaders, + TServices, + TPrincipal, + TRoles, + TResponse, + TResponses, + any > { + const schema = + schemaOrOptions && 'introspect' in schemaOrOptions + ? schemaOrOptions + : undefined; + const config = schema + ? options + : (schemaOrOptions as UploadOptions | undefined); return new EndpointBuilder( this.#method, this.#basePath, @@ -1753,9 +1802,13 @@ export class EndpointBuilder< this.#links, this.#callbacks, { - maxFileSize: options?.maxFileSize ?? 10 * 1024 * 1024, - allowedMimeTypes: options?.allowedMimeTypes, - maxFileCount: options?.maxFileCount ?? 10 + ...config, + allowedMimeTypes: config?.allowedMimeTypes + ? [...config.allowedMimeTypes] + : undefined, + maxFileSize: config?.maxFileSize ?? 10 * 1024 * 1024, + maxFileCount: config?.maxFileCount ?? 10, + schema }, this.#cacheTags ); diff --git a/libs/server/src/ProblemDetails.ts b/libs/server/src/ProblemDetails.ts index c49fbc12..e2c381f5 100644 --- a/libs/server/src/ProblemDetails.ts +++ b/libs/server/src/ProblemDetails.ts @@ -26,6 +26,7 @@ const STATUS_TITLES: Record = { 404: 'Not Found', 405: 'Method Not Allowed', 409: 'Conflict', + 413: 'Payload Too Large', 415: 'Unsupported Media Type', 422: 'Unprocessable Content', 500: 'Internal Server Error', diff --git a/libs/server/src/Server.ts b/libs/server/src/Server.ts index 6da69ffd..ee9546da 100644 --- a/libs/server/src/Server.ts +++ b/libs/server/src/Server.ts @@ -14,13 +14,13 @@ import { requireRole } from '@cleverbrush/auth'; import { ServiceCollection, type ServiceProvider } from '@cleverbrush/di'; -import { Busboy } from '@fastify/busboy'; import { type WebSocket, WebSocketServer } from 'ws'; import { ActionResult, JsonResult } from './ActionResult.js'; import { ContentNegotiator } from './ContentNegotiator.js'; import type { EndpointBuilder, Handler, HandlerMapping } from './Endpoint.js'; import { HttpError } from './HttpError.js'; import { MiddlewarePipeline } from './MiddlewarePipeline.js'; +import { parseMultipart } from './multipart.js'; import { needsBody, resolveArgs } from './ParameterResolver.js'; import { createProblemDetails, @@ -40,8 +40,7 @@ import type { RejectedFile, ServerBatchingOptions, ServerOptions, - SubscriptionRegistration, - UploadOptions + SubscriptionRegistration } from './types.js'; import { VirtualIncomingMessage, @@ -93,107 +92,6 @@ export interface AuthorizationConfig { policies?: Record void>; } -// --------------------------------------------------------------------------- -// Multipart / file-upload helpers -// --------------------------------------------------------------------------- - -async function parseMultipart( - req: http.IncomingMessage, - options: UploadOptions -): Promise<{ - fields: Record; - files: Record; - rejectedFiles: RejectedFile[]; -}> { - const maxFileCount = options.maxFileCount ?? 10; - const maxFileSize = options.maxFileSize ?? 10 * 1024 * 1024; - const allowedMimeTypes = options.allowedMimeTypes; - - return new Promise((resolve, reject) => { - const fields: Record = {}; - const files: Record = {}; - const rejectedFiles: RejectedFile[] = []; - let fileCount = 0; - - const busboy = Busboy({ - headers: req.headers as { - 'content-type': string; - } & http.IncomingHttpHeaders, - limits: { - fileSize: maxFileSize, - files: maxFileCount - } - }); - - busboy.on('field', (fieldname: string, value: string) => { - fields[fieldname] = value; - }); - - busboy.on( - 'file', - ( - fieldname: string, - stream: import('@fastify/busboy').BusboyFileStream, - filename: string, - _transferEncoding: string, - mimeType: string - ) => { - if (fileCount >= maxFileCount) { - rejectedFiles.push({ - filename, - mimeType, - reason: `Exceeded max file count (${maxFileCount})` - }); - stream.resume(); - return; - } - - if (allowedMimeTypes) { - const allowed = allowedMimeTypes.some(pattern => { - if (pattern.endsWith('/*')) { - return mimeType.startsWith(pattern.slice(0, -1)); - } - return mimeType === pattern; - }); - if (!allowed) { - rejectedFiles.push({ - filename, - mimeType, - reason: `MIME type "${mimeType}" not allowed (allowed: ${allowedMimeTypes.join(', ')})` - }); - stream.resume(); - return; - } - } - - fileCount++; - const chunks: Buffer[] = []; - - stream.on('data', (chunk: Buffer) => { - chunks.push(chunk); - }); - - stream.on('end', () => { - const buffer = Buffer.concat(chunks); - files[fieldname] = { - filename, - mimeType, - buffer, - size: buffer.length - }; - }); - - stream.on('error', reject); - } - ); - - busboy.on('error', reject); - busboy.on('finish', () => resolve({ fields, files, rejectedFiles })); - - req.pipe(busboy); - }); -} - /** * Fluent builder for constructing and starting an HTTP server. * @@ -743,9 +641,11 @@ export class Server { // Parse body if needed let parsedBody: unknown; - let uploadedFiles: Record | undefined; + let uploadedFiles: + | Record + | undefined; let rejectedFiles: RejectedFile[] | undefined; - if (needsBody(meta)) { + if (needsBody(meta) || meta.fileUpload) { const contentType = req.headers['content-type'] ?? ''; // Multipart / file-upload path @@ -756,12 +656,33 @@ export class Server { try { const result = await parseMultipart( req, - meta.fileUpload + meta.fileUpload, + this.#maxBodySize ); + if ( + meta.fileUpload.schema && + !meta.bodySchema && + Object.keys(result.fields).length + ) { + throw new HttpError( + 400, + 'Bad Request', + 'This upload endpoint does not declare text fields', + { + errors: Object.keys(result.fields).map( + name => ({ + pointer: `/body/${name.replace(/~/g, '~0').replace(/\//g, '~1')}`, + detail: 'Unexpected multipart text field' + }) + ) + } + ); + } parsedBody = result.fields; uploadedFiles = result.files; rejectedFiles = result.rejectedFiles; - } catch { + } catch (error) { + if (error instanceof HttpError) throw error; const pd = createProblemDetails( 400, 'Malformed multipart request' diff --git a/libs/server/src/contract.ts b/libs/server/src/contract.ts index a51045ca..16ad64f0 100644 --- a/libs/server/src/contract.ts +++ b/libs/server/src/contract.ts @@ -308,3 +308,11 @@ export function omitGroups( } return Object.freeze(result); } + +export { + file, + type UploadConfiguration, + type UploadContract, + type UploadFiles, + type UploadSchema +} from './upload.js'; diff --git a/libs/server/src/index.ts b/libs/server/src/index.ts index e7a1e057..a7cd4275 100644 --- a/libs/server/src/index.ts +++ b/libs/server/src/index.ts @@ -123,4 +123,11 @@ export type { SubscriptionRegistration, UploadOptions } from './types.js'; +export { + file, + type UploadConfiguration, + type UploadContract, + type UploadFiles, + type UploadSchema +} from './upload.js'; export { defineWebhook, type WebhookDefinition } from './Webhook.js'; diff --git a/libs/server/src/multipart.test.ts b/libs/server/src/multipart.test.ts new file mode 100644 index 00000000..68c145f4 --- /dev/null +++ b/libs/server/src/multipart.test.ts @@ -0,0 +1,36 @@ +import type { IncomingMessage } from 'node:http'; +import { PassThrough } from 'node:stream'; +import { expect, it } from 'vitest'; +import { parseMultipart } from './multipart.js'; + +function request() { + return Object.assign(new PassThrough(), { + headers: { 'content-type': 'multipart/form-data; boundary=boundary' }, + complete: false + }) as unknown as IncomingMessage; +} + +it('rejects disconnects and removes request listeners while a file is incomplete', async () => { + const req = request(); + const parsing = parseMultipart(req, {}, 4096); + const rejection = expect(parsing).rejects.toMatchObject({ status: 400 }); + (req as unknown as PassThrough).write( + '--boundary\r\nContent-Disposition: form-data; name="file"; filename="a"\r\n\r\npartial' + ); + req.emit('aborted'); + req.destroy(); + await rejection; + expect(req.listenerCount('aborted')).toBe(0); + expect(req.listenerCount('data')).toBe(0); + expect(req.listenerCount('close')).toBe(0); +}); + +it('settles request and parser errors once without retaining listeners', async () => { + const req = request(); + const parsing = parseMultipart(req, {}, 4096); + const rejection = expect(parsing).rejects.toMatchObject({ status: 400 }); + req.emit('error', new Error('connection failed')); + await rejection; + expect(req.listenerCount('data')).toBe(0); + expect(req.listenerCount('aborted')).toBe(0); +}); diff --git a/libs/server/src/multipart.ts b/libs/server/src/multipart.ts new file mode 100644 index 00000000..d52a70e6 --- /dev/null +++ b/libs/server/src/multipart.ts @@ -0,0 +1,277 @@ +import type { IncomingMessage } from 'node:http'; +import { Busboy, type BusboyFileStream } from '@fastify/busboy'; +import { HttpError } from './HttpError.js'; +import type { FilePart, RejectedFile } from './types.js'; +import type { UploadConfiguration } from './upload.js'; + +function failure(status: number, detail: string, field?: string): HttpError { + const pointer = + field === undefined + ? '/files' + : `/files/${field.replace(/~/g, '~0').replace(/\//g, '~1')}`; + return new HttpError( + status, + status === 413 ? 'Payload Too Large' : 'Bad Request', + detail, + { + errors: [{ pointer, detail }] + } + ); +} + +/** @internal Buffered multipart parsing with a bound on the entire wire body. */ +export async function parseMultipart( + req: IncomingMessage, + options: UploadConfiguration, + maxBodySize: number +): Promise<{ + fields: Record; + files: Record; + rejectedFiles: RejectedFile[]; +}> { + const maxFileCount = options.maxFileCount ?? 10; + const maxFieldCount = options.maxFieldCount ?? 100; + const properties = options.schema?.introspect().properties; + const fields: Record = Object.create(null); + const collected: Record = Object.create(null); + const rejectedFiles: RejectedFile[] = []; + const seenFiles = new Set(); + const contentLength = Number(req.headers['content-length']); + if (contentLength > maxBodySize) { + req.resume(); + throw failure(413, 'Multipart request exceeds maxBodySize'); + } + await new Promise((resolve, reject) => { + let finished = false; + let bytes = 0; + const active = new Map(); + const parser = Busboy({ + headers: req.headers as { 'content-type': string }, + limits: { + fileSize: options.maxFileSize ?? 10 * 1024 * 1024, + files: maxFileCount, + fieldSize: options.maxFieldSize ?? 1024 * 1024, + fields: maxFieldCount, + parts: options.maxPartCount ?? maxFileCount + maxFieldCount + } + }); + const cleanup = () => { + req.unpipe(parser); + req.off('data', countBytes); + req.off('aborted', aborted); + req.off('error', onError); + req.off('close', closed); + for (const [stream, chunks] of active) { + chunks.length = 0; + stream.destroy(); + } + active.clear(); + }; + const fail = (error: unknown) => { + if (finished) return; + finished = true; + cleanup(); + for (const key of Object.keys(collected)) delete collected[key]; + parser.destroy(); + req.resume(); + reject( + error instanceof HttpError + ? error + : failure(400, 'Malformed multipart request') + ); + }; + const countBytes = (chunk: Buffer) => { + bytes += chunk.length; + if (bytes > maxBodySize) + fail(failure(413, 'Multipart request exceeds maxBodySize')); + }; + const aborted = () => + fail(failure(400, 'Multipart request interrupted')); + const closed = () => { + if (!req.complete) aborted(); + }; + const onError = (error: Error) => fail(error); + const checkName = (name: string) => { + if (Buffer.byteLength(name) > (options.maxFieldNameSize ?? 100)) { + fail(failure(413, 'Multipart field name is too long', name)); + return false; + } + return true; + }; + parser.on('field', (name, value, nameTruncated, valueTruncated) => { + if (finished || !checkName(name)) return; + if (nameTruncated || valueTruncated) { + fail( + failure(413, 'Multipart field exceeds its size limit', name) + ); + return; + } + if ( + Object.hasOwn(fields, name) || + seenFiles.has(name) || + (properties && Object.hasOwn(properties, name)) + ) { + fail( + failure( + 400, + 'Duplicate or incorrectly encoded multipart field', + name + ) + ); + return; + } + fields[name] = value; + }); + parser.on('file', (name, stream, filename, _encoding, mimeType) => { + const chunks: Buffer[] = []; + active.set(stream, chunks); + stream.on('error', onError); + stream.on('limit', () => + fail(failure(413, 'File exceeds maxFileSize', name)) + ); + if (finished || !checkName(name)) { + stream.resume(); + return; + } + const property = + properties && Object.hasOwn(properties, name) + ? properties[name] + : undefined; + const multiple = property?.introspect().type === 'array'; + if ( + (properties && !property) || + Object.hasOwn(fields, name) || + (seenFiles.has(name) && !multiple) + ) { + fail(failure(400, 'Unknown or duplicate file field', name)); + return; + } + seenFiles.add(name); + const allowed = + !options.allowedMimeTypes || + options.allowedMimeTypes.some(pattern => + pattern.endsWith('/*') + ? mimeType.startsWith(pattern.slice(0, -1)) + : mimeType === pattern + ); + if (!allowed) { + if (properties) { + fail( + failure( + 400, + `MIME type "${mimeType}" is not allowed`, + name + ) + ); + return; + } + rejectedFiles.push({ + fieldName: name, + filename, + mimeType, + reason: `MIME type "${mimeType}" is not allowed` + }); + stream.on('end', () => active.delete(stream)); + stream.resume(); + return; + } + // Reserve the position at part arrival, not when its stream ends. + const list = collected[name] ?? []; + collected[name] = list; + const index = list.length; + list.push(undefined as unknown as FilePart); + stream.on('data', (chunk: Buffer) => { + if (!finished) chunks.push(chunk); + }); + stream.on('end', () => { + active.delete(stream); + if (finished) return; + if (stream.truncated) { + fail(failure(413, 'File was truncated', name)); + return; + } + const buffer = Buffer.concat(chunks); + chunks.length = 0; + list[index] = { + filename, + mimeType, + buffer, + size: buffer.length + }; + }); + }); + parser.on('filesLimit', () => + fail(failure(413, 'Exceeded maxFileCount')) + ); + parser.on('fieldsLimit', () => + fail(failure(413, 'Exceeded maxFieldCount')) + ); + parser.on('partsLimit', () => + fail(failure(413, 'Exceeded maxPartCount')) + ); + parser.on('error', onError); + parser.on('finish', () => { + if (finished) return; + finished = true; + cleanup(); + resolve(); + }); + req.on('data', countBytes); + req.once('aborted', aborted); + req.once('error', onError); + req.once('close', closed); + req.pipe(parser); + }); + const files: Record = Object.create(null); + for (const [name, values] of Object.entries(collected)) { + files[name] = + properties?.[name]?.introspect().type === 'array' + ? values + : values[0]; + } + if (options.schema) { + for (const [name, property] of Object.entries(properties!)) { + const info = (property as any).introspect(); + if ( + info.type === 'array' && + info.isRequired && + !Object.hasOwn(files, name) + ) + files[name] = []; + } + const result = await options.schema.validateAsync(files, { + doNotStopOnFirstError: true + }); + if (!result.valid) { + const errors = result.getInvalidProperties().flatMap(prop => + prop.errors.map(detail => ({ + pointer: `/files${prop.descriptor.toJsonPointer()}`, + detail + })) + ); + throw new HttpError( + 400, + 'Bad Request', + 'Upload contract validation failed', + { + errors: errors.length + ? errors + : result.errors?.map(error => ({ + pointer: '/files', + detail: error.message + })) + } + ); + } + return { + fields, + files: Object.fromEntries( + Object.entries(result.object!).filter( + ([, value]) => value !== undefined + ) + ), + rejectedFiles + }; + } + return { fields, files, rejectedFiles }; +} diff --git a/libs/server/src/types.ts b/libs/server/src/types.ts index e97acf13..7b8b150c 100644 --- a/libs/server/src/types.ts +++ b/libs/server/src/types.ts @@ -140,6 +140,8 @@ export interface FilePart { * Describes a file that was rejected during multipart parsing. */ export interface RejectedFile { + /** Multipart field name, when available. */ + readonly fieldName?: string; /** Original filename as provided by the client. */ readonly filename: string; /** MIME type of the file (e.g. `'application/xlsx'`). */ @@ -153,6 +155,14 @@ export interface RejectedFile { * `EndpointBuilder.upload()`. */ export interface UploadOptions { + /** Maximum bytes per text field. Default: 1 MiB. */ + maxFieldSize?: number; + /** Maximum text fields per request. Default: 100. */ + maxFieldCount?: number; + /** Maximum UTF-8 bytes per field name. Default: 100. */ + maxFieldNameSize?: number; + /** Maximum parts. Default: maxFileCount + maxFieldCount. */ + maxPartCount?: number; /** * Maximum allowed file size per uploaded file in bytes. * @default 10_485_760 (10 MB) diff --git a/libs/server/src/upload.test.ts b/libs/server/src/upload.test.ts new file mode 100644 index 00000000..94424f56 --- /dev/null +++ b/libs/server/src/upload.test.ts @@ -0,0 +1,51 @@ +import { array, object, string } from '@cleverbrush/schema'; +import { expect, it } from 'vitest'; +import { endpoint } from './Endpoint.js'; +import { file } from './upload.js'; + +it('keeps upload schemas and limits through immutable chaining', () => { + const files = object({ + images: array(file()).minLength(1), + cover: file().optional() + }); + const base = endpoint.post('/files'); + const upload = base + .upload(files, { maxFieldSize: 20 }) + .body(object({ title: string() })) + .summary('Upload'); + expect(base.introspect().fileUpload).toBeNull(); + expect(upload.introspect().fileUpload).toMatchObject({ + schema: files, + maxFieldSize: 20 + }); + expect(file().optional().introspect().extensions?.uploadFile).toBe(true); +}); + +it('rejects ambiguous or unsupported multipart contracts in either chaining order', () => { + expect(() => + endpoint.post('/').upload(object({ image: string() })) + ).toThrow(/file/); + expect(() => + endpoint.post('/').upload(object({ image: file().nullable() })) + ).toThrow(/file/); + expect(() => + endpoint + .post('/') + .upload(object({ image: file() }).acceptUnknownProps()) + ).toThrow(/closed/); + expect(() => + endpoint + .post('/') + .body(object({ image: string() })) + .upload(object({ image: file() })) + ).toThrow(/overlap/); + expect(() => + endpoint + .post('/') + .upload(object({ image: file() })) + .body(object({ image: string() })) + ).toThrow(/overlap/); + expect(() => endpoint.post('/').upload({ maxFileCount: 0 })).toThrow( + /positive/ + ); +}); diff --git a/libs/server/src/upload.ts b/libs/server/src/upload.ts new file mode 100644 index 00000000..96e303cf --- /dev/null +++ b/libs/server/src/upload.ts @@ -0,0 +1,100 @@ +import { + any, + type InferType, + type ObjectSchemaBuilder, + type SchemaBuilder +} from '@cleverbrush/schema'; +import type { FilePart, UploadOptions } from './types.js'; + +/** A buffered file in an upload contract. Safe to import in browser contracts. */ +export function file() { + return any() + .hasType() + .addValidator(value => { + const valid = + value !== null && + typeof value === 'object' && + typeof value.filename === 'string' && + typeof value.mimeType === 'string' && + value.buffer instanceof Uint8Array && + value.size === value.buffer.byteLength; + return { + valid, + errors: valid ? [] : [{ message: 'Expected an uploaded file' }] + }; + }) + .withExtension('uploadFile', true); +} + +/** Flat object of file() and array(file()) fields. */ +export type UploadSchema = ObjectSchemaBuilder< + any, + any, + any, + any, + any, + any, + any +>; +/** Upload metadata shared by the server, typed client, and OpenAPI generator. */ +export interface UploadConfiguration extends UploadOptions { + readonly schema?: UploadSchema; +} +/** The upload state carried through immutable endpoint chaining. */ +export type UploadContract = boolean | UploadSchema; +/** Files received by a handler, inferred from the upload schema. */ +export type UploadFiles = T extends UploadSchema + ? InferType + : Record; + +/** @internal Validate the supported multipart shape in either chaining order. */ +export function validateUploadConfiguration( + upload: UploadConfiguration | null, + body: SchemaBuilder | null +): void { + if (!upload) return; + for (const key of [ + 'maxFileSize', + 'maxFileCount', + 'maxFieldSize', + 'maxFieldCount', + 'maxFieldNameSize', + 'maxPartCount' + ] as const) { + const value = upload[key]; + if (value !== undefined && (!Number.isSafeInteger(value) || value < 1)) + throw new TypeError(`${key} must be a positive safe integer`); + } + if (!upload.schema) return; + const root = upload.schema.introspect(); + if ( + root.type !== 'object' || + root.isNullable || + !root.isRequired || + root.acceptUnknownProps + ) + throw new TypeError('Upload schemas must be required, closed objects'); + const bodyInfo = body?.introspect() as any; + if (bodyInfo && bodyInfo.type !== 'object') + throw new TypeError('Multipart body schemas must be objects'); + for (const [name, property] of Object.entries(root.properties)) { + const info = (property as SchemaBuilder).introspect() as any; + const leaf = + info.type === 'array' ? info.elementSchema?.introspect() : info; + if ( + !leaf?.extensions?.uploadFile || + info.isNullable || + leaf.isNullable || + info.hasDefault || + leaf.hasDefault || + (info.type === 'array' && !leaf.isRequired) + ) + throw new TypeError( + `Upload field "${name}" must be file() or array(file())` + ); + if (Object.hasOwn(bodyInfo?.properties ?? {}, name)) + throw new TypeError( + `Multipart text and file fields overlap: ${name}` + ); + } +} diff --git a/websites/docs/app/knex-schema/page.tsx b/websites/docs/app/knex-schema/page.tsx index 715456f0..079fd765 100644 --- a/websites/docs/app/knex-schema/page.tsx +++ b/websites/docs/app/knex-schema/page.tsx @@ -582,6 +582,23 @@ const read = base.where(t => t.projectId, projectId) +
+

JSONB documents

+

+ Use object({'{ ... }'}).jsonb() for a + document column. Add .acceptUnknownProps(){' '} + on each object that must retain undeclared JSON fields. + Reads, projections and write-returning results preserve + those fields alongside typed, declared properties. +

+

+ Invalid JSON extension values are rejected before + persistence. Tracked ORM document columns detect nested + edits and compare documents structurally; object key + order is not a storage guarantee. Optional and nullable + object columns accept SQL null. +

+
); diff --git a/websites/docs/app/server/page.tsx b/websites/docs/app/server/page.tsx index 77771ae4..6b085820 100644 --- a/websites/docs/app/server/page.tsx +++ b/websites/docs/app/server/page.tsx @@ -255,23 +255,25 @@ return ActionResult.status(202);`)

File Upload

Accept file uploads via multipart/form-data{' '} - by chaining .upload() on an endpoint. File - fields are received as FilePart objects on - the handler context's files property; + with .upload(object(...)). Declare single + files with file() and repeated files with + array(file()). The handler receives + matching + FilePart or FilePart[] fields; non-file form fields are validated against the body schema and available via body.

                          {
-    const avatar: FilePart = files['avatar'];
+    const avatar = files.avatar;
     // avatar.filename, avatar.mimeType, avatar.buffer, avatar.size
     return ActionResult.created({ name: avatar.filename });
 });`)
@@ -339,6 +341,25 @@ server.handle(UploadAvatar, ({ body, files }) => {
                         
                     
 
+                    

+ Omit .body() for file-only requests. Typed + uploads reject invalid fields before calling the + handler. Limits return 413 Problem Details; truncated + content is never delivered as a successful upload. + Required arrays default to empty, so use{' '} + .minLength(1) to require at least one file. +

+

+ The whole multipart body also respects the server's + maxBodySize (5 MiB by default). Text fields + default to 1 MiB each and 100 fields; names are limited + to 100 UTF-8 bytes. Configure these with + maxFieldSize, maxFieldCount, + maxFieldNameSize and{' '} + maxPartCount. Files are buffered in memory + within these bounds. +

+

FilePart type