feat(scheduler)!: durable jobs, progress and PostgreSQL persistence

[andrewzolotukhin]

Original request

Implement the agreed application-agnostic “Durable background jobs and progress” design in Framework, based on the existing scheduler, as a PR against development. Keep contracts and handlers separate and strongly typed; include tests, JSDoc, guides, a migration guide and a changeset.

Approved choices: a unified v5 scheduler redesign; immediate and recurring jobs; function and worker-thread handlers; opt-in whole-handler retries; UTC/IANA calendar schedules; coalesce/skip/replay missed policies; PostgreSQL persistence plus a process-local in-memory adapter; reuse Framework ORM and knex-schema.

What changed

Dependency review follow-up in 86afb348:

• Remove @js-temporal/polyfill and its unused jsbi dependency from the scheduler manifest and npm-generated lockfile.
• Use native Date/Intl.DateTimeFormat for Gregorian calendar arithmetic and IANA offsets, with a bounded formatter cache. Keep gap skipping, earlier-fold selection and persisted recurrence semantics.
• Add 26 regression tests covering half-hour DST, date-line skips, quarter-hour/historical offsets, years 0–99/eras, Date-range exhaustion and host-TZ independence. Update JSDoc, README runtime/tzdata guidance and the existing major changeset.
• Before removal, compare the native resolver against the previous implementation in a one-off check: 5,852 cases across 418 IANA zones, zero mismatches. The comparison dependency is not retained in tests or runtime code.

Review follow-up in 8665e96f:

• Retain all five schema-driven schedule definitions, exported ScheduleSchema, individual schemas and Schemas; infer Schedule/TaskSchedule as a discriminated union.
• Share schema validation across registration and calculation; accept JSON dates and the deprecated maxOccurences spelling, reject both spellings together. Restore one-based public calculator indexes while retaining zero-based persisted cursors.
• Normalize defaults, dates and weekday order before fingerprinting; unchanged registration keeps the original anchor, revision and cursor, including after the end bound passes.
• Rename the explicit persistence option to storageRepository throughout implementation, tests and documentation.
• Remove RowSchema<T>, parallel handwritten row types and double casts from the PostgreSQL adapter. Derive storage/entity/query types from actual schemas, with declaration-emission and metadata/type coverage.
• Add schedule compatibility/validation/type tests, all-five-frequency worker/dispatcher tests, PostgreSQL recurring execution/replacement coverage, and a runnable periodic demo (also exercised in CI).
• Expand README/JSDoc/website, correct the migration guide and retain the major changeset. Schedule definitions are preserved; the old file-registration runtime is not restored.

Original implementation:

• Versioned defineJob contracts, JobHandler<typeof Definition>, separate trusted handler registration and independently managed producer/dispatcher/worker lifecycles.
• Durable queued/running/retry-wait/terminal states, attempt history, database-time leases, stale-owner fencing, cancellation, bounded shutdown and opt-in exponential retries.
• Persisted, ordered progress with exclusive reconnect cursors, typed results, strict JSON/schema boundaries, payload/progress limits, retention cleanup and operational diagnostics.
• Calendar recurrence with persisted cursors, atomic occurrence materialization, future-only schedule revisions, missed/overlap policies and deterministic DST behavior.
• New @cleverbrush/scheduler-postgres package. Framework schemas generate tables, ORM handles routine queries, and isolated native queries implement locking and aggregates. Explicit up/down migrations; transaction-bound producers support business-write + enqueue rollback.
• Shared adapter contract tests; real PostgreSQL concurrency, rollback, lease and SIGKILL/restart tests; packaged worker-thread tests; compile-time consumer tests; separated-files executable demo.
• Updated README/JSDoc/website, v4.x-to-v5 migration guide, fixed release-group membership, major changeset, Docker workspace manifests and CI database integration coverage.

Reasoning and guarantees

Native time-zone rules come from the Node.js runtime's ICU data. Dispatcher runtimes should use aligned tzdata versions; no additional date-time package is required.

One shared transition engine prevents adapters from drifting on retry/ownership semantics. PostgreSQL is optional, rather than a mandatory dependency of scheduler consumers.

This is not an exactly-once workflow engine. Enabled retries rerun the entire handler. Applications own side-effect idempotency, authorization and progress transport. Default maxAttempts is one; lease expiry consumes the same attempt budget. Ordinary functions must cooperate with cancellation; threads can be terminated.

The major changeset joins the existing Framework v5 release train. Changeset status confirms the fixed public package group receives major releases. No package has been published, and no production deployment is part of this PR.

Blog post

Skipped: library/runtime API work, not an end-user application feature. The package guides, API JSDoc, migration guide and documentation website are the consumer-facing documentation.

Screenshots / preview evidence

Framework has no hosted PR preview workflow or configured runtime telemetry service. Local docs QA at http://127.0.0.1:3219/scheduler verified all guide sections and links, with no browser errors. This is a local-only preview, not a public deployment.

The immediate example prints:

1 queued null
2 running null
3 progress { percent: 50 }
4 succeeded null
succeeded { downloadUrl: '/reports/quarterly' }

The periodic example reports Completed both periodic reports. after running both the dispatcher and worker.

Validation

• npm run lint
• npm run build — 22 package builds
• npm run test — 4,403 tests, 216 files, no type errors
• Focused scheduler/adapter suite — 132 tests, including packaged threads
• npm run typecheck:schema-site
• npm run typecheck:docs-site
• npm run test:scheduler:integration — 12 PostgreSQL tests
• npm run test:queries:integration — 53 existing PostgreSQL tests
• API reference generation for both scheduler packages; scheduler reference regenerated for this follow-up.
• node demos/durable-jobs/demo.ts
• node demos/durable-jobs/periodic.ts --fast — two periodic runs completed
• npm pack --dry-run — worker-thread entry included
• npx changeset status --since=origin/development — major release group validated (existing private website file-dependency warnings remain)
• Local documentation browser QA and screenshot from the schedule API update; the dependency-only follow-up does not change the website UI.
• GitHub CI for 86afb348 — both required jobs passed:
  • Lint, Build & Test (Node 24): success
  • PostgreSQL Query Integration: success (including both demos)
• Hosted preview and SigNoz verification: not applicable; this repository has no configured PR deployment or corresponding runtime service.

Notification

Telegram PR-ready notification skipped: the notifier requires an environment URL, but Framework has no hosted PR preview deployment. Local docs QA and CI evidence are linked above.

[andrewzolotukhin]

I think repository is not the best name here, let's come up with something better. What about persistRepository or storageRepository?

[andrewzolotukhin]

Addressed in 8665e96: the constructor option is now storageRepository throughout the implementation, tests, both package guides and the website examples. No alias for the unpublished repository option remains. Verified by the full build, 4,377 tests, PostgreSQL integration tests and both runnable demos. Leaving this thread open for your review.

[andrewzolotukhin]

this looks weird, do we really need this type? Or we can just use typeof?

[andrewzolotukhin]

Yes—the handwritten RowSchema<T> layer was unnecessary. Addressed in 8665e96: removed it, the four parallel row types and all as unknown as casts from this file. Storage and entity types now use typeof the actual schemas; tests derive row types with InferType. Named property maps plus ReturnType<typeof object<typeof fields>> keep declaration emission portable without duplicating field types. Added compile-time query/insert/nullability checks and runtime primary-key/table-prefix checks; package declaration emit and the real PostgreSQL suites pass. Leaving this thread open for your review.

[andrewzolotukhin]

The earlier revision retained recurrence through upsertSchedule, but dropped the reusable schema-based schedule definitions while replacing the worker runtime. Those definitions should remain supported. In 8665e96 I restored the minute/day/week/month/year schemas, the Schemas facade and inferred Schedule type. Existing schedule objects (including maxOccurences) can be passed to upsertSchedule; the public calculator again uses one-based indexes. The worker/runtime registration remains the new defineJob + handle/thread model, as agreed. The README, migration guide and website now show periodic registration plus both dispatcher and worker lifecycles, and node demos/durable-jobs/periodic.ts --fast proves two periodic runs complete.

Validation is complete: 4,377 local tests, both website typechecks, both API reference builds, 12 PostgreSQL scheduler tests, 53 ORM query tests, both runnable demos and local docs browser QA passed. Both required CI jobs are green. Replies are on both inline threads; they are left unresolved for your review. No hosted preview/telemetry is configured; the PR description includes a local docs screenshot and explains the notification skip.

[andrewzolotukhin]

let's avoid using it

[andrewzolotukhin]

Addressed in 86afb34. Removed @js-temporal/polyfill and the now-unused jsbi dependency from the manifest and npm-generated lockfile. Calendar arithmetic and time-zone resolution now use built-in Date/Intl.DateTimeFormat, with a bounded formatter cache; no replacement date-time dependency was added.

The documented behavior is retained: gaps are skipped, repeated times select the earlier instant, and calculations are independent of the host TZ. Added 26 regressions covering half-hour DST changes, skipped dates, fractional/historical offsets, calendar eras and Date-range exhaustion. A one-off comparison against the previous implementation covered 5,852 cases across 418 IANA zones with zero mismatches; that dependency is not retained in tests.

Updated JSDoc, README (including runtime ICU/tzdata guidance) and the major changeset. Validation: 4,403 tests, both website typechecks, PostgreSQL suites, both demos and both required CI jobs pass. Leaving the thread open for your review.