[pournasserian]

Closes #2698

Adds durable queued, scheduled and recurring jobs on one database, with no other infrastructure and safe across instances. cms_job, cms_job_attempt and cms_recurring_job come with a migration per provider. The SDK gets IJob<TPayload>, JobContext, IJobScheduler, IJobRunner and IJobDeadLetters. FluentCMS.Core/Jobs has the hosted runner: workers per queue, claims through IJobClaimDialect, lease heartbeat, retries with backoff and jitter, dead letter with a sweep for a worker that died on its last attempt, graceful shutdown, and recurring jobs through Cronos.

This also fixes two bugs in the SQLite claim from #2737:

• Double run. Its second step rechecked only the lease, and finishing a job clears the lease, so a worker that had read a job just before another finished it could claim it again. The two-runner test ran 60 jobs 77 to 82 times before the fix. The update now rechecks every condition of the read. The PostgreSQL and SQL Server dialects are one atomic statement and do not have this gap.
• Stuck queue. A row whose id is not the uppercase text EF stores was always selected and never matched, in an unbounded loop. The claim now reads the id as text, matches that text, and writes the canonical form. A row whose id is not a GUID is skipped, and the loop is bounded.

Dependencies: Cronos 0.13.0 (new, MIT, approved in principle in Q9), referenced by FluentCMS.Core for cron parsing. No new test package: tests/FluentCMS.Data.Tests/Jobs/ManualTimeProvider.cs is written here rather than adding Microsoft.Extensions.TimeProvider.Testing.

Acceptance

Under the scope change on #2711 (Q19), items that name PostgreSQL and SQL Server apply to SQLite until #2711.

• Two runner instances against one database never execute the same job twice. SQLite only, with both journal modes, and with a variant where a lease lapses mid-job; PostgreSQL and SQL Server have not run. See "Delivery" under "Limits": this holds while a worker's clock runs and its job honours its token.
• A crashed worker's lease expires and the job is retried.
• Dead-lettered jobs are listed and can be retried through the service API, IJobDeadLetters.
• Recurring jobs fire on schedule with TimeProvider controlling time in tests.

Verification

On Windows (arm64), at this commit: dotnet build -c Release -warnaserror, dotnet test -c Release (1893 tests, all passed), dotnet pack -c Release (18 packages and their symbol packages). CI covers Linux.

Before the last commit, the job tests (85) passed 6 rounds of 4 copies at once, which is 24 runs under load, and 5 more one at a time. JobClaimTests passed the 5 single runs and was not in the 4-copy rounds.

Failure injection: JobFaults fails one instance's commands while others reach the database, so the paths through HeartbeatFailed, SettleFailed, WorkerFailed, MaintenanceFailed and RecurringFireFailed now have tests. To check that the tests need the fixes, I undid four of them in turn. Each time the tests meant to catch it failed:

Undone	Failing tests
Transaction around the claim	AFailedClaimDoesNotUseUpAnAttempt (both cases), which proves the attempt was burned
Attempts == job.Attempt in Held	ALeaseIsFencedByTheAttemptAsWellAsTheOwner
Heartbeat stopping the attempt	WhenALeaseLapsesMidJobTheTwoRunnersNeverOverlap; that run hit my time limit before finishing, so others may fail too
Retrying the settle	ABriefOutageDoesNotLoseASuccess (both cases), ABriefOutageDoesNotLoseAFailure

TheClaimSeeksItsIndexWithoutASort runs EXPLAIN QUERY PLAN on the statement the SQLite dialect really sends.

Not verified: anything on a PostgreSQL or SQL Server server. Those migrations and dialects are checked offline only (no pending model changes, column types, index shape).

Limits

• Delivery is at least once. A worker that cannot renew its lease cancels its attempt before the lease ends, so a job that honours JobContext.CancellationToken stops before another can claim it. Two things can still overlap: a job that ignores its token, and a process paused long enough that its heartbeat does not run at all. Whatever such a worker writes on waking matches no row. A worker that dies mid-job also leaves work to be repeated. Jobs should be safe to run again.
• Leases use each instance's clock. Skew between instances above lease minus heartbeat (4 minutes by default) can take a job from a healthy worker. Set against Validate and complete the PostgreSQL and SQL Server providers #2711: the database's clock could be the reference there.
• A hung job holds its worker forever, because the heartbeat keeps renewing it. There is no job timeout.
• Scrubbing is best effort. It cannot catch a secret the job encoded or derived, and it over-redacts: a property name or short value in a message is replaced wherever it stands as a word, so an error can read POST failed with *** ***.

What later issues must do

• Add the content service with optimistic concurrency #2747 (content service): write author_id, created_by and updated_by from context.AuditPrincipal.Id, not context.Principal.Id. Otherwise a scheduled publish records Guid.Empty as its author. Authorization keeps reading Principal. A job runs with Principal.System and Origin.Job, which SystemOperation.Allows accepts; AJobRunsAsTheSystemPrincipalWithOriginJob asserts both.
• In-process event bus with after-commit dispatch and outbox #2697 (event bus): replace NullJobEventSink by registering an IJobEventSink, and move JobFailedEvent and JobDeadLetteredEvent to the SDK if handlers there need them. They are internal for now. A sink that throws is logged and does not affect the job.
• Scheduled publish and unpublish per locale #2699, Trash with retention and purge job #2696, Activity log: writer, event subscribers and retention #2702 (built-in jobs): register with cms.Jobs.AddJob<TJob, TPayload>("stable-name"); that name is what cms_job stores, so it must not change. Add schedules with IJobScheduler.AddRecurringAsync, which is safe at every startup on every instance. Nothing calls it for you yet, so these issues need a startup step. Make jobs safe to run again. A payload that should record who asked implements IJobActorPayload.
• Anything that enqueues inside a transaction: IJobScheduler writes on its own context, so a job can start before the caller's transaction commits, and survives a rollback. If that matters, enqueue after the commit.
• A new issue, job timeouts: a hung job holds its worker forever. Add a per-job or per-queue timeout that cancels the attempt.
• M2 /jobs endpoints: IJobDeadLetters is the service to call, and jobs.manage is theirs to enforce; the service checks no permission. There is no API yet to list live jobs or to change or remove a schedule, and nothing sets cms_recurring_job.enabled to false.
• Retention: succeeded jobs and their attempts are kept forever. The claim's index covers only live jobs, so claims do not slow down as they grow, but the tables do. A retention job needs to remove them.
• Plugin host (M3): JobsBuilder is in core, and plugins reference only the SDK, so plugin jobs need a registration path.
• Configuration: JobRunnerOptions is set in code through cms.Jobs.Options. It is not bound from appsettings.
• Validate and complete the PostgreSQL and SQL Server providers #2711:
  • Run the claims, JobStore and migrations on real servers, and move the two-runner test onto the provider matrix.
  • Check that SQL Server's plan for the claim seeks ix_cms_job_claim under UPDLOCK, READPAST without a sort, so that concurrent claimers do not lock every due row. It depends on the filtered index matching the literal status filter.
  • ISJSON refuses scalar payloads on SQL Server. With no type constraint, ISJSON accepts only an object or array, per its documentation. So the check on cms_job.payload fails the enqueue of an IJob<int> or IJob<string> there, while SQLite and PostgreSQL accept it. ISJSON(x, VALUE) would allow scalars but needs SQL Server 2022, and the plan covers 2019. Validate and complete the PostgreSQL and SQL Server providers #2711 decides between a wrapper object and a rule that payloads are objects.
  • Decide whether clock skew needs the database's clock.

Interpretations beyond the specification

Reported as a mismatch with the spec:

• An actor does not replace the principal. Spec 4.12 says jobs run as System "unless the payload carries an actor". Taken literally, the job would run as that user, and either be denied everything (a principal with no claims) or borrow that user's real roles once IPrincipalRoles reads them from the database, chosen by whatever wrote the payload. So the principal stays Principal.System, and the actor goes to a new OperationContext.Actor, which authorization never reads. OperationContext.AuditPrincipal is Actor ?? Principal. The actor is built from kind, id and name only, so a payload cannot bring claims. System, Anonymous or an empty id fails the attempt with FCMS2507. This adds two members to OperationContext. The spec may want rewording.
• cms_recurring_job has a type column that the spec's column list lacks. A schedule has to say which job it fires, and name alone would allow only one schedule per job type.
• AddRecurring(name, cron) is AddRecurringAsync<TJob, TPayload>(name, cron, payload, queue), because it writes to the database and needs the job and its input.

Choices the spec leaves open:

• IJobDeadLetters is new in the SDK, because the acceptance list asks for a service API and the spec names none.
• Job registration: cms.Jobs.AddJob with an explicit stable name. Enqueue refuses an unregistered job (FCMS2504).
• Attempt numbers only go up, because cms_job_attempt is keyed on them. A retry from dead letter sets max_attempts to the attempts used plus JobRunnerOptions.DefaultMaxAttempts, because the table does not keep a job's original budget. A release at shutdown raises it by one, so a shutdown does not use up an attempt.
• The claim is one transaction with its read and attempt row, so a failure there does not use up an attempt. It costs the write lock on SQLite, so a plain read first keeps an idle poll from taking it.
• Writing an outcome retries with backoff from 1s to 30s, for up to one lease. A success, failure or release that cannot be written would otherwise rerun the job or dead-letter it as unfinished. Going past the real end of the lease is safe, because the write then matches nothing if another worker has the job.
• A worker stops one heartbeat before its lease would lapse, rather than when it has lapsed, so that it stops before another worker can claim the job. That gives up the last renewal attempt.
• Fencing is on owner and attempt. A worker's owner never comes back within a process today, but the attempt keeps a stale write out whatever the owner names; ALeaseIsFencedByTheAttemptAsWellAsTheOwner needs that clause.
• Status values: pending, running, succeeded, dead_lettered. Attempt outcomes: succeeded, failed, lease_expired, released. A crashed attempt's row stays open until the next claim or sweep closes it, which also raises JobFailedEvent.
• The sweep also takes a pending job with no attempts left, which the runner never writes but a claim would never take.
• An unknown type or bad payload retries like any failure rather than dead-lettering at once, because a plugin can come back after a deployment.
• last_error and cms_job_attempt.error hold exception types and messages, no stack trace, capped at 2000 characters. Every string, number and property name of the payload is replaced with *** in the messages, in logs too. A value of 4 characters or more is replaced anywhere; a shorter one only as a whole word, because pin 123 has to go but 12 must not eat 123. A payload that fails to deserialize records only the target type and the exception's type.
• Cron: five fields, UTC, since the table has no time zone. A row edited to a bad cron is logged and skipped, and a row that fails to fire does not stop the others. Runs missed during downtime collapse into one. Nothing stops a run overlapping the previous one. A fired job takes the default priority, attempts and no site, because the table has no columns for them.
• Re-adding a schedule with the same cron keeps next_run_at, so a restart does not skip an occurrence. A changed cron starts from now.
• ix_cms_job_claim is (queue, priority DESC, run_at, id) filtered to live jobs, which needs the claim's status filter as literals on all three dialects. Without the filter, the index would walk finished rows first, since they have the earliest run_at.
• Defaults: 2 workers on default, 5 attempts, 5s poll, 5m lease, 1m heartbeat, 15s maintenance, 30s to 1h backoff with up to 20% jitter, 20s shutdown grace plus 5s to write releases. Every host using AddFluentCms now runs the runner; a queue with 0 workers takes nothing.
• A job with no site runs in the default site, and any job takes its site's default locale. site_id has no foreign key, like cms_document.
• Tables are written through SystemDbContext with ExecuteUpdate, which keeps ids as EF stores them and avoids the UPDATE ... RETURNING that fails under load on SQLite.
• Error codes FCMS2501 to FCMS2507 are a new group for jobs.