Add trash listing, purge and the retention purge job

docs/planning/backend-technical-spec.md

@@ -465,6 +465,7 @@ Rows 2.3 to 2.8, 2.14, 2.15, 2.19, 1.7, 1.8.
 | `PatchAsync(id, patch, expectedVersion)` | partial update of the draft with a JSON merge patch; a null removes a field. Its own method, because a class cannot express absence |
 | `DeleteAsync(id)` | trash, in every locale |
 | `RestoreAsync(id)` | from trash, in every locale |
+| `QueryTrashAsync(ContentQuery)`, `PurgeAsync(id)` | list the trash; delete an entry in it for good (4.5) |
 | `PublishAsync(id, locale?, expectedVersion?)`, `UnpublishAsync(id, locale?)`, `DiscardDraftAsync(id, locale?, expectedVersion?)` | publishing transitions (4.5); publish returns the entry and `warnings` |
 | `ScheduleAsync(id, ScheduleRequest)`, `CancelScheduleAsync` | per locale |
 | `GetVersionsAsync(id, locale?)`, `GetVersionAsync(id, number, locale?)`, `RestoreVersionAsync(id, number, locale?, expectedVersion?)`, `DiffAsync(id, a, b, locale?)` | versions |
@@ -556,6 +557,10 @@ Rows 6.1 to 6.11; decisions 38, 39, 40, 41.
 
 **Versions.** Every save writes an immutable snapshot (decision 38) with author, origin, client id and an optional message. A per-type cap (default 50, `[PublishPolicy(MaxVersions = n)]`) prunes oldest drafts when a version is written; versions that were published are never pruned, and only a publish marks one so, in `status_after`. Diff is computed on demand as a field-level structural diff (blocks and rich text compared as trees) and returned as a list of `{ path, kind, before, after }`. A snapshot holds hidden fields, so reads and diffs go through the reader's field limits as they are today, and need the Management scope as well as the draft read. Access to versions is decided on the entry's current row: history is part of the entry, so a reader the row rule lets in also sees versions from before the entry came within the rule. A restore is a write of the principal's own: what it may not write or read, and what is hidden or read-only, keeps the value the draft holds, except for the system principal. A principal that can write a field it cannot read may not discard or restore at all, whatever that field holds, because the status and version those answer with would check a saved guess against the value.
 
+**Trash.** `DeleteAsync` sets `deleted_at` and `deleted_by` on every locale of the entry, and the global query filter on `cms_document` hides it from reads, queries, versions, publishing and reference checks; the content rows, references and versions stay as they were, so a restore brings back the entry as it was. `QueryTrashAsync` reads the trash as drafts through the query engine, in one locale, so the read grant, its row rule, its locale and field limits and the site hold; it needs `restore` or `purge` on the type and lists only what the principal may read. Every entry carries `deletedAt`, and `deletedBy` where the principal may read who changed an entry. The query filter is off for that whole query, so every row reached through a relation says explicitly that it is not in the trash: a trashed target never matches a filter or a row rule and is never populated. A filter on the id of a relation matches only a target the same read would show: of that type, not in the trash, in the site of a site-scoped type, and published on a published read. `PurgeAsync` needs `content.{type}.purge`, deletes the content rows of every state and locale, the references the entry holds and every version of it, published ones included, and raises `EntryPurgedEvent` per locale. It follows the rules of restore: an entry that is not in the trash, in another site or type, or refused by the row rule of the grant is `EntryNotFoundException`, the same for all of them; one the principal reaches in some locales only is `ContentAccessDeniedException`. References from other entries to a purged one are the values of their fields and stay; a read leaves out a target that is not there, and publishing warns with `FCMS2404` as it did for the trashed one. The values the entry held are free again, since uniqueness counted the trash.
+
+**Retention.** `[PublishPolicy(TrashRetentionDays = n)]` (`PublishPolicy(trashRetentionDays: n)` on the builder, `trashRetentionDays` in schema JSON) sets how long a type keeps its trash; the default is 30 days, the least 1 and the most 36500. The job `content.trash.purge` runs daily at 03:17 UTC from a recurring schedule stored at every startup. Each run purges, per type and site, the entries trashed longer ago than the type keeps them, through `PurgeAsync` as `Principal.System` with `Origin.Job`, one transaction per entry, at most 500 purges per type per run (`Limit` in the payload). Due entries are read oldest first in pages (`PageSize`), so an entry that fails is logged and left for the next run without using up the limit or hiding those behind it, and a type that fails is logged while the others still run. The purge checks inside its transaction that the entry went to the trash before the cutoff, so one trashed again since is left alone. A run logs the number purged and failed. A run that finds nothing due does nothing, so it is safe to run again.
+
 **Reference integrity.** On publish, references to unpublished or trashed entries produce warnings in the response (`warnings[]`) by default; `[PublishPolicy(UnpublishedReferences = Block)]` on a type turns them into a 422. A target counts as published when it has a published row in the entry's locale or its fallback chain. A self-reference is exempt. A cycle is not: under `Block` neither side can go first, and the way out is to publish one side without the reference and set it afterwards.
 
 The answer must not be a way to ask about an entry the principal may not read, since a write takes any id:

src/FluentCMS.Core/Content/ContentService.Trash.cs

@@ -0,0 +1,94 @@
+using FluentCMS.Content;
+using FluentCMS.Core.Authorization;
+using FluentCMS.Core.Content.Query;
+using FluentCMS.Localization;
+using FluentCMS.Schema;
+
+namespace FluentCMS.Core.Content;
+
+internal sealed partial class ContentService
+{
+    public Task<PagedResult<ContentDocument>> QueryTrashAsync(string type, ContentQuery query, CancellationToken cancellationToken = default) =>
+        QueryTrashAsync(type, query, everySystemField: false, cancellationToken);
+
+    // The trash is read like drafts are: by the engine, so the read grant, its row rule, its locale and field limits and
+    // the site all hold. Restore or purge on the type is what lets a principal look. What it cannot read is not listed,
+    // and a type it cannot read answers as an unknown one does.
+    public async Task<PagedResult<ContentDocument>> QueryTrashAsync(string typeName, ContentQuery query, bool everySystemField, CancellationToken cancellationToken)
+    {
+        ArgumentNullException.ThrowIfNull(typeName);
+        ArgumentNullException.ThrowIfNull(query);
+
+        var operation = Current();
+        var type = schema.FindContentType(typeName)
+            ?? throw new ValidationException([new ValidationError("type", QueryErrorCodes.UnknownField, $"'{typeName}' is not a content type that can be read.")]);
+
+        var site = query.Site ?? operation.Site;
+        var locale = await ResolveLocaleAsync(site, query.Locale ?? operation.Locale, cancellationToken);
+        if (locale == ILocaleResolver.All)
+        {
+            throw new ValidationException([new ValidationError("locale", QueryErrorCodes.InvalidParameter, "The trash is read in one locale.")]);
+        }
+
+        var target = new AccessTarget(type.Scope == ContentScope.Global ? operation.Site : site, locale);
+        var permissions = await evaluator.ResolveAsync(operation, cancellationToken);
+        if (!permissions.Allows(type, ContentOperation.Restore, target) && !permissions.Allows(type, ContentOperation.Purge, target))
+        {
+            if (!permissions.Allows(type, ContentOperation.Read, target))
+            {
+                throw new ValidationException([new ValidationError("type", QueryErrorCodes.UnknownField, $"'{typeName}' is not a content type that can be read.")]);
+            }
+
+            throw new ContentAccessDeniedException(type.Name, CorePermissions.OperationName(ContentOperation.Restore));
+        }
+
+        var page = await ReadAsync(typeName, query with { Fallback = LocaleFallback.None }, null, everySystemField, cancellationToken, trashed: true);
+        if (page.Items.Count == 0)
+        {
+            return page;
+        }
+
+        var marks = await store.TrashMarksAsync([.. page.Items.Select(entry => (Guid)entry["id"]!)], type.Localized ? locale : "", cancellationToken);
+        var namesPeople = permissions.CanReadField(type, "updatedBy", target);
+        foreach (var entry in page.Items)
+        {
+            if (marks.TryGetValue((Guid)entry["id"]!, out var mark))
+            {
+                entry["deletedAt"] = mark.At;
+                if (namesPeople)
+                {
+                    entry["deletedBy"] = mark.By;
+                }
+            }
+        }
+
+        return page;
+    }
+
+    // Only an entry in the trash can be purged, so a live one is not found, and one the row rule of the purge grant
+    // excludes is not found either, whatever the id. Every version goes with it, published ones included.
+    public Task PurgeAsync(string type, Guid id, CancellationToken cancellationToken = default) =>
+        PurgeAsync(type, id, trashedBefore: null, cancellationToken);
+
+    // The purge job passes the cutoff it listed the entry by, and an entry put in the trash again since is not found.
+    internal async Task PurgeAsync(string type, Guid id, DateTimeOffset? trashedBefore, CancellationToken cancellationToken)
+    {
+        ArgumentNullException.ThrowIfNull(type);
+
+        var write = await BeginAsync(type, ContentOperation.Purge, cancellationToken);
+        await InTransactionAsync(
+            async () =>
+            {
+                var (documents, row, _) = await ReachAllLocalesAsync(write, ContentOperation.Purge, id, trashed: true, cancellationToken);
+
+                if (await store.PurgeAsync(write.Type, id, trashedBefore, cancellationToken) != documents.Count)
+                {
+                    throw new EntryNotFoundException(write.Type.Name, id);
+                }
+
+                _pending.AddRange(documents.Select(document => new EntryChange(EntryChangeKind.Purged, write.Type.Name, id, document.Locale, document.DraftVersion, write.Operation)));
+                return row;
+            },
+            cancellationToken);
+    }
+}

src/FluentCMS.Core/Content/ContentService.cs

@@ -251,7 +251,7 @@ private async Task<string> ResolveLocaleAsync(SiteId site, string? locale, Cance
         await locales.ResolveAsync(new LocaleRequest(site) { Parameter = locale }, cancellationToken);
 
     // Reads ask the engine, which holds the permission on the type, the row rule, the site and the field limits as one query.
-    private async Task<PagedResult<ContentDocument>> ReadAsync(string typeName, ContentQuery query, Guid? id, bool everySystemField, CancellationToken cancellationToken)
+    private async Task<PagedResult<ContentDocument>> ReadAsync(string typeName, ContentQuery query, Guid? id, bool everySystemField, CancellationToken cancellationToken, bool trashed = false)
     {
         richText.ThrowIfUnknown(query.RichTextFormat);
 
@@ -266,14 +266,21 @@ private async Task<PagedResult<ContentDocument>> ReadAsync(string typeName, Cont
         var site = query.Site ?? operation.Site;
         var locale = await ResolveLocaleAsync(site, query.Locale ?? operation.Locale, cancellationToken);
         var everyLocale = locale == ILocaleResolver.All;
+        if (trashed && everyLocale)
+        {
+            throw new ValidationException([new ValidationError("locale", QueryErrorCodes.InvalidParameter, "The trash is read in one locale.")]);
+        }
+
         var oneLocale = everyLocale ? await ResolveLocaleAsync(site, null, cancellationToken) : locale;
 
         var bound = query with { Site = site, Locale = oneLocale };
         IReadOnlyList<ContentDocument> found = [];
         ContentQueryResult? page = null;
         if (!everyLocale || id is null)
         {
-            page = await engine.ExecuteAsync(typeName, bound, everySystemField, cancellationToken);
+            page = trashed
+                ? await engine.ExecuteTrashAsync(typeName, bound, everySystemField, cancellationToken)
+                : await engine.ExecuteAsync(typeName, bound, everySystemField, cancellationToken);
             found = page.Items;
         }
 

src/FluentCMS.Core/Content/EntryChange.cs

@@ -12,6 +12,7 @@ internal enum EntryChangeKind
     Unpublished,
     DraftDiscarded,
     VersionRestored,
+    Purged,
 }
 
 internal sealed record EntryChange(EntryChangeKind Kind, string Type, Guid Id, string Locale, int Version, OperationContext Operation)

src/FluentCMS.Core/Content/EntryStore.cs

@@ -172,6 +172,42 @@ public Task SetTrashedAsync(Guid id, DateTimeOffset? at, Guid? by, CancellationT
                     .SetProperty(document => document.DeletedBy, by),
                 cancellationToken);
 
+    // When and by whom each of these entries was trashed, in one locale. A trashed row is not read through the query filter.
+    public async Task<Dictionary<Guid, (DateTimeOffset? At, Guid? By)>> TrashMarksAsync(IReadOnlyCollection<Guid> ids, string locale, CancellationToken cancellationToken)
+    {
+        var marks = await context.Documents.IgnoreQueryFilters().AsNoTracking()
+            .Where(document => ids.Contains(document.DocumentId) && document.Locale == locale && document.DeletedAt != null)
+            .Select(document => new { document.DocumentId, document.DeletedAt, document.DeletedBy })
+            .ToListAsync(cancellationToken);
+
+        return marks.ToDictionary(mark => mark.DocumentId, mark => (mark.DeletedAt, mark.DeletedBy));
+    }
+
+    // Everything an entry holds, in every state and locale. The documents go first and only while they are in the trash
+    // and, when a cutoff is given, were put there before it, so that an entry trashed again since it was read as due is not
+    // taken; the caller compares the count with the rows it saw and rolls back on a difference. References to the entry from
+    // others are theirs: they are the values of those entries' fields, and a read leaves out a target that is not there.
+    public async Task<int> PurgeAsync(SchemaType type, Guid id, DateTimeOffset? trashedBefore, CancellationToken cancellationToken)
+    {
+        var typeName = type.Name;
+        var documents = context.Documents.IgnoreQueryFilters().Where(document => document.DocumentId == id && document.Type == typeName && document.DeletedAt != null);
+        if (trashedBefore is { } before)
+        {
+            documents = documents.Where(document => document.DeletedAt < before);
+        }
+
+        var removed = await documents.ExecuteDeleteAsync(cancellationToken);
+        await context.Versions.Where(version => version.DocumentId == id).ExecuteDeleteAsync(cancellationToken);
+        await context.References.Where(reference => reference.SourceType == typeName && reference.SourceDocumentId == id).ExecuteDeleteAsync(cancellationToken);
+        if (type.Localized)
+        {
+            await context.Rows(StorageNames.LocaleTable(type)).Where(row => EF.Property<Guid>(row, StorageNames.DocumentId) == id).ExecuteDeleteAsync(cancellationToken);
+        }
+
+        await context.Rows(StorageNames.Table(type)).Where(row => EF.Property<Guid>(row, StorageNames.DocumentId) == id).ExecuteDeleteAsync(cancellationToken);
+        return removed;
+    }
+
     // Saves the draft of one locale: the document when it is new, the shared row, the locale row, the references and the
     // version, in one save, and prunes the versions over the cap of the type.
     public Task WriteDraftAsync(SchemaType type, Document row, bool isNewDocument, ContentDocument entry, Origin origin, string? clientId, CancellationToken cancellationToken, string? message = null) =>

src/FluentCMS.Core/Content/Query/ContentQueryEngine.cs

@@ -33,11 +33,19 @@ public Task<ContentQueryResult> ExecuteAsync(string typeName, ContentQuery query
 
     // everySystemField is for a content class, which has a property for each of them; a document leaves out the two that
     // are not in the response shape. Field permissions still decide.
-    public async Task<ContentQueryResult> ExecuteAsync(string typeName, ContentQuery query, bool everySystemField, CancellationToken cancellationToken)
+    public Task<ContentQueryResult> ExecuteAsync(string typeName, ContentQuery query, bool everySystemField, CancellationToken cancellationToken) =>
+        ExecuteAsync(typeName, query, everySystemField, trashed: false, cancellationToken);
+
+    // Reads the entries in the trash as drafts. Whoever calls this has decided that the principal may look at the trash;
+    // what a draft read is limited by still holds.
+    public Task<ContentQueryResult> ExecuteTrashAsync(string typeName, ContentQuery query, bool everySystemField, CancellationToken cancellationToken) =>
+        ExecuteAsync(typeName, query with { Status = QueryStatus.Draft }, everySystemField, trashed: true, cancellationToken);
+
+    private async Task<ContentQueryResult> ExecuteAsync(string typeName, ContentQuery query, bool everySystemField, bool trashed, CancellationToken cancellationToken)
     {
         ArgumentNullException.ThrowIfNull(query);
 
-        var compiled = Compile(await access.ForQueryAsync(query.Site, query.Locale ?? "", cancellationToken), typeName, query, everySystemField);
+        var compiled = Compile(await access.ForQueryAsync(query.Site, query.Locale ?? "", cancellationToken), typeName, query, everySystemField, trashed);
 
         long? total = compiled.WithTotal ? await compiled.Filtered.LongCountAsync(cancellationToken) : null;
         var rows = await compiled.Rows.ToListAsync(cancellationToken);
@@ -61,12 +69,12 @@ public async Task<ContentQueryResult> ExecuteAsync(string typeName, ContentQuery
     // Separate from running it, so that a test can read the SQL.
     internal CompiledQuery Compile(string typeName, ContentQuery query) => Compile(access, typeName, query);
 
-    private CompiledQuery Compile(IContentQueryAccess access, string typeName, ContentQuery query, bool everySystemField = false)
+    private CompiledQuery Compile(IContentQueryAccess access, string typeName, ContentQuery query, bool everySystemField = false, bool trashed = false)
     {
         ArgumentNullException.ThrowIfNull(query);
 
         var state = query.Status == QueryStatus.Draft ? ContentState.Draft : ContentState.Published;
-        var scope = new QueryScope(context, schema, dialect, access, options, state, query.Locale ?? "", query.Site);
+        var scope = new QueryScope(context, schema, dialect, access, options, state, query.Locale ?? "", query.Site, trashed);
 
         if (schema.FindContentType(typeName) is not { } type || !access.CanReadType(type))
         {
@@ -84,7 +92,11 @@ private CompiledQuery Compile(IContentQueryAccess access, string typeName, Conte
         }
 
         var row = Expression.Parameter(typeof(QueryRow), "row");
-        var rows = scope.Rows(type);
+        var rows = scope.Rows(type, trashedRoot: trashed);
+        if (trashed)
+        {
+            rows = rows.IgnoreQueryFilters();
+        }
 
         if (new FilterCompiler(scope).Where(type, query.Filter, row) is { } filter)
         {