From 9919a9af13681939e54a4793e28cd1eef731dadf Mon Sep 17 00:00:00 2001 From: Yaraslau Tamashevich Date: Tue, 17 Nov 2026 06:32:34 +0100 Subject: [PATCH 1/3] Specify the one table every surface uses Adds docs/spec/table/table.md: a table as a run-time description that any C++ model can offer (one or many per model), a single pipeline for document tables, model-backed tables, collections and server pages, unloaded cells and data levels with partial pages and keyed refetches, the server level cache and client cache rules, where sort and filter run for a browser client, table configuration (column order/hide/resize/pin, chained sort, Excel-like filters, column groups with quantifiers, values with units), persisted table state through an injected store, and benchmarks with budgets at 100,000 x 300 on an in-memory C++ model. engine.md, database_pages.md and the spec index point at it. Refs #1085 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01No1NzmrVC1p6J5NnZm43g2 --- docs/spec/README.md | 5 +- docs/spec/table/database_pages.md | 12 + docs/spec/table/engine.md | 5 + docs/spec/table/table.md | 3116 +++++++++++++++++++++++++++++ 4 files changed, 3137 insertions(+), 1 deletion(-) create mode 100644 docs/spec/table/table.md diff --git a/docs/spec/README.md b/docs/spec/README.md index 76c803065..a3e6920a6 100644 --- a/docs/spec/README.md +++ b/docs/spec/README.md @@ -136,7 +136,10 @@ rules, exact values and reactive session, headless · mapping for a table, on and off the owner thread, and `table::apply` for a server · [`table/database_pages.md`](table/database_pages.md) — paging a table that -lives in a database: `planQuery`, `Page::totalKind` and the conformance helper +lives in a database: `planQuery`, `Page::totalKind` and the conformance helper · +[`table/table.md`](table/table.md) — the one table every surface +uses: a table as a description any model can offer, the single pipeline, +unloaded cells and levels, the caches, configuration, and the budgets **Reactive state and control** [`reactive/signals.md`](reactive/signals.md) · diff --git a/docs/spec/table/database_pages.md b/docs/spec/table/database_pages.md index 20c8f619e..bb3803696 100644 --- a/docs/spec/table/database_pages.md +++ b/docs/spec/table/database_pages.md @@ -206,6 +206,11 @@ message naming the column. The predicate runs after `validate`, so the usual ref that the refusals are equal values. Which columns a client may offer to sort and filter belongs in the schema and is not part of this seam. +A model that asks for it (`Residual::Allow`) is not refused: it receives a prefilter plan, fetches the rows that +plan admits, and `finishPlan` answers the whole query over them in C++ +([`table.md`, "What a store cannot express"](table.md#what-a-store-cannot-express)). The refusal above is what +a model that does not ask receives. + ## Totals `Page` has `TotalKind totalKind = TotalKind::Exact`, with `enum class TotalKind { Exact, AtLeast }`, written in JSON @@ -213,6 +218,9 @@ by name, `"exact"` and `"atLeast"`, as the module's other wire enums are: - `Exact`: `total` is the count of the filtered view. - `AtLeast`: `total` is a lower bound and the view has at least that many rows; a client shows "100,000 or more". + A total is a lower bound for one of two reasons: the count stopped at the cap, or the filter cannot yet decide some + rows because a cell it reads is not loaded ([`table.md`, "Totals"](table.md#totals)), and `Page::undecided` + is then not zero. - `total` absent: not counted. `apply` reports `Exact` at or below `QueryLimits::totalCap` and omits the total above it. A model @@ -225,6 +233,10 @@ than `totalCap`, reports `AtLeast` with `total = totalCap`. Rules: - A `limit` of zero returns the count and no rows. - The conformance helper holds an `Exact` total to the true count on either side of the cap: where `apply` omits the total it counts the view itself. +- When the oracle reports undecided rows, a total at or below the cap is `AtLeast` on both sides and equals the count + of the rows the filter has admitted; the helper holds the model to that count, to that kind and to the same + `undecided`. Above the cap the rule is the cap's: `AtLeast` with `total = totalCap`, or an `Exact` count, which a + model may report only when `undecided` is zero. `PageWindow::totalKind()` returns the kind of the latest reply that carried a total; `reset` clears it with the total. `PageWindow` stops fetching past an `Exact` total only; past an `AtLeast` total it keeps fetching until a diff --git a/docs/spec/table/engine.md b/docs/spec/table/engine.md index d3ab205d3..a85a3a8ff 100644 --- a/docs/spec/table/engine.md +++ b/docs/spec/table/engine.md @@ -304,6 +304,10 @@ edited included; the next repair holds them again. model's translation and reports every disagreement. `compileFilterGuarded` is the one compile step `apply`, `Engine` and `planQuery` share, so a collator or date parser that throws is `ServiceFailed` in all three. The design is [`database_pages.md`](database_pages.md). +- A table whose rows are cheap and whose slow columns load later marks the unloaded cells `Unloaded`: an unloaded cell + sorts after the empties, a filter over it is three-valued, and its row is undecided rather than rejected. A column + names the data level its cells need, a query names the columns it shows, and a `Page` says which of its cells are + still coming. The design is [`table.md`](table.md). - `escapeLikePattern` is for a database-backed server that maps `contains` to SQL `LIKE`: it keeps `%` and `_` ordinary. - `TableError` serialises with its code by name (`"unsupportedOperator"`), so a @@ -470,6 +474,7 @@ against `wire::decode` both ways. - `docs/superpowers/specs/2026-10-08-table-engine-design.md` — the program-level design this module implements. +- [table.md](table.md) — the one table every surface uses: descriptions, the pipeline, unloaded cells, caches, configuration and budgets. - [../core/callback_scope.md](../core/callback_scope.md) — the lifetime gate on every completion. - [../core/executor.md](../core/executor.md) — owner and worker executors. diff --git a/docs/spec/table/table.md b/docs/spec/table/table.md new file mode 100644 index 000000000..793deee39 --- /dev/null +++ b/docs/spec/table/table.md @@ -0,0 +1,3116 @@ +# `morph::table` — the table + +Design spec for the one table every surface of morph uses: what a table is, how a model offers one or many, the +single pipeline from a model's rows to the screen, slow columns and the caches that hold them, where sort and +filter run, what a user can configure, and the performance all of it is held to. Read [`engine.md`](engine.md) +first, for the engine this builds on, and [`database_pages.md`](database_pages.md) for a model that pages in a +store. + +## Contents + +- [Goal](#goal) +- [A table is a description](#a-table-is-a-description) +- [The pipeline](#the-pipeline) +- [Sources](#sources) +- [The client table model](#the-client-table-model) +- [The renderer](#the-renderer) +- [Every surface on the pipeline](#every-surface-on-the-pipeline) +- [The unloaded cell](#the-unloaded-cell) +- [Semantics of an unloaded cell](#semantics-of-an-unloaded-cell) +- [Data levels](#data-levels) +- [Partial pages](#partial-pages) +- [Keyed queries](#keyed-queries) +- [Arrival](#arrival) +- [The window under changes](#the-window-under-changes) +- [Caching layers](#caching-layers) +- [Where sort and filter run](#where-sort-and-filter-run) +- [Configuring a table](#configuring-a-table) +- [Values with units](#values-with-units) +- [What a store cannot express](#what-a-store-cannot-express) +- [Performance](#performance) +- [Invariants](#invariants) +- [API surface](#api-surface) +- [Tests](#tests) +- [What this does not do](#what-this-does-not-do) +- [Alternatives considered](#alternatives-considered) +- [Failure modes](#failure-modes) +- [Open questions](#open-questions) +- [Neighbouring specs](#neighbouring-specs) +- [Delivery order](#delivery-order) + +## Goal + +The goal is one table implementation, complete and fast, that every surface reuses. Three things define it. + +- **A table is generic.** It is a description a model builds at run time and rows that fit it, not a C++ row + type. Any model can offer a table, or several. +- **One pipeline.** Every surface that shows rows in columns goes through `morph::table`: the same order, the + same filters, the same window, the same renderer. +- **Held to a size.** 100,000 rows and 300 columns, natively and in a browser, with budgets and a benchmark for + every layer. + +### Why one pipeline + +Rows in columns come from four kinds of source: a list of values a document holds, a collection's rows, a +model's table paged by the server, and a typed list action's pages. Each is a source of the one pipeline, and +none has ordering code of its own. + +| What one pipeline gives | Without it | +|---|---| +| One text order on a screen | Two tables side by side order the same text differently, one by a collator's key and one by a fold inside each comparison. | +| A repair when a row changes | A changed row sorts the whole list again, and folds text once per comparison instead of once per row. | +| A window at any size | A screen over a server's pages holds one page, and a header sort orders that page instead of the table. | +| Work off the painting thread where there is one, in bounded steps where there is not | A sort of 100,000 rows blocks the frame it runs in. | +| A table from a description | Every table needs a row struct and an action written for it. | + +`forEach` and `grid` lay rows out without columns, sort nothing and are not tables. + +### Slow columns + +A row can be cheap to read while some of its columns cost a second query per row. Without a way to say "not loaded +yet", a server either holds every page until all of its columns are read, or sends the unloaded cells as empty +cells, which the engine then sorts, filters and counts as real empty values while the client polls for the rest. + +Three things are needed together, because each is useless without the next: a cell state that is neither a value +nor empty; a name on each column for the load it depends on, so a query's columns say which loads it needs; and a +reply that says which of its cells are still coming, with a push when they arrive. + +**The name.** The state is `Unloaded`, and this document says "unloaded cell". `pending` already means "a request +is in flight" on `Engine::pending()`, `PageWindow::pending()`, a cache entry and a `table` element's `pending` +value, which documents read; those keep their name and their meaning. The two are different facts: an unloaded +cell may have no request in flight at all (its level is not wanted, or its load failed), and a table that is +`pending` may hold no unloaded cell. + +## A table is a description + +A table is not a C++ type. It is a **description**, built by a model when it is asked, and rows that fit it. A +model offers no table, one, or many, and what a table's columns are may depend on who asks. + +```cpp +struct TableColumn { + ColumnInfo info; // id, kind, comparator, format, level, dimensions + std::string label; + std::string group; // a declared column group's id, or empty + bool sortable = true; + bool filterable = true; + bool shown = true; // the default visibility +}; + +struct ColumnGroupInfo { + std::string id; + std::string label; + std::vector columns; // 2 to 16 column ids of one kind +}; + +struct TableCapabilities { + bool keys = true; // keyed queries + bool distinct = true; // distinct values +}; + +struct TableDescription { + std::string id; // unique within the model: "designs" + std::string label; + std::vector columns; // in the declared order + std::string key; // the id of the column that is the row's key + std::vector groups; + std::vector levels; // every level a column names, base excluded + std::vector units; // the units its `Measure` columns may hold + TableCapabilities capabilities; + std::string revision; // set by the framework: a hash of everything above +}; +``` + +Everything this design says about a table is said of its description: its levels, its groups, its units, which +columns sort and filter. A client draws a table from the description alone, with no compile-time knowledge of +it. The model leaves `revision` empty; the framework fills it from the description's canonical JSON, so two +descriptions are equal exactly when their revisions are. + +### A model's tables + +A model serves a table by returning what the framework needs to run it: + +```cpp +struct TableContext { + session::Context const& session; // who asks +}; + +/// A table paged in a store, in place of rows held in memory (database_pages.md). +struct PlannedTable { + /// Runs a plan. For a whole plan `rows` is the page in order; for a prefilter plan it is the candidates. + std::function(QueryPlan const&)> run; + /// What the store can push down; see "What a store cannot express". + std::function translatable; + Residual residual = Residual::Refuse; +}; + +struct PlanRows { + std::shared_ptr rows; // columns as the description's + std::optional total; // a whole plan's count + TotalKind totalKind = TotalKind::Exact; + std::vector indices; // a keyed plan: each row's view row + std::optional distinct; // a pushed-down distinct plan +}; + +struct ServedTable { + TableDescription description; + /// The rows the caller may see, for a table held in memory or read lazily: `apply` runs over it. + std::shared_ptr source; + /// In place of `source`, for a table paged in a store. + std::optional planned; + /// The slow levels, one per level the description names. + std::vector> levels; + /// Whether the caller may read a column; empty allows every column of the description. + std::function mayRead; + /// Names everything about the caller that changes the view beyond the query: the row scope and the + /// collation. Empty when every caller sees the same rows in the same order. + std::string scope; + /// The collator, date parser, custom comparators and unit registry this caller's order uses. + Services services; + QueryLimits limits; +}; + +class DesignModel { +public: + table::ServedTable designs(table::TableContext const& context); + table::ServedTable recipes(table::TableContext const& context); +}; + +BRIDGE_REGISTER_TABLE(lims::DesignModel, "designs", &lims::DesignModel::designs) +BRIDGE_REGISTER_TABLE(lims::DesignModel, "recipes", &lims::DesignModel::recipes, table::rowScoped) +``` + +- The model's function runs per request, on the model's strand, with the caller's session, so a description may + differ by principal: columns a user's country adds, columns a role may not see. It returns shared, immutable + things and is expected to be cheap; the description of a table whose columns do not vary is built once. +- **The framework does the rest**, and none of it on the model's strand. When the function returns, the + framework takes the source's snapshot, and then on the server's table executor + (`RemoteServerOptions::tableWork`, a pool) validates the query against the description and `limits`, finds or + computes the view ([The view cache](#the-view-cache)), reads the level snapshots, builds the reply, and wants + what the reply showed as unloaded. A model writes no paging, sorting, filtering, keyed or distinct-value code. +- **Services are the caller's.** `services` is what makes text order by the caller's locale. Left empty it is the + default `Services`, whose collator orders by code point; a table with text columns that users sort sets a + collator. Two callers with different collators have different `scope`s. +- **The row scope is the model's, and it is applied before the framework sees a row.** `source` and + `planned.run` return only rows the caller may see. Nothing downstream filters rows by principal: a level + cache, the view cache and a reply all work on the rows they are given, which is why a cached value cannot + cross a scope. `scope` names the scope so that callers who see the same rows share a view. +- **What a row scope costs.** A model whose callers all see the same rows returns one shared source. A model + with row scopes keeps one source per scope, built when the scope is first asked for and shared from then on; + `table::scoped(source, keys)` makes one from a shared source and the scope's row keys, without copying a cell. + Building a source per request at 100,000 rows is a full pass per request and defeats the view cache, whose key + holds the source's snapshot. +- **A table is always C++ code.** Every table is a registered function of a C++ model; nothing on the server + defines a table from a document or from data alone. What is generic is the table's shape, which that function + decides when it runs. A model's set of table ids is fixed when it is registered; a table's columns are not. + +**Two actions per table, made by the registration.** `BRIDGE_REGISTER_TABLE(Model, "designs", …)` registers: + +| Action id | Takes | Returns | +|---|---|---| +| `Table.designs` | `TableQuery` | `GenericPage`: a page, a keyed reply or distinct values, as the query asks | +| `Table.designs.describe` | nothing | `TableDescription` | + +One generic action taking a table id in its body was the alternative, and loses on every count that matters: + +- **Authorisation** is per model type and action id. An action per table lets a principal read one table of a + model and not another with the authorizer as it is. +- **The catalog** lists actions; a table per action appears in it, with its effects and the record it holds, + without a second directory of tables. +- **The client's cache key** is the action and its body; an action per table keeps two tables' entries apart + without the cache reading bodies. +- **Effects** are declared per action. + +A typed list action that takes a `TableQuery` and returns a `Page` remains a table: it is the same contract +with the description derived from the row type. + +**Changing a row is the application's action.** The registration makes no edit action. A model changes a row +through an action of its own, which declares its effect on the table and returns the keys it changed; a cell +editor on a screen is a column's `cell` template calling that action. A generic, description-driven cell edit +(an edit action per table, editable columns, an editor per kind) is a design of its own and is not in this +contract ([What this does not do](#what-this-does-not-do)). + +**The table's record.** Each registered table is a keyed record named `.` +(`DesignModel.designs`), registered by `BRIDGE_REGISTER_TABLE` beside the table's actions. A record is +registered under a C++ type, and a table has one: `table::TableOf<&DesignModel::designs>`, a tag type made from +the table's function, which is distinct for each table and is no row struct. The registration specialises it +with the model and the id, so everything that names a table in C++ names its function, and a function no +registration names does not compile as a table. The record is what a client watches, what an action's effects +name, and what a level cache announces. + +Registration options after the function are the record's: + +| Option | Effect | +|---|---| +| `table::rowScoped` | The record is registered with `privateKeys`. Required when callers see different rows: a change event then reaches each watcher without keys, except the writer's own, so no watcher learns the key of a row outside its scope. | +| `table::derivedFrom` | A change of any of the records is a change of the table's record, as for any derived record. | + +A row-scoped table pays for its privacy: an arrival that a level cache announces has no caller, so it reaches +every watcher without keys, and each refetches its visible pages instead of its rows +([The window under changes](#the-window-under-changes)). A table that is not row-scoped sends keys to every +watcher the authoriser admits. + +```cpp +struct ChangedRow { RowId key; }; + +/// The rows of one table that an action changed. Its JSON is {"rows": [{"key": …}, …]}. +template +struct ChangedRows { std::vector rows; }; + +struct RenameDesignResult { + table::ChangedRows<&lims::DesignModel::designs> designs; +}; + +BRIDGE_REGISTER_EFFECTS(lims::RenameDesign, ::morph::effects::updatesTable<&lims::DesignModel::designs>) +BRIDGE_REGISTER_EFFECTS(lims::ImportDesigns, ::morph::effects::changesTable<&lims::DesignModel::designs>) +``` + +- `changesTable` is a whole-table change. +- `updatesTable` names rows. An `updates` effect is resolved against the action's result: the result walk + matches each slot to a record by its C++ type, and the keys are collected only at the paths where the result + holds that record. `ChangedRows` is that slot. Its walk reports `TableOf` at its `rows` member with + the key member `key`, so the keys of one table are collected as any record's are, and `updatesTable` on + an action whose result has no `ChangedRows` is refused when the registries are resolved. +- **An action that changes two tables returns two members**, one `ChangedRows` per table, and declares both + effects. One untyped list of keys could not say which table a key belongs to. +- Watches, admission, fan-out, coalescing and `announce` are the record machinery's: a table is one more + record. + +**The table's action holds the record opaquely.** Which records an action's result holds is found by walking +the result's type, and a `GenericPage` has parallel `keys` and `rows` arrays, not objects with a key member. So +the registration states that `Table.` holds the table's record with no path. That statement admits a watch +of the record to a principal who may run the action, and tells a client cache that it cannot find rows in the +value: an entry whose record is held opaquely is marked by every change of the record, keyed or not, and +nothing is merged into it ([The client's caches](#the-clients-caches)). + +### Typed rows are a convenience + +A row struct still describes a table, by reflection: + +```cpp +BRIDGE_REGISTER_TABLE(lims::SampleModel, "samples", table::typed(&lims::SampleModel::sampleRows)) +``` + +`table::typed` builds the description from the row type: columns and kinds from its members, labels and +units from its schema, levels and groups from `columnLevels()` and `columnGroups()`, the key from its record. It +serves the rows through a `RowsSource`. Wherever this document shows a typed form (`columnLevels()`, +`LevelLoaded`, `Page`), it is this layer; the contract underneath is the description, the +`DataSource` and the generic page. + +### Rows on the wire + +```cpp +using GenericRow = std::vector; // one cell per entry of the page's `columns` + +struct GenericPage { + std::string revision; // the description's + std::vector columns; // the columns of each row, in the description's order + std::vector keys; // each row's key + std::vector rows; + // … the members of Page other than `rows`; see "Partial pages" … +}; +``` + +A page of a generic table is row arrays under one header: + +```json +{ "revision": "9f2c", + "columns": ["id", "name", "net", "share", "weight", "dose"], + "keys": [101, 102], + "rows": [[101, "Alpha", [1250, 100], [3, 1000], [3, 10, "kg"], [5, 1, [1, 1000]]], + [102, "Beta", null, null, [200, 1, "g"], [250, 1]]], + "total": 9093, "totalKind": "exact", "offset": 0, + "unloadedRows": [{"index": 1, "levels": ["recipe"]}], + "undecided": 0, "awaiting": [] } +``` + +- `columns` names the columns of each row array: the ones the query asked for + ([Data levels](#data-levels)), in the description's order. `keys` holds each row's key whether or not the key + column is among them, so a reader finds a row without knowing which column is the key. +- A cell is written by its `Cell` alternative: + + | Cell | JSON | + |---|---| + | empty | `null` | + | integer; a date as days, a date-time as seconds | an integer | + | `double`, in a `Number` column only | a number | + | text | a string | + | boolean | `true` or `false` | + | exact value (`Rational`) | `[numerator, denominator]` | + | `QuantityCell` | `[numerator, denominator, [factor numerator, factor denominator]]`: the amount in the cell's own unit and its factor to the column's canonical unit. A factor of 1 is left out, which is also how a plain `Rational` in a `Quantity` column reads. | + | `Measure` | `[numerator, denominator, unit id]`, the unit's string id | + | unloaded | `null`, with the row and level named in `unloadedRows` | + + The column's kind, from the description, says how to read a cell; nothing is guessed from the JSON. The third + element tells a quantity from a measure only as a check, since the kind already does. A cell that does not + read as its column's kind is an invalid cell on the client, as it is on the server. +- An unloaded cell costs nothing per cell: the state is per row and level. +- Every other member is the typed page's, with the same meaning. +- `revision` is the description's. A client holding another revision fetches the description again and resets + its window: the columns changed under it. +- **Why rows and not columns.** Column arrays would be the same bytes. A reply is read by row: a keyed reply + replaces rows, and a reply carries a few rows as cheaply as a page. +- **Size.** A row object repeats every column's id, which is most of the 21 bytes a cell costs in a typed page. + Row arrays drop them (W5), and a query that names the forty columns in or near view instead of all 300 + carries an eighth of the cells (W1). +- A typed table served through `table::typed` is written in this form, its cells read by the row's cell + readers. + +### A table on the server + +- **A `DataSource` per table and scope.** The engine and `apply` take a `DataSource` and `ColumnInfo`s and need + no row type; a generic table is that, with nothing in between. +- **Level caches** are keyed by table record, level and row key. Two tables of one model that show the same + rows, by the same key, share one cache: the cache names both records (`LevelCacheOptions::records`), is + invalidated by a change of either, and announces an arrival on both. One load serves both tables. +- **Announcing and invalidating** are by table record, with the row keys of that table. +- **A table whose columns depend on the session** builds its description in its function. Its rows may be one + `DataSource` with every possible column, narrowed by `mayRead`. The level caches hold raw values for a column + list that does not depend on the caller ([The server level cache](#the-server-level-cache)) and are shared. + +## The pipeline + +```text + in-document rows a collection's rows a model's table + (a list of values) (the session's reply) (description; TableQuery → GenericPage) + │ │ │ + ValueRowsSource VectorSource ViewCache, LevelCache, apply / planQuery ← server + │ │ │ pages, keyed rows, change events + └────────── table::Engine ───┘ PageWindow ← client + (client mode: sort, filter, (server mode: the pages around + repair, off-thread where possible) the visible rows) + │ │ + └──────────────── TableRows ───────────────┘ + rows by view index, counts, changes + │ + the mount: realizes rows and columns in view + │ + MorphTable (Qt Quick), the terminal table +``` + +There is one of each stage. A surface chooses a source; the action's shape chooses the mode +([Where sort and filter run](#where-sort-and-filter-run)); everything below `TableRows` is shared. + +## Sources + +- **`ValueRowsSource`**, in the UI layer, is a `DataSource` over a document's list of values + ([below](#a-documents-rows-as-a-source)). +- **`VectorSource`** holds a collection's rows, as [`../forms/engine.md`](../forms/engine.md) specifies. +- **A model's table** ([A table is a description](#a-table-is-a-description)) is the server-mode source: its + `Table.` action takes a `TableQuery`. The framework answers through `apply` over the table's `DataSource`, + or through a translated plan; a slow level is served from a `LevelCache`. + +### A document's rows as a source + +`ValueRowsSource` turns a list of document values into typed cells, and turns a change of the list into the +smallest row changes. + +- **Columns are built when used.** The source has a column for each column of the node that can be sorted or + filtered, and builds a column's cells only while the column is in the sort chain, has a filter, or has a + values list open. A column no one uses holds no cells, so a table of 300 filterable columns is not held + twice. A column's cells are dropped when its last use ends. +- **A cell is a derived value.** A column's cell for a row is the column's `sortBy` evaluated in the row's + scope, or the row's field of the column's `id`. Each is a tracked computation: when the row, or anything else + its `sortBy` reads, changes, that cell is derived again and the source notifies `Updated` for that row alone. + A change to what one `sortBy` reads for every row derives the column again and notifies `Reset`. +- **A new list is matched by key, then by cell.** A row whose value is the same object as before is kept + without being read. A row whose value is another object (a refetched list shares no storage with the previous + one) has its cells of the columns in use derived and compared with the cells held, and is `Updated` only when + one differs. The comparison is of cells of the columns in use, never of whole row values, so its cost is rows + times columns in use and not rows times fields. Keys the previous list lacks are `Inserted`, keys the new one + lacks are `Removed`. +- **Kinds.** A column part's `kind` (a literal: `text`, `integer`, `decimal`, `number`, `date`, `dateTime`, + `bool`) decides the column's kind. Without it the kind is inferred from every non-null cell of the column: + all integers is `Integer`; integers and exact decimals together is `Decimal`, an integer being an exact + decimal; any `double` among numbers is `Number`; all booleans is `Bool`; all text is `Text`. Any other mix + takes the kind of the first non-null cell, and a cell not of that kind is invalid: it sorts last and is + reported once at the `sort` path. A change that alters the inferred kind is a `Reset` of that column. +- **Text order.** The source orders text with a collator whose sort key is the default collator's fold followed + by the text itself, which is the expression language's order for text: case-folded first, then by its own + bytes. `null` is an empty cell and sorts last in either direction, as the expression language's `sort` places + it. +- It keeps each row's value beside its cells for the row's scope. + +## The client table model + +`ui::TableRows` is what the mount reads, whatever the source: + +```cpp +using SortChain = std::vector; + +struct TableTotal { + std::int64_t rows = 0; + table::TotalKind kind = table::TotalKind::Exact; + bool known = false; // false until a server-mode table's first reply +}; + +struct TableRowRef { + table::RowId const& key; + /// The row as a record keyed by column id. A field is read from the held cell when it is asked for. + Value const& row; +}; + +struct TableCountsView { + TableTotal view; + std::optional source; // rows before the filter, where the model knows it + std::int64_t undecided = 0; + bool pending = false; // a sort, filter or fetch is in flight + std::vector awaiting; // levels the order still waits for + std::vector failed; // levels whose load failed for a row the table shows or orders by +}; + +class TableRows { +public: + virtual ~TableRows() = default; + [[nodiscard]] virtual TableTotal total() const = 0; + /// The row at a view index, or nothing while it is not fetched. + [[nodiscard]] virtual std::optional row(std::int64_t index) const = 0; + /// The levels not loaded for the row at a view index, and which of them failed. + [[nodiscard]] virtual table::UnloadedLevels unloadedLevels(std::int64_t index) const = 0; + /// Tells the model which rows and which columns are in or near view; a server-mode model fetches them. + virtual void visible(std::int64_t first, std::int64_t count, std::span columns) = 0; + /// The keys of a range of view rows, for a range selection; completes when every key is known. + virtual async::Completion, table::TableError>> + keysOf(std::int64_t first, std::int64_t count) = 0; + virtual std::expected setSort(SortChain sort) = 0; + virtual std::expected setFilter(table::FilterSpec filter) = 0; + [[nodiscard]] virtual TableCountsView counts() const = 0; + /// Called after the rows changed, with the operations that turn the old view into the new one. + virtual void onChange(std::function handler) = 0; +}; +``` + +- **`EngineRows`** wraps an `Engine` over a `ValueRowsSource` or a `VectorSource`. Every row is present; + `visible` does nothing; `keysOf` completes at once; a change is the engine's `ViewChange`. +- **`WindowRows`** wraps a `PageWindow`, the table's description and the session. `visible` returns fetches, + which it sends through the session's read that makes no cache entry; replies go to `accept` and `patch`; the + table record's change events go to `refresh` and `invalidate` + ([The window under changes](#the-window-under-changes)). A change is the diff of the rows it held against the + rows it holds, by key. It owns the two timing rules the window itself has no clock for: when a fetch is sent + while the user scrolls, and how often an awaited order is fetched again. + +The mount's windowing, which decides the realized rows and columns of a large table, calls `visible` with both +ranges. The mount applies a `ViewChange` to its keyed rows as it applies a new list, so a row keeps its widget, +its native state and its selection when it moves. + +**A row is a lazy record.** A document reads a row as `row.name`, whichever way the table is served. In server +mode the record is a view over the row's held cells: reading a field converts that one cell, so a row of 300 +columns of which twenty are drawn converts twenty cells. + +**Which columns a table fetches.** A server-mode table asks for the columns it draws and the columns its +expressions read: + +- the columns in the renderer's column window ([The renderer](#the-renderer)); +- for each of those, the columns its `cell` expression reads as `row.`, found when the document loads; +- the columns `rowTone`, `rowLabel`, `onActivate` and a row action read, for every row; +- every column, when an expression reads the row in a way the loader cannot follow (`row[name]`, or the row + passed whole to a function); the loader reports that once as a warning naming the node, since it turns the + column window off for the wire. + +A field read of a column the window does not hold for that row reads as absent and asks the window for the +column, so an expression the loader misjudged costs a fetch and not a wrong value. + +A model-backed `table` node names a `model` and a `table` (the table's id), or a `model` and the `action` of a +typed list action. It lowers to a `WindowRows`: the description comes from the table's `describe` action, +through the query cache, and gives the node its columns, so a node that declares no `columns` shows the table +as described, and one that declares some lays those out and takes the rest of each column from the +description. Its `body` is the query's fixed part, its header sorts and its `filters` set the query's `sort` and +`filters`, and `highlightKey` is a keyed query for that row's view index. A typed list action is read into the +same window: its description is derived from the action's schema, and the reader turns each row object into +cells; it returns whole rows whatever `columns` says. + +## The renderer + +A table realizes a widget only for cells whose row and whose column are in or near view. Both windows are the +mount's, so every backend has them: Qt Quick's `MorphTable` and the terminal's table draw what the mount +realizes. + +### When a table is windowed + +```cpp +struct Virtualization { + std::size_t threshold = 2000; // rows: at or above it, rows are windowed + std::size_t overscan = 10; // rows kept realized beyond the visible range, each side + std::size_t columnThreshold = 40; // visible columns: above it, columns are windowed + std::size_t cellThreshold = 20'000; // rows × visible columns: above it, both are windowed + std::size_t columnOverscan = 4; // columns kept realized beyond the visible range, each side + std::size_t realizePerFrame = 240; // cells created or rebound in one frame at most +}; +``` + +A table under all three thresholds realizes everything, as a short table does. A table of 1,999 rows and 300 +columns is over `cellThreshold` and is windowed both ways; without that bound it would realize 600,000 cells. +A server-mode table is always windowed by rows. + +### The column window + +- **Which columns.** The pinned columns ([Columns](#columns-order-visibility-pinning-width)), then the visible + columns that intersect the horizontal viewport, then `columnOverscan` columns each side. The window runs over + the visible columns in layout order. +- **Extents live in the mount.** The mount holds each visible column's width and the running sum of their + offsets (`ColumnExtents`). Finding the columns at a scroll position is a binary search, and a backend is given + the offsets and widths of the columns in the window and the total width. No per-frame work reads every + column. +- **Widths.** A column's width comes from its header, its declared width, and the cells measured while it was + in the window. As for rows, a windowed table's widths only grow; a width the user set does not change. +- **Anchoring.** When a column to the left of the first fully visible unpinned column grows, the horizontal + scroll offset grows by the same amount in the same frame, so the columns in view do not move. +- **The header is windowed too.** A header cell and a group heading exist for the columns in the window only. +- **A hidden column has no header, no width and no cells.** + +### Cells + +- **Recycling.** A cell that leaves either window goes to a pool and is rebound to the next cell that enters. + Cells drawn by the default cell of one role share a pool, whatever their column; a column with its own + `cell` template has a pool of its own. A row that scrolls in therefore creates no widgets once the pools are + warm, and a column that scrolls in rebinds the cells of the column that left. +- **A frame's budget.** One frame creates or rebinds at most `realizePerFrame` cells. Cells beyond it draw as + blank placeholders of their column's width and are realized over the following frames, nearest the centre of + the viewport first. A fling or a drag of the scroll thumb therefore passes over placeholders and never waits + for realization; rows the view only passes are not realized at all. +- **Row height.** A row's height is the tallest its realized cells have measured, and only grows while the row + is realized, as widths do: a wrapped cell that scrolls in sideways may make its row taller, and one that + scrolls out does not make it shorter. A row that leaves the row window forgets its height; an unrealized row + counts as the mean of the measured heights. When a row above the first visible row grows, the vertical offset + grows with it. A table whose columns all have `wrap: false` has one row height and skips the measurement. +- **A row not fetched yet** draws as a placeholder row of the estimated height. +- **An unloaded cell** draws a placeholder of a fixed width in its column's alignment. A cell whose level + failed to load draws an error mark with the same width, and the status line offers "Retry" + ([The status line](#the-status-line)). +- The levels a server-mode table asks for are those of the columns it fetches + ([Data levels](#data-levels)), so a level shown only far to the right is not loaded until the user scrolls to + it. + +### Keyboard, focus and assistive technology + +Windowing must not change what a keyboard or a screen reader can reach. + +- **Focus is a row key and a column id**, not a widget. The focused cell's row and its column stay realized + while the table has focus, whatever the windows: one row and one column beyond them at most. When the focused + row leaves the view, focus moves to the row now at its view index, or the last row. +- **Keys.** Arrows move the focused cell; `Home` and `End` go to the first and last visible column of the row; + `Ctrl+Home` and `Ctrl+End` to the first and last row of the view; `Page Up` and `Page Down` move by the rows + in view. `Ctrl+A` and `Shift` with the arrows select as they do on a short table. +- **Over rows not fetched.** Moving to a view row the window does not hold moves focus to that index: the + window fetches it, the placeholder row holds focus and is announced as loading, and the row takes the focus + when it arrives. With a total that is a lower bound, `Ctrl+End` goes to the last row known. +- **Counts.** The table reports its row count as the view's total when that is exact and as unknown when it is + a lower bound, and its column count as the visible columns. Every realized row and cell reports its view row + and its column's place in the layout, so "row 5,012 of 9,093" is read out although fifty rows exist. +- **Range selection in server mode.** A backend that selects a range itself needs every key in row order + (`setExistingKeys`), which a server-mode table does not have. There the backend reports the gesture as two + view rows and the lowering resolves it through `TableRows::keysOf`: rows the window holds give their keys, + and the rest are fetched as pages that name no columns, which carry keys alone. A range of more than + `SessionOptions::maxRangeRows` (10,000) rows is refused with a message in the status line. `Ctrl+A` is not + offered in server mode. +- **Selection in server mode** is a set of keys. It is not pruned by the filter, which the client cannot + evaluate; `selectedCount` counts the keys. + +## Every surface on the pipeline + +Every surface that shows rows in columns is on the pipeline. + +| Surface | On the pipeline | +|---|---| +| A document `table` over `rows` | `ValueRowsSource` → `Engine` → `EngineRows`. | +| A document `table` over a model's table (`model` and `table`) or a typed list action (`model` and `action`) | The description from the catalog's `describe` action, rows through a `PageWindow` as `WindowRows`. | +| A forms `collection` or `master-detail` | Its `VectorSource` feeds an `EngineRows` through the same table lowering. | +| A `rows` table bound to a typed page's `rows` | A client-mode table over one page, on the query cache; declared as model-backed it gets a window. | +| `PageWindow` | The server-mode client cache of every model-backed table. | +| `forEach`, `grid`, the expression language's `sort` | Not tables. | + +### The document `table` node on the engine + +Each property of the node, and where it is served. "Rows table" is a table over `rows`; "model-backed" is one +over `model`. + +| Property | Served by | Notes | +|---|---|---| +| `rows` | `ValueRowsSource`, matched by `key` | A changed row is repaired, not sorted again. | +| `model` with `table` or `action`, `body`, `mode` | `WindowRows`, over the table's description | `table` is the id of one of the model's tables. | +| `columns` on a model-backed table | The description's columns, overridden per id by the node's parts | Optional there: absent, the table shows what its description says. | +| `columns`, `cell`, `role`, `align`, `valign`, `width`, `wrap`, `shrink`, a column's `kind` | The mount and the backend; `kind` by `ValueRowsSource` | Cells exist for columns in the window only. | +| `sort`, `sortable`, `sortBy`, `onSortChange` | `TableRows::setSort`; `sortBy` is the column's cell. `Shift`+click and the header menu build a chain ([Sort chains](#sort-chains)) | Values of one kind order as the expression language orders them. A cell not of its column's kind is invalid, sorts last and is reported once at the `sort` path. | +| `pageSize`, `page` | A slice of the engine's view | Refused on a model-backed table, whose window pages. | +| `group` (`by`, `heading`) | The run index is a hidden leading sort key computed by `ValueRowsSource`; headings are placed by the lowering after paging | Rows table only. | +| `rowSpans` | The lowering, on the page's rows | Drawn only unsorted and below the windowing thresholds. Rows table only. | +| `selectionMode`, `selection`, `onSelectionChange`, `selectColumn`, `selectAllLabel`, `rowLabel` | The lowering, by key; select-all selects the view through `Selection::selectAll` | A model-backed table has ranges through `keysOf` and no select-all. | +| `rowTone`, `onActivate` | The row's scope | Their columns are fetched for every row. | +| `empty` | Shown while the view is empty | On a model-backed table it waits for the first reply, and is not shown while rows are undecided. | +| `filters`, `onFiltersChange`, a column's `filterable` | `TableRows::setFilter`, as a `FilterSpec`; set from the header menu ([Filters from the header](#filters-from-the-header)) | Served on every table. | +| `layout`, `onLayoutChange`, `configurable`, a column's `locked` | `ColumnLayout` ([Columns](#columns-order-visibility-pinning-width)) | Every table has the header and table menus unless `configurable` is `false`. | +| `status` | The status line ([The status line](#the-status-line)) | A literal: `true`, `false`, or absent for the default. | +| `persist` | The table's persisted state ([Persisted state](#persisted-state)) | A literal. | +| `highlightKey` | A keyed query for the row's view index, then `visible` around it | Model-backed only. | +| Element values `selection`, `selectedCount`, `sort`, `viewCount`, `pending`, `page`, `pageCount`, `pageFirst`, `pageLast` | `TableRows::counts` and the page slice | On a model-backed table `viewCount` is the reply's total, with `viewCountKind` (`exact` or `atLeast`) and `undecided` beside it. Every table has `layout`, `visibleColumns`, `hiddenCount`, `filters`, `awaiting` and `failed`. | + +A document that uses none of `layout`, `status`, `persist`, a column's `kind` or `locked` loads and orders as it +does without them, with two visible differences that are intended: its tables have header and table menus, and +a table with a state store restores its arrangement. + +The engine's additions for configuring a table are `oneOf`, `distinctValues`, and widths and pinning in +`ColumnLayout`. Everything else is in the UI layer: `ValueRowsSource`, the two `TableRows`, and the lowering +that reads them. + +### Collections and master-detail + +A `CollectionSession` keeps its rows in a `VectorSource` and leaves sorting, filtering and selection to the +engine. Its list is an `EngineRows`, and a `collection` or `master-detail` node lowers its table through the +same lowering as a document `table`, so the interpreter draws a collection's rows with the table every other +screen has. The chrome around the table (header, confirm, editor) is its own lowering and is not part of this +design. A collection is always in client mode: its `v-query` is executed with an empty body and its rows arrive +whole. A collection too large for that is a model's table shown by a `table` node. + +### Screens over a server's pages + +A screen that binds a `rows` table to a `Page`'s `rows` shows the page the query fetched, as a client-mode +table over those rows: a header sort orders the page. To get a window, a header sort that orders the table and +row fetches on arrival, it declares the table with `model` and `table` or `action`. The framework does not +guess that a `rows` expression is a page: an expression can be anything, and a table that changed mode because +of the shape of an expression would be the surprise this design removes. + +### What the interpreter does for a table + +The interpreter does not sort, filter or page a table. It builds the node's `TableRows`, reads rows from it by +view index, and does what is the document's around them: group headings, row spans, selection across pages, the +status line and the element values. The order of a table is the engine's on every surface. The expression +language's own `sort` function is not a table and orders as it does. + +## The unloaded cell + +`Cell` has an alternative for a value that is not loaded: + +```cpp +/// A cell whose value is not loaded yet. It is neither a value nor empty. +struct Unloaded {}; + +using Cell = std::variant; // `Measure`: see "Values with units" +``` + +`CellState` has `Unloaded`, after its other values. `KeyColumn::read` maps an `Unloaded` cell to +`CellState::Unloaded` before +it looks at the column's kind, so every kind treats it alike, and stores no key for it. + +Why a cell alternative inside the engine: + +- **No second call to forget.** A cell is read only through a `RowSnapshot`, as a `Cell`. An unloaded state carried + in the `Cell` reaches key building, `Engine::cellAt` and a renderer on a `DataSource` together with the value's + absence. +- **Atomic with the snapshot.** A snapshot is immutable; a cell and its "loaded" state cannot disagree across + threads when they are one object. +- **Arrival needs no new source API.** A level arriving is the row's cells changing from `Unloaded` to values: + `VectorSource::updateRow` or `RowPatcher::patchRow`, a new snapshot sharing every other chunk, and + `RowChange{Updated}`. `TableSnapshot`, `VectorSource`, `RowPatcher`, `ColumnSink` and `ChangeKind` are unchanged. +- **Finer than a level when needed.** A level is the unit of loading, but nothing forces every cell of a level to + arrive together. + +Cost: every reader of `Cell` must know the alternative. `Cell` does not grow (an empty struct is smaller than +`QuantityCell`), and apunloaded keeps every existing alternative's index. `displayText` is the one `std::visit` over +a `Cell`; its last branch formats the value, so the new alternative does not compile until it has a branch. + +**The wire is a second channel.** A page says which cells are unloaded beside its rows, per row and level +([Partial pages](#partial-pages)): a generic page holds `null` in an unloaded cell, and a typed row holds whatever +the server left in the member. A table node reads the page through a window, which always reads `unloadedRows`. +A reader that takes a typed page's `rows` and nothing else shows an unloaded member as empty; that is left as a +stated limit of typed pages read outside a table, because closing it means changing every row type +([Alternatives](#alternatives-considered)). + +## Semantics of an unloaded cell + +### Unloaded cells in the order + +`compareKeys` ranks a cell before it compares keys: valid first, then empty and invalid (which tie), then unloaded +(which tie). The rank ignores the key's direction, so unloaded cells are last ascending and descending. `RowOrder` +still breaks every tie by source row, so the order stays strict and total and every sort stays stable. + +Unloaded is ranked after empty, and does not tie with it, so an unloaded cell is never placed among the empties +it might not belong to. Under a single sort key the rows whose place is unknown are one tail at the end of the +view. Under a chain, a row with an unloaded cell in a later key is last among the rows that tie on the keys +before it, so there is one such tail per run of equal earlier keys. + +### Unloaded cells in filters + +A filter over unloaded cells is three-valued. `CompiledFilter::evaluate` returns `Match::Yes`, `Match::No` or +`Match::Unknown`: + +- A column filter on an unloaded cell is `Unknown`, whatever its entries: `isEmpty` and `notEmpty` included, because + whether the cell is empty is not known, and `exclude` included, because whether an exclude entry matches is not + known. On any other cell it is `Yes` or `No` exactly as `passes` decides. +- A group is decided by its quantifier over its columns' answers, by the table in + [Column groups](#column-groups). A plain group is `Yes` when any of its columns is `Yes`; otherwise `Unknown` + when any is `Unknown`; otherwise `No`. +- A row is `No` when any column filter or group is `No`; otherwise `Unknown` when any is `Unknown`; otherwise `Yes`. + +So a row is rejected by what is known even while other cells load, and admitted by a group as soon as one loaded +column matches. `FilterRun` drops a row at the stage that answers `No`; a row a stage answers `Unknown` stays a +candidate, marked, and runs through the later stages, any of which may still reject it. + +A row that evaluates `Yes` is in the view. A row that evaluates `Unknown` is **undecided**: it is not in the view, +it is not rejected, and it is counted. `Engine::undecidedRowCount()` and `Page::undecided` report the count, so a +table's status line says "40 rows still loading" under an empty or short view rather than "no rows". + +Undecided rows are held out of the view, not shown in it, for three reasons. A filter that shows rows it may +reject lets a user select and act on rows that do not match. The count of such a view is an upper bound, which +`TotalKind` cannot say, while the count of the decided rows is a lower bound, which `AtLeast` says. And it keeps +the engine's rule in one sentence: a cell matches an operator only when the cell is known to. + +### Totals + +`total` counts the rows in the view. When `undecided` is not zero the view may still grow, so the total is a lower +bound and `totalKind` is `AtLeast`. `AtLeast` therefore has two causes, the cap and undecided rows, and +`Page::undecided` tells them apart: a client shows "100,000 or more" for the first and "12 so far, 40 loading" for +the second. `apply` still omits `total` above `QueryLimits::totalCap`. A sort on an unloaded column changes no +membership and leaves the total `Exact`. The rule a store-backed model and the conformance helper follow is in +[`database_pages.md`, "Totals"](database_pages.md#totals). + +### Unique values + +An unloaded cell is not a value and is not the empty value. `distinctValues` +([Filters from the header](#filters-from-the-header)) leaves out a row whose cell in the +column is unloaded, and a row another column's filter leaves undecided, and reports `partial = true` when it left +any out, so a filter menu built from the list can say that more values may appear. A `oneOf` entry on an unloaded +cell is `Unknown`, as every operator is. + +### Selection and edits + +- A `Selection` holds `RowId`s and is unaffected by what a row's cells hold. A row with unloaded cells can be + selected. `selectAll(engine)` selects the view, so it does not select undecided rows. With `pruneOnFilter`, a + selected row that becomes undecided has left the view and is pruned, as any row leaving the view is. +- `CellEdits::commit` refuses an edit of a cell that is `Unloaded` in its engine's view: the cell gets the + `FieldError` "not loaded yet" and the injected commit is not called. An edit of a value the user has not seen is + an overwrite of whatever arrives. A `CellEdits` without an engine cannot check, and the server's action is then + the only guard. +- An edit of a loaded cell in a row whose other cells are unloaded commits as usual. The commit's settled row may + still hold `Unloaded` cells. + +### On arrival + +Arrival is an update. The source publishes a snapshot in which the row's cells hold values and notifies +`RowChange{Updated, rows}`; the engine's repair does the rest. A repair's input holds the previous undecided rows +beside the previous view (`JobInput::previousUndecided`), because a dirty row may have been in either, and its +result holds the new ones (`JobResult::undecided`). + +- The rows' keys are rebuilt on copies of the key columns, and each row is evaluated again. +- A row that was undecided and is now `Yes` enters the view at its sorted place (`Inserted`), and the rows below + it shift by one. One that is now `No` produces no view operation. +- A row that was in the view is merged into the order at its true place (`Moved`) and reported `Changed`. +- `Engine::onUndecidedChange` receives the new count whenever it changed, after `onViewChange` when the same job + changed the view, and on its own when the job changed nothing else. Without it a row going from undecided to + rejected would leave "40 rows still loading" on screen. +- Under `ReorderPolicy::Deferred` a row that was in the view keeps its displayed index until the view settles, as + any updated row does. A row entering from the undecided set was not displayed and takes its true place at once, + so `Deferred` stops displayed rows from jumping and does not stop them from shifting. + +A cell may also go from a value back to `Unloaded` (a server dropping what it loaded). That is the same update in +the other direction: the row moves to the unloaded tail, or becomes undecided and leaves the view. + +## Data levels + +A **level** is a named unit of loading. A column of a table's description names the level its cells need: + +```cpp +struct ColumnInfo { + std::string id; + ColumnKind kind = ColumnKind::Text; + std::string comparator; + std::string format; + /// The data level the column's cells need. Empty is the base level, which every row has when it exists. + std::string level; +}; +``` + +Levels are strings, like column ids, so a table whose columns depend on the session names levels the same way. + +**A cell of a base-level column is never `Unloaded`.** No level names such a cell, so no load would resolve it. It +is a defect of the source and is refused, not shown: key building fails its job with +`TableErrorCode::SourceContract` naming the column (from `apply` as the returned error, from an `Engine` through +`onError`, the view staying as it was). + +A table's description says each column's level, and its `levels` lists them. A typed row declares its levels +once, on the row type, and `table::typed` puts them in the description: + +```cpp +struct ColumnLevel { std::string_view column; std::string_view level; }; + +struct DesignRow { + // … members … + static constexpr std::array columnLevels() noexcept { + return {{{"component", "recipe"}, {"quantity", "recipe"}}}; + } +}; +``` + +`RowsSource` fills `ColumnInfo::level` from that declaration, and the row's schema carries `x-level` on each +named property, which `deriveColumns` copies to a collection's column as it copies `x-comparator`. An entry +naming a column the row does not have is refused where the columns are reflected. + +Which levels a row has loaded is not a property of the description. A generic table's level columns are read +from the level's cache ([The server level cache](#the-server-level-cache)): a row the cache does not hold has +`Unloaded` cells there. A typed table says it with a predicate: + +```cpp +/// Whether `level` is loaded for `row`. Empty means every level is loaded. +template +using LevelLoaded = std::function; +``` + +`RowsSource` and the typed `apply` take a `LevelLoaded` after `comparators`. A cell of a column whose +level it refuses is `Unloaded`; the member's own value is not read. + +A query says which columns the reply carries: + +```cpp +struct TableQuery { + std::vector sort; + FilterSpec filters; + PageRequest page; + /// The columns the reply carries, by id. Absent means every column the caller may read; empty means none, + /// and the reply then holds keys alone. + std::optional> columns; + /// When set, the reply holds only the view's rows with these keys (see "Keyed queries"). + std::optional> keys; + /// When set, the reply holds a column's or a group's distinct values and no rows (see "Filters from the + /// header"). + std::optional distinct; + /// The user asked for failed loads to be tried again: the level caches load the failed keys this query + /// needs without waiting out their retry delay. + bool retry = false; +}; +``` + +**The query names columns, and the levels follow from them.** A client names the columns it draws and reads +([The client table model](#the-client-table-model)); the server sends those cells and loads the levels those +columns need. This is the one definition of what a view "shows": the columns its query names. Two things rest +on it. + +- **Bytes.** A page of 200 rows and 300 columns is about 1.27 MB as row objects and takes 15 ms to decode + natively; in a browser that decode runs on the thread that paints, for the twenty columns the user can see. A + page of the forty columns in or near view is an eighth of it (W1, W2). +- **Loads.** A level is loaded for a page's rows only when a named column needs it, so a level whose columns + are all out of the column window is not loaded by scrolling down. + +The query cache is not involved: a window's fetches make no cache entry +([The client's caches](#the-clients-caches)), so a column that scrolls into view changes no cache key. The +window holds, for each page, the columns it has, and fetches the ones it lacks. + +`validate` bounds `columns` by `QueryLimits::maxColumns` (512), puts it in the description's order and removes +duplicates. A name that is no column is `UnknownColumn`, and one the caller may not read is +`ColumnNotReadable`, as for a sorted or filtered column. A client that is refused either way holds a stale +description: it fetches the description again, fits its layout to it and asks once more. + +```cpp +struct LevelsNeeded { + /// Levels of the sorted and filtered columns: the view's membership and order depend on them for every row. + std::vector order; + /// Levels of the columns the query names: needed only for the rows of the reply. + std::vector display; +}; + +[[nodiscard]] LevelsNeeded levelsNeeded(TableQuery const& query, std::span columns, + std::function const& mayRead = {}); +``` + +- `order` is the levels of every sort key's column, every filtered column that has an entry, every column of + every group, and the column or group of a distinct request: the columns `apply` builds keys for. +- `display` is the levels of the columns in `query.columns`, or of every column the caller may read when + `columns` is absent: a client that does not say what it shows is assumed to show everything. +- Both are sorted, without duplicates, and never hold the base level. A level may be in both. +- The function is pure and reads no rows. It is called on the query `validate` returned. + +The two lists differ in how much a server must load. An `order` level is needed for every row the rest of the +filter does not reject, because any of them may enter or move. A `display` level is needed for the rows of the +reply only. A server loads nothing else: a view with no column of a level named, sorted or filtered does not +need the level, and its load never starts. A client-mode table calls the same function with its engine's sort +and filter and the columns of its column window. + +A query is a request, not a subscription: nothing tells the server that a client removed the sort or filter that +needed an `order` level. The level cache finishes the loads it was asked for; what it loaded stays useful to the +next query that needs the level. + +`apply` reads cells only of the sorted and filtered columns, and the reply's writer reads cells only of the +named columns for the reply's rows, so a source may produce every other cell lazily or not at all. + +## Partial pages + +A reply says which of its cells are still coming. The members below are a typed `Page`'s; a `GenericPage` +([Rows on the wire](#rows-on-the-wire)) carries the same members with the same meaning beside its `columns`, +`keys` and `rows`. + +```cpp +/// A row of a page with levels that are not loaded. +struct UnloadedRow { + std::int64_t index = 0; // into Page::rows + std::vector levels; // sorted, without duplicates, never the base level + std::vector failed; // the subset of `levels` whose load failed; usually empty +}; + +template +struct Page { + std::vector rows; + std::optional total; + std::int64_t offset = 0; + TotalKind totalKind = TotalKind::Exact; + /// The rows of this reply that have unloaded levels, ascending by index. A row not listed is fully loaded. + std::vector unloadedRows; + /// Rows the filter can neither admit nor reject until a level loads. They are in no page. + std::int64_t undecided = 0; + /// The order levels with a load wanted or in flight for at least one row. Not empty means the order and + /// the membership of the view may change without any row being edited. + std::vector awaiting; + /// The order levels whose load failed for at least one row, and is not being tried again yet. + std::vector failed; + /// A keyed reply only: the view row of each of `rows`. + std::vector indices; + /// A distinct-values reply only; `rows` is then empty. + std::optional distinct; +}; +``` + +- A cell of row `i` in column `c` is unloaded exactly when `c`'s level is in the `levels` of the `UnloadedRow` + with `index == i`. A generic row holds `null` there; the member of a typed row holds whatever the server + left, and a client does not read it. +- `unloadedRows` lists the unloaded levels of the levels the reply needed (`order` and `display`), not every + level of the table: a level no named column has is not looked up, so its state is not known to the reply. +- **A failed load is still unloaded.** A cell whose load failed has no value, so the engine treats it as any + unloaded cell: it sorts last and its filter is `Unknown`. What differs is what the client is told, so a table + can stop saying "loading": the level is in the row's `failed`, and for an order level in `Page::failed` + instead of `awaiting`. `awaiting` and `failed` together are the order levels that leave rows unloaded. +- The members beyond `rows`, `total`, `offset` and `totalKind` are absent-tolerant: a `Page` read without them + is a fully loaded page. + +`PageRows`, what `apply(DataSource const&, …)` returns, carries `totalKind`, `undecided`, `awaiting`, `failed` +and, for a keyed query, `indices`. The framework builds a generic page from it and takes each row's unloaded +levels from the level snapshots it read; the typed `apply` copies the same members into a `Page` and builds +`unloadedRows` from its `LevelLoaded`. + +```cpp +/// What is not loaded for one row: views into the page or window that holds the row. +struct UnloadedLevels { + std::span levels; + std::span failed; +}; +``` + +A `PageWindow` keeps each held row's unloaded levels (`unloadedLevels(index)`), and the `undecided()`, +`awaiting()` and `failed()` of the latest page reply; `reset` clears them. Unloaded cells do not make a page +incomplete: an unloaded row is a row the server has sent. + +## Keyed queries + +A query with `keys` set asks for specific rows of the view through the table's own action, so the model's +scope and authorisation apply to it as to a page. + +- The view is the one a page of the same sort and filter has. The reply's `rows` are the view's rows whose key + is in `keys`, in view order, with the query's `columns`; `indices` holds each one's view row; `unloadedRows` + describes them. A key that is not in the view (the row was removed, is rejected, or is undecided) has no row + in the reply. +- `page` does not apply; `validate` refuses a keyed query with a non-zero `page.offset` as `InvalidSpec`, one + that also sets `distinct` as `InvalidSpec`, and more than `QueryLimits::maxKeys` (1,000) keys as + `LimitExceeded`. Each text key is bounded by `QueryLimits::maxTextLength`. +- **A keyed reply carries the view's `total`, `totalKind`, `undecided`, `awaiting` and `failed`**, as a page + does. A window does not adopt them: keyed replies can arrive out of order, and only page replies, which a + revision orders, set the window's totals. It compares them with its own, and a difference tells it that rows + it did not ask about have entered or left the view + ([The window under changes](#the-window-under-changes)). +- The server answers a keyed query from the view it has cached for the sort and filter + ([The view cache](#the-view-cache)): a key's view row is a lookup, not a sort. +- **A store-backed model** pushes a keyed query down when its predicate accepts `translatable("", PlanUse::Keys)`: + it translates `QueryPlan::keys` and reports each row's view row. Otherwise the query is answered over the + store's candidates in C++, or refused when the model does not allow that + ([What a store cannot express](#what-a-store-cannot-express)). `standardCorpus` has a keyed query. A client + whose keyed fetch is refused with `Unavailable` falls back to `invalidate`. + +## Arrival + +No part of the path polls. Each step is caused by the one before it. + +1. **The level cache** ([Caching layers](#the-server-level-cache)) finishes a load for some rows and stores the + values. A read of the cache from that moment returns them. +2. **An engine on a source fed from it** (a client-mode table, or a server that keeps one) is given the rows as + `RowChange{Updated, rows}` and repairs them, as in [On arrival](#on-arrival). Its `ViewChange` names exactly the + rows that entered, moved or changed. +3. **The server tells the watchers.** The table is a registered record + ([The table's record](#a-models-tables)), and the cache announces the rows that arrived, after it stored + them: + + ```cpp + /// Reports a change to rows of `record` that no execute made. Callable from any thread. + /// `keys` absent reports the whole type. An unregistered record is refused and nothing is sent. + std::expected RemoteServer::announce(std::string_view record, + std::optional> keys); + ``` + + `announce` builds its `effects::Changes` with the helpers an execute's changes are built with, so the record's + derived records are included and more than `effects::kMaxChangeKeys` keys become a whole-type change. It posts + to the server strand and runs `fanOut` with no caller, as a time fact does: watches, re-authorisation and + coalescing apply unchanged, and a time-fact reporter naming the record is asked again. An announced change is + not handed to the change observers, so an arrival never invalidates the cache that announced it. The level is + not in the event: the keys are enough, and a client asks for what it shows. + + Keys cross as JSON. `table::changeKeys(std::span)` writes an integer key as a JSON number and a + text key as a JSON string; `table::rowIdOf(glz::raw_json const&)` reads one back, and an event key that is + neither makes the client treat the event as a whole-type change. The table's key column and the keys an + action's `ChangedRows` names must write the same JSON for the same row; a test of each table holds them to it. + + A row-scoped table's record has `privateKeys`, and an announcement has no caller, so it reaches every + watcher without its keys: each watcher refetches its visible pages instead of its rows. That is the price of + not telling a watcher the key of a row it may not see. It is one page request per watcher per coalesced + event, answered from the view cache. + + A load that fails is announced the same way, once, with the failed keys, so a client stops showing them as + loading. +4. **The client's cache** takes the event. A window answers with a keyed fetch for the event's keys, or with + page fetches ([below](#the-window-under-changes)); the query cache marks the entries concerned and they fetch + their page again ([Caching layers](#the-clients-caches)). +5. **The server answers** the keyed query or the page through the same action, from the view cache and the + level cache: nothing is loaded again, and the view is computed again only when the arrival was of a level + it orders by, once for all its watchers. +6. **The window patches** the rows in place and reports their view rows; the renderer reports exactly those rows + as changed. + +The record's `changes` events are not only arrivals. An action that edits a row declares `updates` or `changes` on +the same record, and its event reaches the same caches. A client cache therefore treats every event alike and +never assumes an event is an arrival. + +## The window under changes + +A window holds the rows of a model-backed table around the visible rows, and for each page the columns it has +fetched. `PageWindow` is the window of every `table` node: a row's key is its entry of the page's +`keys`. `PageWindow` holds a typed page's rows for a C++ client, with a `RowKey` function for the key +and no column window, since a typed row is whole. + +```cpp +struct PageFetch { + PageRequest request; + std::vector columns; // the columns asked for; a held page asks for the ones it lacks + std::uint64_t query = 0; + std::uint64_t revision = 0; +}; +struct RowsFetch { + std::vector keys; + std::vector columns; + std::uint64_t query = 0; + std::uint64_t sequence = 0; +}; +enum class Refresh : std::uint8_t { Nothing, Rows, Pages }; +struct RefreshPlan { Refresh kind = Refresh::Nothing; RowsFetch rows; }; + +explicit PageWindow(std::int64_t pageSize = 200, std::size_t maxPages = 8, RowKey key = {}, + std::size_t maxKeys = 1000, std::size_t maxColumns = 96); + +std::vector visible(std::int64_t first, std::int64_t count, std::span columns = {}); +std::vector superseded(); // fetches in flight for pages no longer near the visible rows +void cancelled(PageFetch const& fetch); +RefreshPlan refresh(std::span keys); // a keyed change of the table's record +RefreshPlan owed(); // after each accept +void invalidate(); // a whole-type change, a resync, a refused keyed fetch +void accept(PageFetch const& fetch, Page reply); +std::vector patch(RowsFetch const& fetch, Page reply); +``` + +The window has no clock and sends nothing: it returns what to fetch, and `WindowRows` decides when +([When a fetch is sent](#when-a-fetch-is-sent)). + +### Pages and columns + +- A held page stores its cells by column, so a column that scrolls into the column window is added to the page + and one that has left it is dropped without touching the others. A window holds at most `maxColumns` columns + a page; over that, the columns farthest from the column window go first. +- `visible(first, count, columns)` covers the visible rows and one page either side. For a page it does not + hold, or holds stale, it returns a fetch of the whole page with `columns`. For a page it holds complete but + without some of `columns`, it returns a fetch of the same rows naming only the columns the page lacks. One + fetch per page is in flight at most. +- `accept` of a column fetch adds the columns when the reply's `keys` are the page's keys, row for row. When + they are not, the rows moved between the two fetches, and the window invalidates. +- `accept` of a page reply whose `total`, `totalKind` or `undecided` differ from the window's adopts them and + marks every other held page stale: the view changed between two replies, whatever the events said. +- `superseded()` returns the fetches in flight for pages that are no longer within one page of the visible + rows. The client cancels them through their stop tokens and tells the window with `cancelled`. + +### `refresh(keys)`: rows or pages + +- `Pages` (it calls `invalidate`) when the window has no key function; when `awaiting()` is not empty, because + nearly every arrival then moves rows; or when the keys it would ask for exceed `maxKeys`. +- Otherwise `Rows`, asking for **every key of the event**, held or not, and for one **probe** per run of held + rows: the key of the run's first row. A held row with no unloaded level is asked for too: the event may be an + edit. +- `Nothing` when the window holds no row. +- While any page fetch is in flight, it also records the event's keys. + +An event's keys are asked for whether or not the window holds them because an edit can bring a row the window +has never seen into its pages, and the author of an action cannot know how each client has sorted. The reply's +view rows say what happened ([`patch`](#patchfetch-reply)). + +**`owed()`** closes the race between an event and a page reply. The reply to a page fetch and the event for a +change leave the server separately and arrive in either order, so an event can precede the reply whose rows it +concerns. After each `accept` the client calls `owed`, which returns `Rows` with the keys recorded while the +fetch was in flight. The recorded keys are dropped when no page fetch is in flight. + +### `patch(fetch, reply)` + +`patch` replaces a held row when the reply has a row with its key at the view row the window holds it at, +updates its unloaded levels, and returns the view rows it replaced, ascending. It calls `invalidate` instead +when the reply shows that rows shifted: + +| The reply shows | Meaning | +|---|---| +| A requested key the window holds has no row | The row left the view. | +| A key the window holds is at another view row | The row moved, or a row above it entered, left or moved across it. The probes are held keys, so this catches a shift caused by a row the window does not hold. | +| A key the window does not hold is at a view row inside a held page | A row entered the pages. | +| `total`, `totalKind` or `undecided` differ from the window's | A row entered or left somewhere in the view. Checked only on the reply to the latest keyed fetch sent, so an old reply cannot report an old total. | + +Why this is enough: a change that leaves every held row at its view row, with the same content, and leaves the +totals as they were, changes nothing the window shows. Any other change moves a held row, which a probe or a +requested key reports; or brings an unheld key into a held page, which its view row reports; or changes a +total. A check that fires wrongly, because a keyed reply and a page reply crossed, costs one page fetch and +ends there, since the refetched page agrees with itself. + +Each `RowsFetch` has a sequence, and the window remembers the latest sequence that asked for each key: a reply +row whose key a later fetch has asked for is ignored, missing or not, so an out-of-order reply cannot put older +values over newer ones. A reply of an older query is ignored. `patch` never changes which rows are held or +their order. + +### `invalidate()` + +`invalidate` takes a new revision and marks every held page stale. It keeps the held rows showing and keeps the +fetches in flight. + +- `visible` returns a whole-page fetch, tagged with the current revision, for each stale page around the visible + rows that has no fetch in flight. +- `accept` takes a whole-page reply whose revision is not older than the page's held content, and replaces the + page. The page is complete only when the reply's revision is the current one; otherwise it stays stale and the + next `visible` fetches it again. A reply older than the held content, or of an older query, is ignored. +- So a window invalidated more often than a round trip takes still advances by one reply per round trip, and is + current one round trip after the last event. The client calls `visible` after every `accept`. + +### When a fetch is sent + +`WindowRows` owns two delays. Neither asks for anything because time passed: each only holds back or merges a +request that a scroll or an event caused. + +- **Scrolling.** A fetch for a page next to a held page is sent at once: that is ordinary scrolling, and the + held margin hides the round trip. A fetch for a page farther away (a drag of the scroll thumb, a fling, + `Ctrl+End`) is sent when the visible rows have stayed within one page for `SessionOptions::tableSettle` + (80 ms). Columns follow the same rule sideways. Every fetch `superseded()` returns is cancelled. A drag of the + thumb across 100,000 rows therefore sends a fetch where it rests, not five hundred on the way. +- **An awaited order.** While `awaiting()` is not empty, every event invalidates. `WindowRows` sends the first + refetch at once and the next no sooner than `SessionOptions::tableAwaitedRefetch` (250 ms) after it, so a + watcher of a level that loads for a minute fetches its visible pages four times a second at most, and once + more after the last arrival. A loader announces per batch of rows, not per row. + +## Caching layers + +A value that is slow to load is loaded once and kept until something says it is wrong. Five layers hold what a +table shows on the way to the screen, and one notification, the table record's change event, refreshes all of +them. + +| Layer | Holds | Key | Bound | What removes or refreshes a value | +|---|---|---|---|---| +| Server level cache, `table::LevelCache` | The loaded values of one level, as the loader returned them, for every principal | table record, level, row key | `maxBytes`, least recently used first | `invalidate`; an action's effects on a record in `invalidatedBy`; eviction at the bound | +| Server view cache, `table::ViewCache` | The order and the undecided rows of a view | table record, `ServedTable::scope`, validated sort and filter | `maxBytes`, least recently used first | A new snapshot of the source; a new generation of an order level | +| Client query cache, the interpreter's `Session` | Whole replies of a query: a table's description, a distinct-values list, a page read by a `rows` table | model type, instance token, action, canonical body | `keepUnused` after the last query leaves | A change event marks it and it fetches again; `resync`; a move onto it after `staleTime` | +| Client `PageWindow` | The pages of a model-backed table around the visible rows, the columns each holds, each row's unloaded levels | query, then page index | `maxPages` (8) of `pageSize` (200) rows, `maxColumns` (96) columns | A change event (`refresh`, `invalidate`); `reset` on a new sort or filter | +| Renderer | The realized cells and the column widths measured from them | row key and column id | The two windows | The mount's diff of the rows it is given | + +Nothing below the server level cache ever causes a level to be loaded twice: a client cache that fetches again is +answered from the level cache. + +### The server level cache + +The framework owns the cache; the model supplies only the function that reads the values. + +```cpp +enum class LevelRead : std::uint8_t { Loaded, Unloaded, Failed }; +enum class LevelUse : std::uint8_t { Display, Order }; + +/// One state of a level. Immutable: nothing it returns changes while it lives, and holding it blocks no one. +class LevelSnapshot { +public: + virtual ~LevelSnapshot() = default; + /// Writes the row's cells for the level's columns, in the order of `LevelSource::columns()`. + [[nodiscard]] virtual LevelRead read(RowId const& key, std::span out) const = 0; + /// Differs between two snapshots that differ in any value. + [[nodiscard]] virtual std::uint64_t generation() const = 0; +}; + +/// What the framework needs of a level, whatever the type of its values. +class LevelSource { +public: + virtual ~LevelSource() = default; + [[nodiscard]] virtual std::string_view level() const = 0; + /// The level's columns. The same for every caller. + [[nodiscard]] virtual std::span columns() const = 0; + [[nodiscard]] virtual std::shared_ptr snapshot() const = 0; + /// Says the keys are needed. `Unavailable` when an order use cannot fit (rule 10). + virtual std::expected want(std::span keys, LevelUse use, bool retry = false) = 0; +}; + +struct LevelCacheOptions { + std::vector records; // the table records it serves; one, or several that share rows + std::string level; + std::vector columns; // the level's columns, for every caller, in a fixed order + std::size_t maxBytes = 256U << 20U; + std::size_t expectedBytes = 0; // a value's size before any is stored; 0: 64 bytes a column + /// Records whose change invalidates values. Default: `records`. + std::vector invalidatedBy; + exec::IExecutor* loads = nullptr; // where the loader runs; never the server strand + std::size_t batch = 64; // most keys handed to one loader call + std::chrono::seconds orderLease{60}; // how long an order use keeps its keys from eviction + std::chrono::seconds retryAfter{30}; // before a failed key is loaded again; doubles to `retryMax` + std::chrono::seconds retryMax{600}; + std::function clock; // empty: the steady clock +}; + +struct LevelCacheStats { + std::size_t hits, misses, loaderCalls, loadedKeys, failedLoads, droppedLoads, invalidatedKeys, evictedKeys, + leasedKeys, refusedOrders, bytes; +}; + +template +class LevelCache final : public LevelSource, public effects::IChangeObserver { +public: + /// Reads the level for `keys`. Called on `options.loads`; may block. It returns a value for every key it + /// read, including a row that has nothing at this level, whose value is then empty. A key it returns no + /// value for failed; throwing fails every key of the call. + using Loader = std::function>(std::span keys)>; + using SizeOf = std::function; // deep bytes + using Cells = std::function out)>; // in `columns` order + + LevelCache(LevelCacheOptions options, Loader loader, SizeOf sizeOf, Cells cells); + /// For `Value` = `std::vector` only: the value is the cells, and its size is their deep size. + LevelCache(LevelCacheOptions options, Loader loader); + + void invalidate(std::span keys); + void invalidateAll(); + void onLoadFailed(std::function)> handler); + [[nodiscard]] LevelCacheStats stats() const; +}; + +using CellLevelCache = LevelCache>; +``` + +**What a value is laid out against.** A level's columns are `LevelCacheOptions::columns`: one list for every +caller, declared when the cache is made. A generic value is one cell per entry of that list, in its order; a +typed value is turned into the same cells by `Cells`. A caller's description may hold fewer of the level's +columns than the list, in another order; the framework maps each of the caller's columns to its place in the +list by id and passes it through `mayRead`. A cached value is never laid out against a description, because +descriptions differ by caller and the cache is shared. + +**Size is deep.** `maxBytes` bounds what the values own, not `sizeof(Value)`: a `std::vector` of twenty +cells is 24 bytes by `sizeof` and about 1.1 kB in truth. `CellLevelCache` measures cells and their texts +itself. A cache of another value type is given `SizeOf`; there is no shallow default. + +**How the framework uses it**, for every query of a table: `levelsNeeded` on the validated query; a snapshot of +each needed level; the view, from the view cache or `apply`, over base columns from the table's source and +level columns from the snapshots, `Unloaded` where a snapshot holds no value; the reply; then `want` with the +keys the reply needed: the reply's rows for a `display` level, every row the filter did not reject for an +`order` level. The reply is built entirely from the snapshots taken at its start, so its rows and its +`unloadedRows` describe one state of each level. + +Rules: + +1. **Hit.** A key the cache holds is served to every query, page, session and client until it is invalidated or + evicted. No age expires it. +2. **One load.** `want` starts a load only for keys neither held nor in flight. Queries that need the same rows + while a load runs wait for that load's arrival event like any other client. +3. **Order of work.** Keys are loaded in batches of `batch`, `Display` keys before `Order` keys and the most + recently wanted first, so the rows a user is looking at now come before the rows a sort needs eventually. +4. **Raw values, scoped reads.** The cache holds what the loader returned and knows no principal. A value reaches + a reply only through a row of the caller's own source and a column `mayRead` allows, both decided before the + cache is read. A level whose values differ by principal is not one level: it is declared as one level per + scope, or not cached. +5. **Invalidation is explicit**, and there are three sources. + - The application calls `invalidate(keys)` or `invalidateAll()`. + - An action's effects. The server hands every change an execute or a time fact derived to each attached + observer, on the server strand, before it queues the change's events. A keyed change of the row's own record + invalidates those keys; a whole-type change of it invalidates the level. The action's own event is the + notification, so the cache announces nothing. + - A change of another record in `invalidatedBy` (a recipe, when the level is a design's recipe data). Its keys + are not row keys, so the whole level is invalidated, and the cache announces the table's record without keys, + because no event for the table's record exists yet. +6. **An effect invalidates every level of the rows it names.** Effects name records, not levels, so a change to a + row is taken to change everything derived from it. That over-invalidates by one row's reload per edited row. A + level that does not depend on the row's own record says so by leaving the record out of `invalidatedBy`. +7. **A load in flight does not land stale.** Each row has a generation that an invalidation advances. A load + remembers the generation of each key when it starts and stores a value only where it is unchanged; the rest + are dropped and counted, and the query that follows the invalidation's event wants them again. +8. **Stored, then told.** A load publishes its values before it announces their keys; an invalidation removes + its values before any event for it is queued. A client that fetches because of an event therefore reads the + state the event is about. +9. **Bound.** Over `maxBytes`, the least recently used values that no order lease holds are evicted. An evicted + value is absent, its cells are unloaded in the next reply, and that reply's `want` loads it again. An + eviction sends no event: the clients that hold the value hold a correct one. +10. **An order use is leased, and refused when it cannot fit.** A sort or filter needs the level for every + candidate row at once. + - `want(keys, Order)` gives each key a lease until `orderLease` from now. A leased value is not evicted, so + a view that is ordering by the level does not lose rows to another query's loads, and two order uses + cannot evict each other in turn. Each query that needs the level for order renews the lease; a level no + query has ordered by for `orderLease` is evictable again. Leases are compared with the clock when a + request reads them; nothing is scheduled. + - The cache decides whether the use fits before it loads: the keys times the mean size of its stored values + (`expectedBytes` while it holds none), plus the bytes other leases hold, against `maxBytes`. Over it, + `want` returns `Unavailable`, the framework refuses the query naming the column, and nothing is loaded. + The second of two order uses that do not fit together is the one refused, while the first's lease lives. + - Sizes are not known before a load. When leased values come to exceed `maxBytes` while they load, the cache + stops loading for order, drops the lease and announces the table's record without keys; the next query + meets the estimate, which is now measured, and is refused. + - A `Display` load that finds nothing evictable is stored over the bound and is the first evicted. The + overshoot is at most one batch for each page being answered. +11. **A failed load is remembered.** A key the loader returned no value for is `Failed` in every snapshot until + it is loaded, and is announced once, with its key, so that clients stop showing it as loading. `want` does + not load a failed key again until `retryAfter` has passed since the failure, doubling with each failure in + a row up to `retryMax`; the delay is read from the clock when a query wants the key. `want(…, retry: true)`, + which a query carrying `TableQuery::retry` makes, loads the failed keys it names at once. An invalidation + clears the failure with the value. A failed key is not in `awaiting`, so a level that cannot be read does + not make every window refetch on every event. +12. **Snapshots are immutable, and nothing waits on a reader.** The values are held in 256 shards by key hash, + each an immutable map behind a shared pointer; a snapshot is the array of shard pointers. A store, a removal + and the observer call copy the shards they touch and swap them under a mutex held for the swap alone. A + reply may hold a snapshot for as long as its `apply` takes, 40 ms at 100,000 rows, and the server strand's + observer call still returns in the time its keys take. A value's recency and lease are atomics beside the + value and are no part of a snapshot's state. A whole-level invalidation swaps in empty shards. +13. **Threads.** `snapshot`, `want` and `invalidate` may be called from any thread. The loader runs on `loads`. + The observer call runs on the server strand and only removes entries: it does not load, wait or call out. + +The server learns of a cache through `RemoteServer::attach(std::shared_ptr)`, which the +application calls after it constructs the server. `attach` gives the observer its announce sink +(`effects::AnnounceSink`, a function taking a record name and optional keys, holding the server weakly), and +registers it for derived changes. `IEnvelopeService::bound` is not used for this: it runs at the first envelope a +service answers, which may be long after the first page, or never. + +### The view cache + +Sorting and filtering 100,000 rows takes 33 to 46 ms. Without a cache every request pays it: each page of a +scroll, each keyed fetch, and, while an order level loads, every watcher once per arrival. The server therefore +keeps the views it has computed. + +```cpp +struct ViewCacheOptions { + std::size_t maxBytes = 64U << 20U; // across the server's tables +}; +``` + +- **Key.** The table's record, `ServedTable::scope`, and the validated query's sort and filter in canonical + form. `page`, `columns`, `keys` and `distinct` are not in it: every page, every column set and every keyed + fetch of one sort and filter read one entry. +- **Valid while** the table's source has the snapshot the entry was computed from and each of the view's order + levels has the generation it read. A request finds an entry, compares those, and uses it or computes again. + An entry is never repaired in place. +- **Holds** the view's order as source rows (4 bytes a row), the undecided rows, `total`, `awaiting` and + `failed`, and, built at the first keyed query, the view row of each source row (4 bytes a row). A view of + 100,000 rows is 400 kB, or 800 kB once a keyed query has asked. +- **One computation.** Requests that need the same entry while it is computed wait for that one computation. So + fifty watchers of one view that refetch after an arrival cost one `apply`, not fifty. +- **Where it runs.** Finding, computing and reading an entry, and encoding the reply, run on + `RemoteServerOptions::tableWork`, over immutable snapshots. Neither the model's strand nor the server strand + waits for a sort. +- **Bound.** Over `maxBytes` the least recently used entries go. An entry is also dropped when it is found + invalid. +- **What it does not hold.** Cells: a reply reads them from the source and the level snapshots. Distinct + values: a values list runs the filter again without the column's own entry, and its reply is cached by the + client's query cache. +- A table served through `planned` has no entry: the store orders it. + +A scoped table whose `scope` is left empty would share views between callers who see different rows. The +framework cannot see that from the outside, so `table::rowScoped` and an empty `scope` together are refused +when the table is first served, as `SourceContract`. + +### The client's caches + +Both client caches hold pages until a change event marks them. Neither refetches because time passed. + +**The query cache** is told three things about a result that holds records, none specific to tables. The first +two a holder type declares, as a type declares `schemaMarker()`: + +```cpp +static constexpr auto openMembers() noexcept; // Page: "awaiting", "distinct" +static constexpr std::string_view partialMember() noexcept; // Page: "unloadedRows" +``` + +The result walk that records where an action's result holds each record also records these two members' paths, +and the catalog carries them as `ActionRecordInfo::open` (a list: the entry is open while any is set) and +`ActionRecordInfo::partial`. The cache reads the values at those paths and knows nothing else about the result's +shape. A member counts as set when it is a non-empty array, an object, `true`, or a number other than zero. The +third is `ActionRecordInfo::opaque`, which a registration states when a result holds a record at no path +([The table's record](#a-models-tables)). + +- **An open entry is marked by every change of its record.** While `open` is set, a row the entry does not hold + may come to belong to it, so a keyed event marks the entry whether or not it holds one of the keys. This is what + makes an undecided row that becomes admitted refresh the page it now belongs to. +- **A partial entry is not merged into.** While `partial` or `open` is set, an `updates` effect on the record acts + as `changes` for that entry: it is marked and fetched again. A merge replaces rows by key and cannot update a + sidecar indexed by position, so it would leave a merged row's unloaded state describing the row it replaced. The + cache applies the same downgrade to the effects of a discarded `latest` reply. +- **An opaque entry is always both.** The cache finds an entry's rows by the record's key member at the + record's paths, and a `GenericPage` has neither. Every entry of a `Table.` action (a page a document + query reads, a distinct-values list) is marked by every change of the table's record and is never merged + into. A table's description is an entry of `Table..describe`, which holds no record: it is replaced when + a reply names another `revision`. + +**A window's fetches are not queries and make no entry.** The session has a read for them and a way to watch +without an entry: + +```cpp +/// Runs a read action and resolves with its reply, or fails as an entry's fetch fails. Makes no cache entry +/// and reads none. A stop request sends the wire's `cancel` for the call, where the server takes one, and +/// drops the reply in either case. +async::Completion +Session::read(std::shared_ptr const& channel, std::string const& actionId, Value const& body, + std::stop_token stop = {}); + +/// Watches `record` for as long as the returned holder lives, and hands its events to `handler` on the +/// runtime's owner. Resolves when the server has the watch. +async::Completion +Session::watchRecord(std::string const& record, std::function handler); +``` + +- A watch is counted per holder, as it is per entry: the session's watch set names a record while any entry or + any holder needs it. +- **Watch, then fetch.** `WindowRows` sends its first page fetch after `watchRecord` resolves, as an entry + watches before it fetches, so no change between the reply and the watch is missed. +- `read` does not route its reply through the cache's effects: a read has none, and the caller's own change + events reach it like anyone's. +- On reconnect and on `resync` the session calls each holder's handler with a whole-type change. + +**Freshness.** + +- `staleTime` decides whether moving onto an entry fetches. Such a fetch re-reads a page and is answered from the + server's caches: it costs a page, never a load. A table with levels needs no `refreshEvery`, and one that + has it still loads nothing twice. A `PageWindow` has no stale time. +- On reconnect and on `resync` the session sends its watch set again and marks every entry stale; a window is + invalidated. Events missed while disconnected are covered by those fetches. +- Offline, a held page shows as it was, its unloaded cells unloaded. No arrival reaches it until the connection + returns, when the reconnect rule applies. + +**Column widths.** The Qt Quick table measures a column from its realized cells, and in a windowed table a +column's width only grows. An unloaded cell draws a placeholder of a fixed width, so a level arriving widens each +of its columns at most once, to its widest realized value, and never narrows one. + +### Which client cache a screen uses + +A `rows` table bound to a query's page reads through the query cache. With the rules above it is correct and +push-driven: an arrival marks the entries concerned, each fetches its page again, and the mount's keyed rows +repaint only the rows whose content changed. What it pays is the transfer: a page of 200 rows and 300 columns +for an arrival of twenty rows ([Performance](#performance), W7 against W4). + +A model-backed `table` reads through a `PageWindow` ([The client table model](#the-client-table-model)): the +node owns a window, feeds the mount's visible range to `visible`, sends the window's fetches through the +session, and gives the mount the window's rows. An arrival there transfers the rows that arrived. + +### A persistent tier + +Loaded values do not survive a server restart: the cache starts empty and loads on demand. A tier that keeps them +needs a durable generation per row, written with the data it describes, or a restart serves values an action +invalidated while the server was down. That is a storage design, not a cache option, and is not part of this one. +`LevelCache` keeps its storage behind its own members so that a tier can be put under it. + +## Where sort and filter run + +morph has two modes. In client mode an `Engine` sorts and filters a `DataSource` the client holds entirely. In +server mode a table's action takes a `TableQuery`, the server sorts, filters and pages, and the client holds +pages. A table over rows the document holds, and a collection, are in client mode; a model's table is in server +mode. + +A browser client is the Qt WebAssembly build, which is single-threaded. That decides three things: + +- **No worker.** An `Engine` there runs its jobs as steps on the owner thread, the one that paints. A step's + size is set for the browser, not for native code: the size that takes 4 ms natively is divided by the factor + B2 measures, so a step keeps to half a frame where the code is slower. The work is not done in parallel with + painting, and a recompute's diff and a `Deferred` repair's layout are one step each, 10 to 55 ms natively at + 100,000 rows ([`engine.md`, "Not here yet"](engine.md#not-here-yet)). Client mode in a browser is therefore + held to the bound of rule 4 below, 20,000 rows, and B1 holds the worst step at that size. +- **A 32-bit address space.** The heap cannot exceed 4 GiB and is usually configured lower. +- **Slower code.** WebAssembly runs this kind of code slower than native; the factor is measured by benchmark B2 + and taken as 1.5 to 3 until it is. + +**What the table weighs.** A `Cell` is as large as its largest alternative, two `Rational`s, about 56 bytes. A +`VectorSource` of 100,000 rows and 300 columns is 30 million cells: about 1.7 GB before any text. A page of rows +costs about 21 bytes a cell on the wire (212 bytes for a row of ten small cells), so the whole table is about +630 MB of JSON: over a minute at 100 Mbit/s before it is parsed. Holding the table in a browser is not slow, it +is impossible; and a table with a slow level is not held by the server in full either. A window of 1,600 rows is +480,000 cells and about 10 MB of JSON if every column is sent, and 64,000 cells and about 1.3 MB for the +forty columns in or near view, spread over eight replies. + +**The rule.** Where sort and filter run follows from what the client holds, and what the client holds follows +from the action's shape: + +| The source | Mode | Sort, filter, totals, distinct values, the undecided set | Selecting all | +|---|---|---|---| +| A model's table: its action takes a `TableQuery` (the schema carries `x-table`) | Server | On the server, once for each view and state of its data | Not offered on the client; an action over the query does it | +| Rows the client holds entirely: a document's `rows`, a collection's reply | Client | An `Engine` on the client | `Selection::selectAll` | + +A table is in server mode, and must be, when any of these holds: + +1. the server pages it, because the rows are too many to send; +2. a column that can be sorted or filtered has a level other than the base one: a client cannot order by values + it does not hold, and the level cache is on the server; +3. the rows a principal may see are decided per request; the server filters those rows in either mode, and client + mode is then possible only over the scoped result; +4. it has more than 20,000 rows or more than 1,000,000 cells. Under that, the whole table is at most about 21 MB + of JSON and 56 MB of cells, which a browser holds, and a three-key sort with a filter over 10,000 rows takes a + few milliseconds on one thread. + +The action's shape decides, not the application at run time and not the framework by measuring the data. A table +that changes mode when it grows past a threshold changes its text order (a server orders by its own collator and +a client does not re-sort a page), its selection behaviour and its totals on the day it grows. The author chooses +once by writing a paged action or a plain list. The framework checks the choice: a document table over a plain +list whose result exceeds the bound in 4 is reported as a warning naming the table, as the document's other +warnings are, and a client-mode sort or filter on a column with a level is refused. + +**What the user waits for** in server mode, at 100,000 rows and 300 columns, in a browser. The server figures are +measured natively on the engine; the rest are estimates until the named benchmarks run. + +| Action | Path | Time | +|---|---|---| +| Click a header to sort | request, `apply` (sort one key: 33 to 46 ms measured), encode a page, reply, decode, repaint | about 150 ms on a 50 ms round trip (B3) | +| Type in a text filter | debounce (250 ms), then request, `apply` (text filter: 9 ms measured), reply, decode, repaint | about 350 ms after the last key (B3) | +| Scroll into pages not held | request, a read of the cached view, reply | about 110 ms a page; the held margin of one page each side hides it at reading speed | + +A text filter is debounced through the query's `debounce`, so a word typed quickly is one request, not one per +key. The table keeps showing its rows, with the header marking the sort in flight, until the reply replaces them. + +**Hybrids.** Two were considered and one is kept for later. + +- *Sorting or filtering the held rows locally while the reply is in flight* is rejected. The window holds a slice + of the old order. Its local re-sort is not a slice of the new order, so the rows move once to a wrong place and + again to the right one, and a local filter shows a subset of 1,600 rows as if it were the subset of 100,000. +- *Local work when the window holds the whole view* (an `Exact` total no larger than the rows held, nothing + awaited) is sound: a filter that leaves 300 rows could be re-sorted with no request. It needs an `Engine` on the + client with the server's collator and a rule for handing back to the server when the view grows. It is not + specified here; the 150 ms above is the cost of leaving it out. + +## Configuring a table + +A user arranges a table: the order, visibility, pinning and widths of its columns, a chain of sort keys, and +filters set from each column's header. All of it is state of the client's table, in three values, each of which +a document may bind (then it is controlled, as `selection` is) or leave to the table: + +| Value | Shape | Held by | +|---|---|---| +| `layout` | `{"order": [id…], "hidden": [id…], "pinned": n, "widths": {id: units}}`, and where used `"groups"` ([Column groups](#column-groups)) and `"units"` ([Display](#display)) | `table::ColumnLayout` | +| `sort` | `[{"column", "direction"}…]`, most significant first | `TableRows::setSort` | +| `filters` | A `FilterSpec`: column filters and group filters | `TableRows::setFilter` | + +It works the same in client and server mode: layout reaches the server only as the columns a query names; sort +and filters go where the mode puts them. + +### Columns: order, visibility, pinning, width + +- **Reorder.** Dragging a header moves its column. With the header focused, `Ctrl+Shift+Left` and + `Ctrl+Shift+Right` move it, and the header's menu has "Move left" and "Move right". `Alt` with an arrow is + not used: in a browser it is Back and Forward. The selection column and a group heading are not columns and + do not move. +- **Hide and show.** The header's menu has "Hide column"; the table's menu opens a column chooser, a list of every + column with a check box, in order, with the same move actions. At least one column stays visible. +- **Pin.** The first `pinned` columns of the layout do not scroll sideways: they stay at the leading edge, and + are always in the column window. The header's menu has "Pin column", which moves the column to the end of + the pinned ones, and "Unpin". The selection column is always pinned and is not counted. Pinned columns may + take at most half the table's width; while the table is narrower than that allows, the last pinned columns + scroll with the rest, and the stored count is not changed. At 300 columns this is what keeps a row's name + beside whichever of its values the user has scrolled to. +- **Resize.** Dragging a header's trailing edge sets the column's width in units; a width the user set stays as + set, whatever the cells measure. Double-clicking the edge removes it, and the column takes its declared + width again. +- **Reset.** "Reset columns" restores the declared order, visibility, pinning and widths. + +`ColumnLayout` holds the order and the hidden set, with `setWidth(id, std::optional)`, `widths()`, +`setPinned(n)`, `pinned()`, and `value()` and `apply(value)` for the stored shape. + +**A layout survives a changed table.** Ids are the columns' string ids. One rule fits a layout to a table's +columns, whether the layout is a stored one (`apply`) or the table's own when its description takes a new +revision (`rebuild`): + +- an id the table does not have is dropped, from `order`, `hidden` and `widths` alike; +- a column the layout does not mention is shown, and placed after the nearest column before it in the declared + order that the layout does place, or first when there is none, so a new column appears where its author put + it and not at the end; +- `pinned` is clamped to the visible columns; +- if that leaves no column visible, the first declared column is shown. + +**Persistence.** The layout is part of the table's persisted state ([Persisted state](#persisted-state)). + +**Document vocabulary.** `layout` (the value; bound, it is controlled), `onLayoutChange` (after a user's change, +with `event` the new layout), `configurable` (a literal: `true`, the default, `false`, or a list of `reorder`, +`hide`, `pin`, `resize`), and on a column `locked` (a literal flag: the user cannot move, hide, pin, unpin or +resize it). `locked` is not called `fixed`: `fixed(n)` is a column's width sizing. Element values: `layout`, +`visibleColumns` (ids in order) and `hiddenCount`. Every key is optional. + +**With levels.** A hidden column that is neither sorted nor filtered is not in the column window, so no query +names it and its level is not loaded for it. Showing it fetches that column for the held pages. + +**With the column window.** Hiding a column returns its realized cells to their pool and creates none; moving +one creates none unless it moves into the window. + +### Sort chains + +The engine, `TableQuery` and the document's `sort` take a list of keys; what a user needs is a way to build one. + +- A plain click on a sortable header sorts by that column alone, ascending, or descending when it already is + the chain's first key, ascending. +- `Shift`+click adds the column as the last key, ascending; on a column already in the chain it flips that key's + direction and keeps its place. `Shift+Enter` and `Shift+Space` on a focused header do the same. +- The header's menu has "Sort ascending", "Sort descending", "Add to sort" and "Remove from sort", so the chain + can be built without a modifier key, on a touch screen included. +- The table's menu opens a sort dialog: the keys in order, each with its column, its direction, move up, move + down and remove, and "Add key". +- Every header in the chain shows its direction; with more than one key each also shows its place in the chain, + from 1. A header's accessible description says "sorted ascending, key 2 of 3". +- A chain has at most `QueryLimits::maxSortKeys` (8) keys in either mode. At the bound, adding a key is refused + with a message in the status line, not by dropping the first key. + +`sort` and `onSortChange` keep their shapes. `Table::sort` carries the whole chain to the backend. + +### Filters from the header + +A filterable column's header has a filter button, drawn filled while the column has a filter and named so for +assistive technology ("Name, filtered"). It opens a menu with three parts. + +1. **Values.** A search field over a list of the column's distinct values, each with a check box and the number + of rows that have it, under "(Select all)" and, when the column has empty cells, "(Blanks)". Typing narrows the + list; "(Select all)" then means the values the search shows. +2. **Conditions**, by the column's kind, each one `FilterEntry`: + + | Kind | Conditions | + |---|---| + | `Text` | equals, does not equal, contains, begins with, ends with; match case | + | `Integer`, `Key`, `Decimal`, `Quantity`, `Number` | =, ≠, <, ≤, >, ≥, between | + | `Date`, `DateTime` | the same, with a date picker; and the values list as a tree of years, months and days | + | `Bool` | the values list alone | + | `Custom` | is empty, is not empty | + + Two conditions may be joined by "and" or "or" (`combine`). +3. **Clear filter.** The table's menu has "Clear all filters". + +A column is filterable when it has an `id` and the document does not say `filterable: false`. `filters` is served +on every table, bound or held by the table, with `onFiltersChange`. + +**The value-set operator.** `FilterEntry::oneOf` is a list of operand texts: + +```cpp +struct FilterEntry { + // … the existing operators … + /// Matches a cell whose value equals any of these. Each is parsed for the column's kind, as `eq`'s operand is. + std::optional> oneOf; +}; +``` + +- It matches as `eq` does, against any member: text by fold unless `caseSensitive`, numbers exactly. It never + matches an empty or invalid cell, and is `Unknown` on an unloaded one. +- Ticked values are one `oneOf` entry in `include`; "(Blanks)" ticked adds an `isEmpty` entry beside it, which + `combine: any` joins. Unticking a few values of many is the same entry in `exclude`. A menu sends whichever side + is smaller, so unticking one value of 50,000 sends one value. +- `compileFilter` parses the members once into a set; a row is matched by one lookup, whatever the set's size. +- `QueryLimits::maxSetValues` (10,000) bounds one entry, `maxQuerySetValues` (20,000) the members of every set + of one query together, and `maxQuerySetBytes` (1 MiB) their text; over any of them is `LimitExceeded`. The + entry bound alone would allow a query of millions of values across its columns. When both the ticked and the + unticked side exceed it, the menu disables the list and says so: the user narrows with a condition. +- In a plan it is `FilterOp::OneOf` with `PlannedCondition::values`; a store that accepts `FilterSet` translates + it as a set membership test, and otherwise it is applied in C++ + ([What a store cannot express](#what-a-store-cannot-express)). `standardCorpus` adds a `oneOf` query for each + column, in include and in exclude. + +**Distinct values.** + +```cpp +struct DistinctRequest { + std::string column; + /// In place of `column`: a declared group's id, for values pooled over its columns. + std::string group; + /// Narrows to values whose text contains this, folded; empty for all. + std::string search; + /// For `Date` and `DateTime`: `Value`, or the bucket to count by. + DateBucket bucket = DateBucket::Value; // Value, Year, Month, Day + std::int64_t offset = 0; + std::int64_t limit = 200; +}; + +struct DistinctValue { + std::string text; // as the cell shows it + std::string operand; // as a filter entry takes it + std::int64_t count = 0; +}; + +struct DistinctValues { + std::vector values; // in the column's ascending order + std::int64_t distinct = 0; // values matching `search`, before offset and limit + TotalKind distinctKind = TotalKind::Exact; + std::int64_t blanks = 0; // rows whose cell is empty + std::int64_t invalid = 0; // rows whose cell is invalid; they are under no value + bool partial = false; // rows left out because a cell was unloaded +}; + +[[nodiscard]] std::expected +distinctValues(DataSource const& source, TableQuery const& query, DistinctRequest const& request, + ApplyOptions const& options = {}); +``` + +- **Which rows.** The rows that pass `query.filters` with the named column's own entry in `filters.columns` + removed. So each column's list shows what the other columns' filters leave, and a value the column's own filter + unticked stays in the list to be ticked again. Groups stay applied as they are. +- **Which values.** Valid cells, grouped by the key the column sorts and filters by: text that folds alike is one + value, shown as its first row's text. Empty cells are counted in `blanks`. Invalid cells are not values and + are counted in `invalid`; the menu says under the list how many cells could not be read, and offers nothing + to tick for them. +- **Unloaded.** A row whose cell in the column is unloaded, and a row another column's filter leaves undecided, is + in no count, and sets `partial`. The menu then says the list may grow. +- **Order and bound.** Values are in the column's ascending order. `limit` is clamped to `QueryLimits::maxLimit`; + `distinct` is counted up to `QueryLimits::totalCap` and is `AtLeast` at the cap. A column of 100,000 distinct + values is paged and searched, never sent whole. +- **Dates.** With a bucket, a value is a year, a month or a day, its `operand` the two ends of a `between`; the + menu draws the tree by asking for years, then the months of an opened year, then its days. +- **Levels.** The column's level is an `order` level of the request: it is needed for every candidate row. +- **Client mode.** `Engine::distinctValues(request)` answers from the key columns the engine holds. +- **Server mode.** `TableQuery::distinct` carries the request through the table's own action, so the model's + scope applies; the reply is a `Page` with no rows and `Page::distinct` set. A store-backed model pushes it down + as a grouped count or answers it over its candidates + ([What a store cannot express](#what-a-store-cannot-express)); where it refuses, the menu offers conditions + only. +- **Validation.** `validate` holds a distinct request to the query's rules. Exactly one of `column` and `group` + is set. The column, or every column of the group, exists and the caller may read it (`UnknownColumn`, + `ColumnNotReadable`): a values list is a read of the column. `search` is at most `maxTextLength` bytes and + `offset` at most `maxOffset`. A query that sets both `keys` and `distinct` is `InvalidSpec`. In the filter, a + group's `exclude` entries count toward `maxEntries` with its `include` and are bounded by `maxTextLength` as + they are, and so is every member of a `oneOf`. +- **Caching.** A distinct reply is an entry of the query cache, keyed by its body like any query, so reopening a + menu costs nothing. `Page` declares `distinct` as an open member beside `awaiting`: any change of the table's + record + marks the entry, because any row may change a count. + +Top-N filters are not included: "the ten largest" is a property of the view's order, not of a cell, and belongs +with a ranking function over the view. + +### Column groups + +Several columns can hold the same kind of thing: `companyA`, `companyB` and `companyC` each hold a company. A +**column group** names them, so one filter asks about all of them: "a company is F", "F and S are both there", +"at least two of them are F or S". The columns stay ordinary columns, shown, hidden, sorted and filtered one by +one as well. + +**Declaring a group.** A typed row declares its groups once, beside its levels: + +```cpp +struct ColumnGroup { std::string_view id; std::span columns; }; + +struct DesignRow { + static constexpr auto columnGroups() noexcept; // {"company", {"companyA", "companyB", "companyC"}} +}; +``` + +`DataSource::groups()` returns them (empty by default); the row's schema carries `x-group` on each member +property, and `deriveColumns` copies it. A document table over `rows` declares `groups` as parts +`{id, label, columns}`. A group has 2 to `QueryLimits::maxGroupColumns` (16) columns of one kind, and a column +is in at most one declared group; a declaration that breaks either is refused where the columns are read. + +A user may also form a group: the column chooser has "Group columns…", which offers columns of one kind and asks +for a name. Such a group is part of the layout (`"groups": [{"id", "label", "columns"}]` in its stored shape) and +is fitted to the table as the layout is: a column the table lacks leaves the group, and a group left with fewer +than two columns is dropped, with any filter on it. + +**The filter.** `GroupFilter` is the engine's filter over several columns, with a name, a quantifier, a mode +and excludes: + +```cpp +enum class GroupMode : std::uint8_t { + Columns, // count the columns whose cell satisfies the entries + Together, // every entry is found, each in a column of its own +}; + +struct ColumnCount { + enum class Kind : std::uint8_t { Any, All, AtLeast, Exactly } kind = Kind::Any; + std::int64_t n = 1; // for AtLeast and Exactly +}; + +struct GroupFilter { + std::string group; // a declared group's id; or + std::vector columns; // the columns themselves + GroupMode mode = GroupMode::Columns; + ColumnCount match; // Columns mode only + Combine combine = Combine::Any; // Columns mode: how one column's cell must satisfy `include` + std::vector include; + std::vector exclude; +}; +``` + +In JSON a group is `{"group" | "columns", "mode", "match", "combine", "include", "exclude"}`, with `match` one of +`"any"`, `"all"`, `{"atLeast": n}` and `{"exactly": n}`. A filter that sets only `columns`, `combine` and +`include` is the plain group: the entries match on any of the columns. + +Write k for the number of columns and say a column *satisfies* the entries when its cell matches any of them +(`combine: any`) or every one of them (`combine: all`), by the rules of a column filter: an empty cell matches +nothing but `isEmpty`, and an invalid cell nothing but `notEmpty`. + +- **`Columns` mode.** Count the columns that satisfy the entries. The row passes when the count is at least 1 + (`any`), k (`all`), at least n, or exactly n. +- **`Together` mode.** The row passes when the entries can be given a column each, no column twice, such that + each entry matches its column's cell. `match` and `combine` are not used. It takes at least 2 entries and at + most as many as the group has columns, so at most 16. +- **`exclude`.** The row is rejected when any column's cell matches any exclude entry: "none of the companies is + X". +- Groups combine with each other and with column filters by AND, so "F somewhere and S somewhere, in the same + column or not" is two `any` groups. + +The cases, as filters on the group `company`: + +| Wanted | Filter | +|---|---| +| At least one of the columns is F | `{"group": "company", "include": [{"eq": "F"}]}` | +| F and S are both there, in two of the columns | `{"group": "company", "mode": "together", "include": [{"eq": "F"}, {"eq": "S"}]}` | +| At least one of F, S, T in any of the columns | `{"group": "company", "include": [{"oneOf": ["F", "S", "T"]}]}` | +| At least two of the columns hold F or S | `{"group": "company", "match": {"atLeast": 2}, "include": [{"oneOf": ["F", "S"]}]}` | + +The second and the fourth differ, which is why both exist: + +| `companyA`, `companyB`, `companyC` | any is F | F and S together | at least 2 are F or S | exactly 1 is F | +|---|---|---|---|---| +| F, S, X | yes | yes | yes | yes | +| F, F, X | yes | **no** | **yes** | no | +| S, F, F | yes | yes | yes | no | +| F, X, Y | yes | no | no | yes | +| X, Y, Z | no | no | no | no | + +Entries are any `FilterEntry`, not only values: "two of the three dates are after 1 May" is `atLeast: 2` with a +`gt` entry. + +**Deciding `Together`.** It is a matching between entries and columns, found in C++ for each row; no store is +asked to express it. The engine reads each of the row's k cells once and gets, per cell, the set of entries it +matches (for value entries one lookup of the cell in a map from value to entries; a condition is tested per +entry), then looks for a matching that covers every entry by augmenting paths over those bit sets. With at most +16 entries and 16 columns one row costs at most 16 augmentations of at most 16 steps, and a row whose cells +match no entry is rejected after the k reads. + +**Unloaded cells.** An entry on an unloaded cell is `Unknown`, so a column may satisfy the entries, not satisfy +them, or be unknown. Write y for the columns that satisfy and q for the unknown ones. + +| Quantifier | Yes when | No when | Otherwise | +|---|---|---|---| +| `any` | y ≥ 1 | y + q = 0 | `Unknown` | +| `all` | y = k | y + q < k | `Unknown` | +| `atLeast n` | y ≥ n | y + q < n | `Unknown` | +| `exactly n` | y = n and q = 0 | y > n, or y + q < n | `Unknown` | +| `together` | a matching covers every entry using known matches only | none does even counting every unknown cell as matching every entry | `Unknown` | +| `exclude` | (rejects) a known cell matches | no cell matches and none is unknown | `Unknown` | + +So `atLeast 2` over three columns is `Yes` with two known matches and the third unloaded, `No` with no match and +one unloaded, and `Unknown` with one match and one unloaded. A row is decided as soon as its known cells decide it, +and invariant 1 holds for groups as for columns. + +**The menu.** A group has a filter entry of its own: in the table's menu, in each member column's header menu +("Filter Company, 3 columns…"), and, while its columns stand side by side in the layout, as a header over them +with its own filter button. Its menu is a column's, with two additions: + +- the values list is pooled: `DistinctRequest::group` names the group, a value's count is the rows that have it + in at least one of the group's columns, "(Blanks)" is the rows with an empty cell in at least one of the + group's columns, which are the rows ticking it admits (an `isEmpty` entry under `any`), and the rows + considered are those passing every other filter, this group's own removed. A row with an unloaded cell + in the group contributes its known cells and sets `partial`; +- a line under the list chooses how the ticked values must be found: "in any column" (the default), "in every + column", "in at least N columns", "in exactly N columns", or "all of them, each in a column". The first four + are one `oneOf` entry with a `match`; the last is one `eq` entry per ticked value in `together` mode, and is + offered while at least 2 values are ticked and no more than the group has columns. + +**On a server.** Over rows in memory, `apply` evaluates every form above. A model that pages in a store is +covered by the next section. + +### The status line + +A table has one line under its rows where it says what the rows alone cannot. It is part of the table, drawn +by each backend from `Table::status`, and it is where every refusal this document mentions is shown. + +| Part | Shows | Examples | +|---|---|---| +| Counts, at the leading edge | The view's size and what is still coming, from `TableRows::counts` | "9,093 rows"; "100,000 or more"; "12 so far, 40 loading"; "3 of 9,093 selected" | +| State | A level that failed, with "Retry"; a fetch that failed, with its error | "Recipe data could not be loaded. Retry" | +| Message, at the trailing edge | The latest refusal or notice, until the user's next gesture on the table or for six seconds | "A table sorts by at most 8 columns"; "Copied 200 rows, 12 cells were not loaded" | + +- "Retry" sends the table's query again with `TableQuery::retry` set, which makes the level cache load the + failed keys the reply needs at once ([The server level cache](#the-server-level-cache), rule 11). +- The line is a polite live region for assistive technology: a message is read out once, a count when it + settles. +- **When it shows.** A model-backed table has it always: its counts are not visible any other way. A table + over `rows` shows it while it has a state or a message, and otherwise takes no space. A document sets + `status: true` to show it always or `status: false` to never show it; with `false` a message goes to the + session's notifications instead, so a refusal is never silent. +- The counts are also element values (`viewCount`, `viewCountKind`, `undecided`, `selectedCount`, `awaiting`, + `failed`), so a document that hides the line can say the same things in its own words. + +### Copy + +`Ctrl+C`, and "Copy" in the table's menu, put the selection on the clipboard as text. + +- **What.** The selected rows, in view order, by the visible columns, in layout order: one line a row, cells + separated by tabs, each cell its display text as the column formats it, in the display unit the user chose. + A tab or a line break inside a cell's text is written as a space. A column's own `cell` template is not + copied; its value's text is. With no row selected, the focused cell's text is copied. "Copy with headers" + puts the columns' labels on a first line. +- **Rows and columns the client does not hold.** A server-mode table fetches them: keyed queries naming the + visible columns, `maxKeys` rows a request, for at most `SessionOptions::maxCopyRows` (10,000) rows. Over + that the copy is refused with a message in the status line. +- **Unloaded cells** are copied as empty, and the message says how many there were. Copying does not wait for + a level. +- The text reaches the clipboard through `Backend::copyText`. + +Writing a view to a file (CSV, a spreadsheet) is not part of the table: it is an action of the application +over the same query, which runs on the server and is not bound by what a client holds. + +### Persisted state + +A table's arrangement outlives the screen: its column order, hidden and pinned columns, widths and display +units, its sort chain, its filters, the groups a user formed, and its saved views. Every table stores and loads +it through one interface the application injects. Selection and scroll position are not arrangement and are +not stored. + +```cpp +/// Which table's state. No member names a person: a store keeps one person's state apart from another's itself. +struct TableStateKey { + std::string application; + std::string model; // the model's type; empty for a table over a document's rows + std::string table; // the table's id; for a `rows` table, the node's id + std::string qualifier; // the screen's id, or what the table's `persist` names + std::string part; // "state" or "views" +}; + +struct StoredTableState { + std::string json; + std::string revision; // names this stored document; opaque +}; + +enum class SaveOutcome : std::uint8_t { Saved, Conflict, Refused }; +struct SaveResult { + SaveOutcome outcome = SaveOutcome::Saved; + std::string revision; // of what is stored now + std::string reason; // for Refused +}; + +class ITableStateStore { +public: + virtual ~ITableStateStore() = default; + /// The stored document, or nothing when none is stored. + virtual async::Completion> load(TableStateKey const& key) = 0; + /// Stores `json`. With `expected` empty it replaces whatever is stored; otherwise it stores only when the + /// stored document has that revision, and answers `Conflict` with the current revision when it has not. + virtual async::Completion save(TableStateKey const& key, std::string json, + std::string expected = {}) = 0; + virtual async::Completion remove(TableStateKey const& key) = 0; +}; +``` + +- **One place.** The store is `SessionOptions::tableState`, a `std::shared_ptr`. The session is + what every table surface reaches: a document table over rows, a model-backed table, a collection, in client + and in server mode. A table mounted from C++ with no session is given the store through its `TableRows`. With + no store, a table keeps its state for as long as it is mounted. +- **Asynchronous.** Every call returns a completion; nothing waits on the thread that paints. A store delivers + its completions on an executor it owns, as a preferences store does. +- **The person is the store's.** A key names no person, because a client must not be able to name someone + else's. The framework's default store derives the person on the server from the verified session; an + application's own store scopes by whoever it knows is signed in. +- **What a key distinguishes.** Tables of two instances of one model type share a key and so an arrangement: + the state belongs to the kind of table on a screen, not to the data it shows. A table over `rows` needs an + `id` on its node to have a key; without one it persists nothing, and a `persist` that says otherwise is + refused when the document loads. +- **No size limit.** A state is as large as the arrangement it holds: a filter that ticks 20,000 values is + stored whole, and a table keeps as many views as the user saves. Neither the interface nor the framework + bounds a document, and a store may not truncate one. An operator who wants a ceiling sets a quota on the + server ([The framework's stores](#the-frameworks-stores)); a save over a quota is `Refused`, and the table + says so in its status line and keeps its arrangement. + +**What is stored.** Two JSON documents per table, each with a version: + +```json +{ "version": 1, + "layout": { "order": ["id", "name"], "hidden": ["notes"], "pinned": 1, "widths": {"name": 24}, + "groups": [{"id": "g1", "label": "Company", "columns": ["companyA", "companyB"]}], + "units": {"weight": {"mass": "g"}} }, + "sort": [{"column": "name", "direction": "asc"}], + "filters": { "columns": {}, "groups": [] }, + "view": "Open orders" } +``` + +```json +{ "version": 1, "default": "Open orders", + "views": [{ "name": "Open orders", "layout": {}, "sort": [], "filters": {} }] } +``` + +- `part: "state"` is the arrangement in use; `part: "views"` is the saved views. +- `view` names the saved view the arrangement was taken from, or is absent. The menu shows the name, and shows + it as modified when the arrangement differs from that view; nothing more is stored for it. +- **Versions.** A document of an older version is brought forward by the framework, one version at a time, when + it is read. One of a version the framework does not know is not used and is reported; the table starts from + its defaults, and writes its own version when the user next changes something. + +**Fitting stored state to the table.** State is fitted to the description the caller receives, never the other +way round, and a table is never refused over its state: + +- columns are fitted as a layout is ([Columns](#columns-order-visibility-pinning-width)); +- a sort key on a column the table lacks, or that is not sortable, is dropped; +- a filter on a column or group the table lacks, one that does not compile for the column's kind, and a unit + the description does not list are dropped, each reported once at the table's path; +- a user-formed group that is left with fewer than two columns is dropped with its filter; +- what was dropped because the caller's description lacks the column or group is **carried**: it is kept + beside the fitted state and written back as it was, a carried column keeping its place after the column it + followed. A table whose columns depend on the session therefore keeps a user's place for a column that is + absent now and returns later. What was dropped for any other reason is gone at the next save. + +**When it is loaded.** A table loads its state before it shows rows. Its `TableRows` starts in a restoring +state: a server-mode table sends no query, and a client-mode table draws placeholder rows. When the load +completes it applies the fitted state and then makes its first query, so the first rows are in the stored order +and under the stored filters, with no rows of the default order shown first and no second fetch. + +- The session keeps every state it has loaded or saved, by key, for its lifetime. A table mounted a second + time is restored from it at once, with no request; the first mount of a screen is the only one that waits. +- A load that has not completed after `SessionOptions::tableStateTimeout` (300 ms) is given up on for the first + query: the table starts from its defaults. If the state arrives later it is applied, at the cost of one more + query, unless the user has touched the table meanwhile (scrolled it, selected in it or changed its + arrangement), in which case it is kept for the next mount and not applied under the user's hands. +- A load that fails is reported once through the session and the table starts from its defaults. Its changes are + still saved. + +**When it is saved.** A change marks the state unsaved; a save goes out when the arrangement has not changed for +`SessionOptions::tableStateDebounce` (500 ms), and at once when the table unmounts. One save per key is in +flight; a change made during it is one more save after it. A failed or refused save is reported in the status +line and the table keeps the arrangement. A client that is closed loses at most the changes of the last +debounce. + +**Two windows, two devices.** The two parts differ. + +- `state` is saved with no expected revision: the last save wins, whole document. Nothing is merged: an + arrangement is a choice, and half of one window's filter joined to half of another's is neither's. +- `views` is a list, and losing a view another window saved is losing work. A change to it is one operation + (add, replace, rename or remove one view, or set the default). The table saves with the revision it loaded; + on `Conflict` it loads again, applies its one operation to what is stored now, and saves once more. +- A mounted table does not follow saves made elsewhere; it reads its state when it mounts. + +#### The framework's stores + +| Store | Keeps state in | Use | +|---|---|---| +| `PreferencesTableStateStore` | The application's `IPreferencesStore`, through the server's `morph.tableState` model, the person derived on the server | The default. A host that has a preferences store gives every table persistence with no further wiring. | +| `MemoryTableStateStore` | A map | Tests, and a host that wants state for a session only | +| A host's own | Anything: a settings file, a browser's local storage for a build with no server | The interface is all a host implements | + +**`PreferencesTableStateStore` has its own path to the preferences store.** It does not go through the +preferences model, whose `maxValueBytes` (16 KiB) bounds a preference and would bound a table's state. The +client half calls a model of its own, `morph.tableState`, through the interpreter's `Dispatcher`, so it travels +the same bridge and session as the screen's models; the server half, `TableStateModel`, reads the person from +the verified session exactly as the preferences model does, refuses a caller with no verified person, and +writes to the same `IPreferencesStore` the application already has. Table state therefore has one home, the +application's preferences storage, and no second database. + +How it keeps a document of any size in a store built for small values: + +- **Where.** Entries of the caller's person and the key's `application`, under the scope + `morph.table.`, the digest being of the key's `model`, `table` and `qualifier`. A digest keeps the + scope inside a preference name's length whatever the ids are; the head entry repeats the key in clear for + whoever reads the store. +- **Small documents.** A document of at most `TableStateOptions::segmentBytes` (12 KiB) is one entry at the + point ``: `{"revision": …, "document": …}`. Nearly every state is this. +- **Large documents** are written as segments. Each segment is an entry at the point + `..` whose value is a JSON string holding a slice of the document's text, so every + entry is valid JSON, as a preferences store expects, and no longer than `segmentBytes`, which is inside the + size such stores are built for. The head entry at `` is `{"revision": …, "segments": n, "bytes": b}`. +- **Order of writes.** The segments of a new revision first, then the head, then the removal of the previous + revision's segments. A reader takes the head and then the segments it names, so it reads either the old + document or the new one and never a mixture. A save that stops halfway leaves segments no head names; the + next save of that key removes them. A head whose segment is missing is a failed load. +- **`expected`.** The model compares it with the head's revision before it writes. A preferences store has no + conditional write, so two saves that reach the server within the same moment can both pass the comparison, + and the later then wins. That window is the store's call time, not a user's think time, and it is the only + case in which a view saved in one window is lost to another; a host store with a conditional write closes + it. +- **Transport.** A call is bounded by an envelope (8 MiB). `save` carries a document of up to 1 MiB; a larger + one is sent as parts of 1 MiB under one upload id and committed when the last arrives. `load` returns the + first 1 MiB and the document's revision and size, and the client asks for the rest by offset, starting again + if the revision changes meanwhile. Loads made in one turn of the owner go out as one call naming all their + keys, so a screen of five tables waits for one round trip. +- **Bounds.** Each key member is 1 to 256 bytes. `TableStateOptions::maxBytesPerPerson` is the operator's + quota across one person's stored state; it is unset by default, and then nothing limits a state. + +A document's `columns` and `saved` preference points that name a table hold no value of their own: their +editors read and write the table's state. + +**Document vocabulary.** `persist` is a literal on a table: `true` (the default), `false` (nothing is loaded or +saved), or a string, which becomes the key's qualifier in place of the screen's id, so the same table on two +screens shares one arrangement, or two tables of one screen keep theirs apart. + +**Bound values.** A `layout`, `sort` or `filters` the document binds is the document's: the stored value of that +part is neither applied nor saved, and the rest of the state persists as usual. A literal `sort` is the table's +default, and a stored one replaces it. + +### Saved views + +A saved view is a name with a `layout`, a `sort` and `filters`. A table's views are the `views` part of its +persisted state; the table's menu lists them, applies one, saves the current arrangement under a name, renames +and removes one, and marks one as the default. Applying one sets the three values, through their handlers where +they are bound, and records its name in the state's `view`. The view the `default` member names is applied +when the table mounts and no `state` is stored for it. A stored view is fitted to the table as its state is. + +## Values with units + +A column can hold amounts in different units, and of different kinds of unit: 0.3 kg in one row, 200 g in the +next, 0.3 % in a third. Such a column sorts and filters by what the values mean: 200 g is less than 0.3 kg, and +neither is comparable with 0.3 %. + +A `Quantity` column stays what it is: one dimension, every cell converted to the column's canonical unit by the +factor the cell carries. `Measure` is the kind for a column whose cells name their unit. + +### Measures and the unit registry + +```cpp +/// A unit known to a table: which dimension it measures and how it converts to that dimension's base unit. +struct UnitInfo { + std::string id; // "kg", "g", "percent"; what a cell and an operand name + std::string symbol; // "kg", "g", "%"; what is shown + std::string dimension; // "mass", "ratio" + math::Rational toBase; // exact: 1 for kg, 1/1000 for g, 1/100 for percent + std::uint32_t decimals; // places shown in this unit +}; + +using UnitId = std::uint32_t; + +/// The units of a process, immutable once built and safe to read from any thread. +class UnitRegistry { +public: + [[nodiscard]] std::optional find(std::string_view id) const; + [[nodiscard]] UnitInfo const& info(UnitId unit) const; + /// Dimensions in the order they were first registered. + [[nodiscard]] std::span dimensions() const; + /// The units of one dimension, largest base factor first. + [[nodiscard]] std::span unitsOf(std::string_view dimension) const; +}; + +/// An exact amount in a named unit. +struct Measure { + math::Rational amount; + UnitId unit = 0; +}; +``` + +- `Cell` has `Measure`, its last alternative, after `Unloaded`. It is smaller than `QuantityCell`, so a `Cell` + does not grow. +- `ColumnKind` has `Measure` (`"measure"`). `ColumnInfo::dimensions` is the dimensions the column allows, + in the order their groups sort; empty allows every dimension, in the registry's order. +- `Services::units` is a `std::shared_ptr`. A `Measure` column with no registry is a + `SourceContract` error. +- **The application fills the registry at run time**, on the server, from its own configuration or data: which + units exist is the application's knowledge, and may be a table in its database. `UnitRegistryBuilder` takes + `UnitInfo`s and builds an immutable registry; `addUnitsOf()` adds the units of one of morph's compile-time + unit enums, as a convenience for an application that has them. +- **A client learns units from the table's description** (`TableDescription::units`, the units of the + dimensions the table's columns allow), and from nowhere else: it has no registry of its own. +- **A unit's id** is 1 to 32 bytes, holds no white space, and does not begin with a digit, a sign, a point or + a comma; the builder refuses any other, and a second unit of the same id. An operand is a number, a space and + an id ([Filters on values with units](#filters-on-values-with-units)), so an id with a space in it, or one + that begins like a number, could not be told from the amount. The ratio's unit is the one empty id. +- **A `UnitId` is stable within a lineage.** The number is a place in one registry, and cells that hold it + outlive that registry in a level cache. So a registry is changed by building its successor from it: + `UnitRegistryBuilder{previous}` keeps every unit's number, gives a new unit the next one, and leaves a + removed unit's number as a tombstone that `find` does not return and that is never given out again. A + registry and its successors share a `lineage()`. A cell holding a removed unit's number is an invalid cell. +- **Changing the units.** The application builds the successor and swaps it in; requests in flight finish on + the one they started with. Every description that lists units takes a new `revision`, so each client fetches + the description again and resets its window, which re-reads its rows and redraws them. Key columns built + with the previous registry are dropped with the snapshot they belong to, and the view cache's entries with + them. A level cache holds amounts and unit numbers, not base amounts, and remembers the lineage they are + of: under a successor its values stay, and under a registry of another lineage, built from nothing, the + cache invalidates the level when it is next read, because its numbers mean nothing there. A stored display + unit or filter operand names a unit by its string id, and one naming a removed unit is dropped when it is + applied. +- Amounts and factors are `Rational`s. A value's key is its amount times its unit's factor, with checked + arithmetic: a product that does not fit is an invalid cell, as for a quantity. No value passes through a + double. +- **A plain number is a ratio.** The registry always holds the unit `""` of the dimension `ratio` with factor 1, + and `percent` is a ratio with factor 1/100. So 0.3 % is 0.003, equals a plain 0.003, and sorts among plain + numbers. Percent means a hundredth, and treating it as a dimension of its own would make 50 % and 0.5 two + values. A count of things is not a ratio: it is given a unit of its own (`pcs`). +- On the wire a unit is always its string id, never its number. A generic page writes a measure as + `[numerator, denominator, unit id]` ([Rows on the wire](#rows-on-the-wire)). A typed row's member is a + `table::MeasureValue`, written `{"num", "den", "unit"}` as a `Rational` is with the unit's id beside it. + `RowsSource` resolves the id through the registry it is given; a document's row value of that shape is read + the same way by `ValueRowsSource`. The row's schema marks + the property `x-measure` and lists the column's units as `x-units` (`{id, symbol, dimension, num, den, + decimals}`), so a client parses, converts and shows without asking. + +### Order of values with units + +A value's key is a pair: its dimension's place among the column's dimensions, and its amount in the dimension's +base unit. + +- Values of one dimension are ordered by base amount, whatever their units: 0.2 kg, 300 g, 0.5 kg. +- Dimensions form groups, in the column's declared order or the registry's. The sort direction reverses the + amounts inside each group and leaves the groups where they are: a descending sort shows 0.5 kg, 300 g, 0.2 kg, + then 0.3 %, 0.2 %. Comparing kilograms with percent means nothing, so the direction has nothing to reverse + between them, and a group that stays put is easier to find; empty cells keep their place for the same reason. +- A cell whose unit the registry does not hold, or whose dimension the column does not allow, is invalid. Valid + cells come first, then empty and invalid, then unloaded, in either direction, as in every column. An unloaded cell + has no dimension. +- Ties break by source row. Equal amounts in different units (0.3 kg and 300 g) tie. + +### Filters on values with units + +An operand names its unit: `"250 g"`, `"0.3 kg"`, `"0.5 percent"`. On the wire it is a decimal with a point, a +space and a unit id, whatever the user's locale; the client turns what was typed into that form. + +- **Comparisons** (`lt`, `le`, `gt`, `ge`, `between`) and `eq` match a cell of the operand's dimension by base + amount, and are `No` on a cell of another dimension: "more than 250 g" keeps 0.3 kg and drops 200 g and 0.3 %. + They are never `Unknown` for a loaded cell. +- **`ne`** matches every valid cell that is not equal to the operand, a cell of another dimension included: + 0.3 % is not 250 g. +- **Equality is by amount.** `eq "300 g"` matches 0.3 kg. `oneOf` is a set of such keys. +- **`between`** takes two operands of one dimension, in any of its units; of two dimensions it is `InvalidValue`. +- **An operand with no unit is a ratio.** `"0.003"` is the plain number, which equals 0.3 %. On a column that + does not allow ratios it is `InvalidValue`: the engine does not guess a unit. A menu always sends a unit, + because it makes the user choose one. +- An operand whose unit the registry lacks, or whose dimension the column does not allow, is `InvalidValue` + naming the column. +- A group of `Measure` columns filters as any group does; every form of [Column groups](#column-groups) applies. +- An unloaded cell is `Unknown`, as everywhere. + +**What a user types.** `ui::parseMeasure(text, units, locale)` reads "0.3kg", "0.3 kg", "300 g" and, in a locale +whose decimal separator is a comma, "0,3 kg": the locale's number, optional space, then a unit's symbol or id, +matched exactly and then without regard to case. Text with no unit takes the unit the menu shows as chosen. + +### The values list and the menu + +`distinctValues` on a `Measure` column groups by key, so 0.3 kg and 300 g are one value with one count, in key +order: the list comes out grouped by dimension, each group in order of amount. `DistinctValue` has `dimension` +and `unit`: the dimension's name, and the unit of the first row that had the value. The menu draws a heading per +dimension, and its conditions take a number and a unit picker offering the units of the dimensions the column +holds. + +### Display + +A cell is shown in its own unit, to that unit's decimals. A user may choose a display unit per dimension for a +column, from the header menu; the layout's stored shape holds it as `"units": {column: {dimension: unit}}`. The +shown amount is then the exact conversion, rounded half to even to the display unit's decimals for showing only; +order and filters always use the exact amount. A `Measure` column aligns to the end with digits of one width, as +an `end` column does, with the symbol after the number, so amounts in one unit line up. Changing the display +unit changes the cells' text and may widen the column once. + +### Units on a server + +Over rows in memory `apply` does all of it. A store that keeps a base amount and a dimension beside each value +can order and filter by them: it accepts `translatable("", PlanUse::Measure)`, and a condition reaches it as the +operand's dimension and base amount (`PlannedCondition::dimension`, and `value` as a `Rational`). A store that +does not is answered through a prefilter ([What a store cannot express](#what-a-store-cannot-express)). +`standardCorpus` adds, for a `Measure` column, operands in a unit other than the cell's and of another +dimension. + +## What a store cannot express + +A store-backed model answers a query in two parts: the store narrows, and `apply` decides. `planQuery` pushes +down what the store can express; when something is left over, the model fetches the rows the pushed part admits +and the server runs the whole query over them in C++. A model that allows this (`Residual::Allow`) refuses +nothing merely because its store cannot say it. + +```cpp +enum class Residual : std::uint8_t { + Refuse, // a query the store cannot express whole is `Unavailable` + Allow, // it is answered by a prefilter and `finishPlan` +}; + +[[nodiscard]] std::expected +planQuery(TableQuery const& query, std::span columns, QueryLimits const& limits = {}, + std::function const& mayRead = {}, Services const& services = {}, + std::function const& translatable = {}, + Residual residual = Residual::Refuse); + +struct QueryPlan { + // … sort, filters, groups, offset, limit, totalCap, keys … + /// The plan is a prefilter: its rows are a superset of the view, unordered and unpaged, and `finishPlan` must + /// be called on them. + bool prefilter = false; +}; + +/// Over the candidates as a `DataSource`: what the framework calls with the rows a `PlannedTable` returned. +[[nodiscard]] std::expected +finishPlan(QueryPlan const& plan, TableQuery const& query, DataSource const& candidates, + ApplyOptions const& options = {}); + +/// The typed convenience: the rows through a `RowsSource`, the result copied into a `Page`. +template +[[nodiscard]] std::expected, TableError> +finishPlan(QueryPlan const& plan, TableQuery const& query, std::vector const& candidates, RowKey key, + ApplyOptions const& options = {}, /* kinds, comparators, loaded as for apply */ ...); +``` + +- **What `translatable` means.** It says what the store can push down: `translatable(column, use)` for `Sort` + and `Filter`, and for the uses a translator written for those two would not recognise (`Keys`, `Distinct`, + `FilterSet` for a `oneOf`, `GroupCount` for a `match` other than `any` or a group `exclude`, `Measure` for a + sort or filter on a column of values with units) a first question + with no column, `translatable("", use)`. A predicate that only looks at column names refuses an empty one, and + an empty predicate accepts none of them. +- **A whole plan.** When the store accepts every part of the query, the plan is as + [`database_pages.md`](database_pages.md) specifies: the store sorts, filters, pages and counts. +- **A prefilter plan.** When any part is not accepted and `residual` is `Allow`, the plan holds only the column + filters and `any` groups the store accepted, has no sort, `offset` 0 and `limit` of + `QueryLimits::maxCandidates + 1`, and `prefilter` is set. For a `together` group the plan adds a necessary + condition the store does know: one `any` group per entry, since every entry must match some column. The model + runs it and the rows go to `finishPlan`, called by the framework for a `PlannedTable`, which is `apply` over + them with the whole query: the parts the + store already applied are applied again, which changes nothing, so the answer is the oracle's by construction. + Sort, paging, totals, undecided rows, keyed replies and distinct values all come from that one call. +- **The bound.** `QueryLimits::maxCandidates` (100,000, the total cap) bounds the rows a prefilter may return. + `finishPlan` given more refuses with `Unavailable`, whose message asks for a narrower filter; a model may count + the prefilter first and refuse without fetching. Under the bound the total is exact, because every candidate + was seen. +- **Cost.** Each request reads the candidates from the store and runs `apply` over them: at 100,000 candidates + that is the store's read of 100,000 rows plus the figures of S1, S2 and G1 to G4, tens of milliseconds for the + C++ part (X1, measured over the fixture's rows in memory). The default bound is the size at which that part + stays within S1 and G4. A store's own read of the candidates comes on top, so a store-backed model whose reads + are slow sets a lower `maxCandidates`. A deep page costs what the first does; nothing is kept between requests. +- **When a model still refuses.** When its rows are costly to materialise by the hundred thousand, it keeps + `Residual::Refuse`, or lowers `maxCandidates`, and the menu does not offer what it refuses. +- **Distinct values** follow the same rule and meet the bound sooner: the column's own filter is removed, so with + no other filter the candidates are the whole table. A values list over more than `maxCandidates` rows is + refused and the menu offers conditions only; a store that can group pushes `Distinct` down instead. +- **A translator that knows only whole plans is never wrong.** `residual` defaults to `Refuse`, so a model that + does not ask for prefilters receives either a plan made only of what its predicate accepted, or `Unavailable`. + It never receives a + prefilter plan, and so never returns a superset as an answer. +- **Conformance.** `conformanceFailures` holds a model to `apply` whichever way it answered. For a prefilter + plan it also checks that the candidates contain every row of the oracle's view + (`ConformanceOptions::candidates`, the keys of the rows the model's prefilter returned), and it reports, per + use, how many queries + were pushed down and how many finished in C++, so a model that pushes nothing down is seen. + +## Performance + +The target is a table of 100,000 rows and 300 columns that stays fluent on the screen. Every layer has a named +benchmark, a budget, and the regression it exists to catch. Budgets marked **M** come from figures measured on +this code ([`engine.md`, "Measurements"](engine.md#measurements)); those marked **P** are provisional and are +replaced by the first recorded baseline plus its spread. + +**The fixture.** Every measurement here, at every size, runs against one pure C++ model: no database, no SQL +and no store translator is in any benchmark's path. + +- `bench::WideTableModel`, in `tests/table/wide_table_model.hpp` and `.cpp`, built as the static library + `morph_wide_table_model` and linked by the table, UI, net and Qt Quick test executables, so the server, wire, + cache, end-to-end and frontend cases all drive the same model. +- It is a registered model type that serves three tables through `BRIDGE_REGISTER_TABLE`: `wide`, the + 100,000 × 300 table; `small`, 1,000 × 30 over the same generator; and `scoped`, whose description adds and + removes columns by the caller's principal. `wide` and `small` share the rows they have in common and one level + cache. `EditWideRow` changes one cell, declares `updatesTable` on `wide` and on `small`, and returns a + `ChangedRows` for each; `ReplaceWideRows` regenerates the rows and declares `changesTable`; `EditWideRecipe` + changes a record the slow level depends on. + Together they exercise descriptions, pages, keyed replies, distinct values, effects, invalidation and + arrivals, and a model with more than one table. +- Its rows are a column store: one typed array per column, generated from `WideTableOptions{rows, columns, + seed}`, read by a `DataSource` whose snapshot hands the arrays out in runs. The same seed gives the same table + on every machine. There is no row type anywhere in it: the description is built from the options, and a reply + is the generic page written from the arrays. +- It is built in the order the design lands ([Delivery order](#delivery-order)): its rows and generator + first, as `bench::WideTableSource`, which needs nothing but a `DataSource`; registered as a model with its + three tables when the generic contract lands; its slow level with the level cache. A benchmark is recorded + from the step that lands what it measures. No baseline is taken of a thing that does not exist. +- Its slow level is a `LevelCache` whose loader waits a configured time per key (`WideTableOptions::loadDelay`) + or on a gate a test holds, so a load's duration is known and can be subtracted, held or released. + +**The data.** Rows × columns of 1,000, 10,000 and 100,000 by 30 and 300. Per 300 columns: 100 integers, 60 +decimals, 30 quantities in two units, 10 values with units over three dimensions and ten units, 70 texts with +accents and mixed case, 69 of 4 +to 60 characters and one of 200 to 2,000, 20 dates and date-times, 10 booleans; one column in twelve has 10% +empty cells. Three of the text columns form one column group and sixteen another. Twenty columns (ten decimals, +ten texts) are at the level `recipe`, loaded for 70% of the rows unless a benchmark says otherwise. + +**How figures are taken.** Release (`-O3`), the same compiler for a comparison, median and 10th to 90th +percentile over at least seven runs at 100,000 rows, the machine's load average recorded beside each table as +`engine.md` records it. The benchmark machine also hosts CI runners, so a timing is never a CI gate: the gates +are the counts (cells read, loader calls, requests sent, widgets realized, bytes), which are exact and run as +ordinary tests. Timings are run by tag, `"[.benchmark]"`, and written into this section. Each benchmark first +asserts that its feature was active (unloaded cells present, the cache hit, the page at 300 columns), so a figure +is never taken over a path the feature did not reach. + +### Benchmarks: server + +`morph_table_tests "[.benchmark]"`, beside the existing cases. + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| S1 | `apply`, sort one numeric key at the `recipe` level, page of 200, 100,000 × 300, the level read through its snapshot | ≤ 48 ms (M: 33.0 ms with all cells loaded, and L1's 8 ms for the level's 100,000 reads); ≤ 1.10 × the same query with no unloaded cell | The unloaded rank slowing every comparison | +| S2 | `apply`, text `contains` on a `recipe` column | ≤ 12 ms (M: 9.25 ms); ≤ 1.15 × the two-valued filter | Three-valued evaluation costing more than a state test | +| S3 | Cells read by `apply`, per column, through a recording source | Exactly the sorted and filtered columns; zero for the other 298 (gate) | `apply` building keys, or reading unloaded state, for every column | +| S4 | `levelsNeeded`, 300 columns, 8 sort keys, 32 filtered columns | ≤ 50 µs (P) | A lookup that scans the columns per key | +| S5 | Engine repair when a level arrives for 200 and for 1,000 rows of a sorted 100,000-row view | ≤ 15 ms (P) and ≤ 35 ms (M: 27.4 ms for 1,000 updates); one repair and no full sort on `EngineStats`, and the arrived rows at the view rows the oracle gives them (gates) | An arrival recomputing the view; an arrival that changes nothing | +| S6 | Repair with 50,000 undecided rows when 1,000 are decided | ≤ 40 ms (P); one repair, no filter run over all rows, and each decided row in the view or rejected as the oracle has it (gates) | The undecided set rebuilt per arrival; a decided row left undecided | +| S7 | Ten consecutive page requests of one sorted view | One view computed, by `ViewCacheStats` (gate); requests two to ten ≤ 2 ms each (P) | A request that sorts again | +| S8 | Resident memory of the source at 100,000 × 300 | Recorded; the fixture's column store, against a `VectorSource` of the same rows, expected near 1.7 GB | A server holding a wide table as cells | +| S9 | A keyed query of 20 keys against that view: the first, which builds the view rows by source row, and the next | ≤ 3 ms and ≤ 0.5 ms (P); no view computed (gate) | A keyed fetch that sorts | + +### Benchmarks: server level cache + +`morph_table_tests "[.benchmark]"`, with a loader that sleeps a fixed time per key so that load time is known and +subtracted. + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| L1 | The hit path of one request: a snapshot and 200 reads (a page), and 100,000 reads (an `order` level) | ≤ 20 µs and ≤ 8 ms (P) | A copy of the cache per request; a lock per read | +| L2 | Bytes per cached row at twenty columns, and for 100,000 rows | ≤ 1.5 kB a row, ≤ 150 MB (P; twenty cells are about 1.1 kB) | Per-value allocation overhead | +| L3 | `invalidate` of 1,000 keys, and the observer call for a 4,096-key change, which runs on the server strand; the same while a reader holds a snapshot for 40 ms | ≤ 0.5 ms and ≤ 1 ms (P), with or without the reader (gate: the call does not wait for it) | Invalidation that scans the cache, or waits for a reply being built | +| L4 | Loader calls when 8 threads want the same 200 keys | 200 keys loaded, once (gate) | No in-flight set | +| L5 | Loader calls for a second query and a second client | Zero (gate) | A miss on a held key | +| L6 | The cache's own time from loader return to announce, per batch of 64, at 100,000 held rows | ≤ 1 ms (P); shards copied ≤ keys stored (gate) | Storing under a lock held across the announce; copying the cache per batch | +| L7 | Two order uses of one level whose candidates do not fit `maxBytes` together | The first loads its candidates once; the second is refused with `Unavailable`; loader calls equal the first's candidates (gates) | Two sorts evicting each other's rows in turn | +| L8 | Ten page requests over rows whose loads failed, inside `retryAfter`; then one with `retry` | Zero loader calls, `awaiting` empty and `failed` set; then one load of the page's keys (gates) | A failed level wanted again by every request | +| L9 | Bytes counted for 1,000 values of twenty cells with texts, against the allocator's figure | Within 10% (gate) | A bound that counts `sizeof` and never evicts | + +### Benchmarks: many clients + +`tests/net`, one server and fifty clients in one process, each with a window on the fixture's `wide`. + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| C1 | Fifty watchers of one sorted view while its order level loads for 100,000 rows in batches | Views computed ≤ level generations read, never one per watcher (gate); refetches per watcher ≤ 4 a second (gate); the server strand's longest task ≤ 2 ms (P) | A sort per watcher per arrival; a strand that sorts | +| C2 | The thread each request's sort runs on, and the model strand's time per request | A `tableWork` thread (gate); ≤ 50 µs on the model's strand (P) | A model that answers nothing else while its table sorts | +| C3 | Fifty watchers with fifty different sorts | Bytes held ≤ `ViewCacheOptions::maxBytes` (gate); views computed per second recorded | A cache without a bound | + +### Benchmarks: wire + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| W1 | Bytes of a generic page of 200 rows and 40 named columns in an `ok` envelope | ≤ 200 kB (P; 21 bytes a cell in a typed page gives 168 kB); no cell of a column the query did not name (gate) | A reply that carries the table's width | +| W2 | Encode of that page from the column store, and decode on the client | ≤ 2 ms and ≤ 3 ms (P) | A writer or reader whose cost grows faster than the cells | +| W3 | Bytes added by `unloadedRows` when all 200 rows have one unloaded level | ≤ 1% of W1 (P) | The sidecar repeating per cell what is true per level | +| W4 | A keyed reply of 20 rows and 40 columns: bytes, encode and decode | ≤ 20 kB, ≤ 0.5 ms (P) | A keyed reply carrying the page | +| W5 | The same 200 × 30 page as generic rows and as a typed `Page` | Generic ≤ 0.7 × typed in bytes (P) | An encoding that repeats what the header says | +| W6 | A 300-column `TableDescription`: bytes, encode, decode, and requests for it per mounted table | ≤ 60 kB, ≤ 2 ms (P); one request (gate) | A description fetched per page | +| W7 | The page of W1 with `columns` absent, all 300: bytes, encode, decode | ≤ 1.5 MB (P; 1.27 MB as a typed page) and under `kMaxEnvelopeBytes` (gate); ≤ 10 ms and ≤ 15 ms (P) | A whole page that does not fit an envelope | + +### Benchmarks: client cache and window + +`morph_table_tests` and `morph_ui_tests`, `"[.benchmark]"`, beside `bench_interpreter.cpp` and `bench_mount.cpp`. + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| K1 | `PageWindow::accept` of a 200 × 300 page | ≤ 0.1 ms (P) | Rows copied instead of moved | +| K2 | `refresh` with 900 event keys against 1,600 held rows in eight runs | ≤ 0.5 ms (P); the fetch holds the 900 keys and eight probes (gate) | Matching keys by scanning the held rows per key | +| K3 | `patch` of 200 rows | ≤ 0.3 ms (P) | The same scan | +| K4 | Query cache: a 4,096-key event against an entry of 200 rows; storing a decoded 200 × 300 page | ≤ 2 ms and ≤ 30 ms (P) | A walk of the whole value per key | +| K5 | Rows whose cells are evaluated again after a page refetch that changed 20 of 200 rows | 20 (gate) | A refetch repainting the page | +| K6 | Page fetches sent, and cancelled, while the scroll thumb is dragged across 100,000 rows in two seconds and released | At most three sent, and every one not at the resting place cancelled (gate) | A fetch per page passed | +| K7 | Fetches sent by scrolling sideways by one viewport over held pages | One column fetch per page around the visible rows, naming only the columns not held (gate); bytes ≤ W1 | A sideways scroll that fetches rows it holds | +| K8 | Cells converted to document values for a realized row of 300 columns, twenty drawn | Twenty (gate) | A row record built whole | + +### The frontend at 300 columns + +Windowing rows alone is not enough at 300 columns. A table that realizes, for each realized row, a widget for +every column, in one horizontal scroll area as wide as all the columns together, holds 50 rows × 300 columns at +thirty visible rows and a margin of ten each side: 15,000 cell widgets, 300 created for each row that scrolls +in. For scale, mounting 14,001 nodes takes 19 ms on the recording backend, which creates no Qt items. With the +columns windowed too ([The renderer](#the-renderer)) the same view is about 50 × 20: 1,000 widgets. + +`morph_qt_quick_tests "[.benchmark]"`, a 1,280 × 800 window, a model-backed table of the fixture's `wide`, +100,000 × 300, at a fixed height. The step 0 baseline is a `rows` table of 2,000 × 300, which is the size a mount +with no column window holds. + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| F1 | From a page reply to the first frame showing its rows | ≤ 250 ms (P) | Cell creation that scales with all columns | +| F2 | Cell and header widgets alive | ≤ realized rows × (pinned columns + columns in view + 2 × `columnOverscan`), and one header cell for each of those columns (gate) | A change that realizes more than the windows | +| F3 | Frame time scrolling vertically: at 20 rows a second, in a fling at 2,000 rows a second, and under a drag of the thumb across the table | p50 ≤ 8 ms, p99 ≤ 16.6 ms, in each | 300 widgets created per row scrolled in; realization that keeps up by missing frames | +| F4 | Frame time scrolling horizontally across all columns: at 10 columns a second, and in a fling at 100 columns a second | p50 ≤ 8 ms, p99 ≤ 16.6 ms, in each | Layout or rendering cost that follows the full width | +| F5 | Memory per realized row | ≤ 64 kB a row (P) | Widget growth | +| F6 | An arrival for the 50 realized rows, twenty cells each: frames it takes and bindings evaluated | One frame ≤ 16.6 ms (P); 1,000 cells evaluated and no other (gate) | An arrival rebuilding rows | +| F7 | Column-width work on that arrival | Each of the twenty columns grows at most once; ≤ 2 ms (P) | A rescan of every realized row per cell | +| F8 | Cells created, and cells rebound, per frame in F3's and F4's flings | Created: zero once the pools are warm; rebound: ≤ `realizePerFrame` (gates) | A scroll that allocates; a frame that realizes without bound | +| F9 | Work per frame that reads every column: extents, widths, header | Zero reads outside the column window, counted by `ColumnExtents` (gate) | Widths computed over 300 columns per frame | +| F10 | The focused cell after scrolling 10,000 rows and 200 columns away, and the row and column counts the table reports | The cell is realized and holds focus; counts are the view's total and the visible columns (gates) | Focus lost to recycling; counts of the realized rows | + +How a frame is timed: + +- **In CI**, under `QT_QPA_PLATFORM=offscreen` with the software scene graph, the test drives the scroll position + in fixed steps through `QQuickRenderControl` and times `polishItems`, `sync` and `render` for each step. The + figure is CPU time for one frame's work. It is comparable between commits on one machine and says nothing about + a GPU. The counts in F2, F6 and F7 are exact there and are the gates. +- **Locally, on a display**, with the threaded render loop and vsync, the same test records the intervals between + `QQuickWindow::frameSwapped` and reports p50 and p99 and the frames over 16.6 ms. This is the figure the 60 + frames a second target is held to, and it is recorded here with the machine and its load. + +F2 to F4 are the budgets the column window exists to meet: a table that realizes every column of its realized +rows is recorded against them first, and that record is the measure of what the window buys. + +### Benchmarks: end to end + +`tests/net`, a socket server and one client in one process, the loader held or timed. + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| E1 | First page of a sorted 100,000 × 300 table whose `recipe` level is not loaded: request to accepted page | ≤ 1.10 × the same table with no level; ≤ 120 ms (P) | A reply that waits for a load | +| E2 | From the loader's return to the patched rows, for 20 rows | p50 ≤ 80 ms, p99 ≤ 150 ms (P; 50 ms of it is the coalescing window) | A second round trip, or a page where rows would do | +| E3 | Requests the client sends from first page to patched rows | One page request, then one keyed request (gate) | Polling | +| E4 | The same arrival through the query cache | One page request per marked entry (gate); time recorded against E2 | The cost of a screen that is not on a window | + +### Benchmarks: browser + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| B1 | Client-mode engine with no worker, native: a three-key sort then a text filter, request to result and worst owner step, at 10,000, 20,000 and 100,000 rows | ≤ 5 ms, ≤ 12 ms (P) and ≤ 70 ms; worst step ≤ 5 ms at 20,000 (P) and ≤ 12 ms at 100,000 (M: 3.27 ms, 57.3 ms, 9.6 to 11.6 ms) | A step that runs past the frame at the size client mode is allowed in a browser | +| B2 | The same, and S1, S2 and W2's decode, compiled to WebAssembly and run under Node | Recorded as a factor over native; expected 1.5 to 3 (estimate) | Nothing: it replaces the estimate this design's browser figures use | +| B3 | Server mode: from a sort change, and from the last key of a text filter, to the accepted page, over a socket with 50 ms added each way | ≤ 150 ms; ≤ 250 ms debounce + 150 ms (P) | A sort or filter change that costs more than one round trip | +| B4 | Requests sent while typing a ten-letter word in 1.5 s | One (gate) | A filter that is not debounced | +| B5 | The WebAssembly build in a headless browser: mount of a `rows` table of 2,000 × 300, then one screen of vertical scroll | Recorded in step 0, against F1 and F3 natively | Nothing: it is the browser's baseline, taken before anything is built for it | + +The table module is header-only and builds for WebAssembly without Qt. Step 0 builds the table tests for that +target and runs them under Node, so the count gates of the single-threaded path run there as they do natively; +B2's factor is a timing and is recorded here with its date, as every timing is, and the engine's +single-threaded step size is set from it. B5 and U8 need a browser and the Qt WebAssembly build and are run +locally, under a headless browser. U8 is the exit criterion of the step that lands the column window, so the +browser is measured when the frontend's cost is decided and not after everything else is built. + +### Benchmarks: configuring + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| D1 | `distinctValues` on a text column with 50,000 distinct values in 100,000 rows, first 200, keys built | ≤ 30 ms (P; a sort of one text key with its keys is 46.3 ms) | Grouping by comparing texts instead of keys | +| D2 | The same with a three-letter search | ≤ D1 (P) | A search that folds every value again | +| D3 | The same with a filter on two other columns, one at an unloaded level | ≤ 1.5 × D1 (P); `partial` set (gate) | The cascade running the full filter per value | +| D4 | Compile a `oneOf` of 10,000 values; filter 100,000 rows with it | ≤ 5 ms; ≤ 15 ms (P; a text `contains` is 9.25 ms) | A membership test that scans the set per row | +| D5 | Lookups per row by that filter | One (gate, counted by the set) | The same | +| D6 | Wire bytes of a query with a 10,000-value `oneOf` of the fixture's texts, 16 bytes a value on average; of unticking one value of 50,000 | ≤ 250 kB (P) and under `maxQuerySetBytes` (gate); one value (gate) | A menu that sends the larger side | +| D7 | Adding a third key to a two-key sort of 100,000 rows | ≤ 50 ms (M: a three-key sort is 40.4 ms) | Nothing new in the engine; it bounds the gesture | +| D8 | Cell widgets created and released by hiding a column, and by moving one within the window | None created; the hidden column's cells, one per realized row, returned to their pool; every other cell still bound to its row and column (gates) | A layout change that rebuilds rows; a hide that does nothing | +| D9 | A server-mode distinct request: menu opened to list shown, over a socket | ≤ 150 ms (P) | A request per keystroke of the search (it is debounced, gate: one request per pause) | +| T1 | Parse, fit and apply a stored state for 300 columns with 8 sort keys and 32 filters, the store's own time excluded | ≤ 2 ms (P) | Fitting that scans the columns per stored id | +| T2 | Bytes of that state, and store entries and calls for it; the same for a state whose filter ticks 20,000 values | Recorded; one entry and one call for the first; the second reads back equal, through segments of at most `segmentBytes` (gates) | A store that shortens a state; one that writes every state as segments | +| T3 | Queries sent by a table mounting with a stored sort and filter | One (gate) | A first page fetched before the state is applied | +| T4 | Saves sent for 50 changes within a second | One (gate) | A save per change | +| T5 | Calls to load state when a screen of five tables mounts, and when it mounts a second time | One, then none (gates) | A round trip per table; a state fetched again by the session that holds it | + +### Benchmarks: column groups + +100,000 rows; a group of 3 and a group of 16 text columns; the group's keys built. + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| G1 | `any`, a `oneOf` of 2 values: 3 columns, 16 columns | ≤ 10 ms; ≤ 45 ms (P; about one set lookup a cell) | Compiling the set once per column | +| G2 | `any`, a `oneOf` of 1,000 values: 3 columns, 16 columns | ≤ 1.2 × G1 (P) | Cost that follows the set's size | +| G3 | `atLeast 2` and `exactly 1`, the same sets | ≤ 1.2 × G1 (P) | Counting that does not stop when decided | +| G4 | `together`, 2 entries over 3 columns; 16 entries over 16 columns | ≤ 12 ms; ≤ 120 ms (P) | A search over assignments instead of a matching | +| G5 | Per row: cell lookups, and augmentations in `together` | k lookups in either mode for value entries; at most one augmentation per entry, none for a row with an unmatched entry (gate) | A lookup per entry and cell; a matching attempted on rows that cannot match | +| G6 | Pooled distinct values over 3 and 16 columns, first 200 | ≤ 3 × D1; ≤ 16 × D1 (P) | Pooling by merging per-column lists per request | +| G7 | G1 and G4 with a third of the group's cells unloaded | ≤ 1.2 × the loaded figure (P); undecided rows equal the oracle's (gate) | Three-valued counting done by a second pass | + +### Benchmarks: values with units + +100,000 rows; one column of three dimensions and ten units, against a `Quantity` column of the same amounts. + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| M1 | Build keys and sort the `Measure` column | ≤ 1.2 × the `Quantity` column (P) | A registry lookup by name per cell; a dimension compared as text | +| M2 | Filter `gt "250 g"` | ≤ 1.2 × the `Quantity` column's `gt` (P) | Converting the operand per row | +| M3 | Compile 1,000 operands with units; `ui::parseMeasure` of 1,000 typed texts | ≤ 1 ms each (P) | A scan of the registry per operand | +| M4 | `distinctValues` on the column | ≤ 1.2 × D1 on a decimal column (P) | Grouping by shown text instead of key | +| M5 | Registry lookups per cell while building keys | One, by id (gate) | The same as M1 | + +### Benchmarks: a store's remainder + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| X1 | `finishPlan` over 100,000 candidate rows of the fixture with a `together` group and a sort | ≤ S1 + G4 (P) | Work beyond one `apply` | +| X2 | Rows a prefilter plan admits for a `together` of two values, the plan applied to the fixture's rows, against the view's size | Recorded; the per-entry `any` groups are in the plan (gate) | A prefilter that narrows nothing | +| X3 | Queries pushed down and finished in C++, per use, over the corpus | Reported; a model's expected split is asserted in its own test | A model that silently stops pushing down | + +### Benchmarks: every surface + +The budgets above are for the pipeline, and every surface is on it, natively and in the browser. These cases hold +each surface to them and record what each costs before it moves. + +| Id | Measures | Budget | Catches | +|---|---|---|---| +| U1 | A document `rows` table of 100,000 rows: a header click on a text column, to the sorted view | ≤ 60 ms (M: the engine sorts one text key in 46.3 ms) | A surface sorting outside the engine. The interpreter's pipeline is recorded first as the figure this replaces. | +| U2 | The same table sorted, one row's value changed | ≤ 10 ms when the list shares its unchanged rows, ≤ 40 ms when it is a fresh decode of 100,000 rows (P); one repair, no full sort, cells derived for the column in use only (gates) | A changed row sorting the list; a diff that reads every field | +| U3 | Text folds performed by a sort of 100,000 rows on one text key | 100,000: one per row (gate) | Folding inside the comparison | +| U4 | Memory of a 10,000 × 300 `rows` table as document values and as the source's cells, with one column sorted | Recorded; the source holds one column's cells (gate) | A second copy of the rows | +| U5 | A collection of 20,000 rows: mount, sort, one row edited | As U1 and U2 scaled; recorded | The collection leaving the shared path | +| U6 | A model-backed table of 100,000 × 300: F1 to F7, E1 to E3 on a real screen | Those budgets | A lowering that fetches pages where rows would do | +| U7 | Cell widgets alive in the terminal backend at 100,000 × 300 | Rows in view × columns in view (gate) | A backend that ignores the column range | +| U8 | U1 and U2 at 20,000 rows, and U6's scrolls, in the WebAssembly build in a browser | Recorded against the native figures; the frame budgets of F3 and F4 hold. The exit criterion of the column window's step | A path that works natively and not on one thread | + +**What a baseline is for.** A baseline is the figure of a path before the step that moves it onto the +pipeline, recorded once in step 0 and kept beside the budget it is held to afterwards: + +- *A `rows` table ordered outside the engine* (U1 to U4): a comparison that folds both texts each time it is + called folds about 3.4 million strings to sort 100,000 rows, where the engine folds 100,000 once. +- *A reply of every column* (W7 against W1 and W4): a page of 200 × 300 for an arrival of twenty rows. +- *A sort per request* (S7): ten page requests of one view as ten sorts, against one. +- *Every column realized* (F2): 15,000 cell widgets for 50 realized rows. +- *A table held as cells* (U5, S8): about 56 bytes each. + +## Invariants + +1. **Soundness of the filter.** For every way of replacing the unloaded cells of a table with loaded cells, a + row `apply` puts in the view is in the view of the completed table, and a row it rejects is rejected there. +2. **Soundness of the order.** Two rows with no unloaded cell in any sort column have the same relative order + before and after any completion. +3. **Counts.** view rows + undecided rows + rejected rows = source rows. `totalKind` is `AtLeast` whenever + `undecided > 0`. +4. **Unloaded is not empty.** `isEmpty` and `notEmpty` are `Unknown` on an unloaded cell; `displayText(Unloaded{})` + is the empty string and is never used as a key. A cell whose load failed is unloaded. +5. **No unloaded cell at the base level.** One is refused as `SourceContract`. +6. **Nothing unneeded is read or sent.** `apply` reads cells only of sorted and filtered columns; a reply holds + cells only of the columns its query names; `levelsNeeded` names no level outside the query's sorted, filtered + and named columns. +7. **Absent means loaded and whole.** A source that never produces `Unloaded`, columns with no `level`, a query + without `columns` and `keys`, and a `Page` without `unloadedRows`, `undecided`, `awaiting` and `failed` are a + fully loaded table of every column: no cell is unloaded, no row is undecided, and every result is the one + [`engine.md`](engine.md) specifies. +8. **A window's totals come from page replies only**, and a window never shows a row at a view row the server did + not report for it in the reply that supplied it. +9. **No event is lost to ordering.** For any interleaving of a page reply and an event for one of its rows, the + window ends holding the row as the server had it after the change. +10. **No polling.** Nothing is requested because time passed. Time only delays or merges a request that an + event or a gesture caused: the server's coalescing window, the window's settle and awaited-refetch delays, + a filter's debounce, a state save's debounce. Leases and retry delays are read from a clock when a request + arrives and schedule nothing. +11. **Loaded once.** Between two invalidations of a row's level, the loader is called for that row and level at + most once, however many queries, pages, sessions and clients need it, unless the value was evicted or its + load failed. +12. **Nothing stale lands.** A value whose load began before an invalidation of its row is never stored. +13. **Visible before told.** A change event for a row is sent only after a read of the cache would return the + row's new state, whether that state is a value, a failure or its absence. +14. **No value crosses a scope.** A reply holds a row only from the source the model returned for that caller, + and a cell only in a column `mayRead` allows. A cached value is laid out against the cache's own column list, + never a caller's description, and a cached view is shared only between callers of one `scope`. +15. **A table is its description.** Nothing a client or the framework needs to show, sort, filter or page a + table comes from a C++ type: a model that can build a `TableDescription` and a `DataSource` has a table. +16. **One order.** Every table on a screen orders and filters through `morph::table`: the same keys, the same + collation rules, the same place for empty, invalid and unloaded cells. +17. **A document keeps its rows.** A document that names none of this design's vocabulary shows the same rows in + the same order on the pipeline, except where [The document `table` node](#the-document-table-node-on-the-engine) + says otherwise. +18. **One mode per table**, decided by its declaration, for its lifetime. +19. **Nothing realized out of view.** The cells and header cells alive are bounded by the rows and columns in or + near view, the pinned columns and the focused cell's row and column, whatever the table's size. +20. **A changed row is a repair** on every surface that holds its rows. +21. **Arranging is not ordering.** A change of layout never changes the view's rows or their order, never runs a + sort or a filter on the server, and fetches only the cells of columns the window does not hold for the + rows around the view. +22. **A value list is what the other filters leave.** A column's distinct values and counts are those of the rows + passing every other column's filter, and the sum of the counts, `blanks` and `invalid` is that row count + when `partial` is false. +23. **A set filter is its conditions.** `oneOf` with values v1…vn matches exactly the rows `eq v1` or … or `eq vn` + match. +24. **A group filter is its definition.** For every row with no unloaded cell, a `Columns` filter passes exactly + when the count of satisfying columns meets `match`, and a `Together` filter exactly when an assignment of + entries to distinct columns exists; `{columns, combine, include}` alone is the `any` case. +25. **A prefilter only narrows.** The rows of a prefilter plan contain every row of the view, and the answer is + `apply`'s over them. A model that did not ask for a prefilter never receives one. +26. **Units do not change an amount.** Two `Measure` cells of one dimension order and compare as their amounts in + the base unit do, exactly; two of different dimensions are never equal and never ordered by amount. A unit's + numeric id means the same unit in every registry of one lineage. +27. **A stored arrangement never breaks a table.** Whatever a store holds, the table shows at least one + column and applies only keys and filters on columns it has. +28. **State before rows.** A table with a store shows its first rows only after its stored state is applied or + its load has timed out or failed; a table whose load completes in time sends exactly one first query. +29. **One store.** Every table's arrangement is read and written through the injected `ITableStateStore` and + nowhere else, no key it is given names a person, and no layer shortens what it is given. +30. **A cached view is the view.** A reply built from a view cache entry equals the reply `apply` would build + from the same source snapshot and level snapshots. +31. **A shift is seen.** After a keyed reply is patched, either every held row is at the view row the server + has it at and the window's totals are the server's, or the window is invalidated. +32. **A refusal is shown.** Every refusal of a user's gesture (a ninth sort key, a set too large, a range or a + copy too long, a save refused) appears in the status line or, where a document hides it, in the session's + notifications. + +## API surface + +Headers marked (new) do not exist before the step that lands them ([Delivery order](#delivery-order)). + +| Header | Holds | +|---|---| +| `table/data_source.hpp` | `struct Unloaded`; `Cell`'s alternatives `Unloaded` and then `Measure`, last; `ColumnInfo::level`, `dimensions`; `ColumnKind::Measure`; `TableErrorCode::SourceContract` (`"sourceContract"`); `displayText` returns empty text for `Unloaded`; `UnitInfo`, `UnitId`, `UnitRegistry` with `lineage()`, `UnitRegistryBuilder` (from a previous registry), `addUnitsOf`, `Measure`; `Services::units`; `DataSource::groups()`; `cellBytes`. | +| `table/sort.hpp` | `CellState::Unloaded`; `KeyColumn::read` maps `Unloaded`; `compareKeys` ranks valid, then empty and invalid, then unloaded. | +| `table/filter.hpp` | `enum class Match { No, Yes, Unknown }`; `CompiledFilter::evaluate(Column const&, KeyColumn const&, std::uint32_t) -> Match`; `passes` is `evaluate == Yes`; `FilterRun::takeUndecided() &&`; `FilterEntry::oneOf`; `GroupMode`, `ColumnCount`; `GroupFilter::group`, `mode`, `match`, `exclude`; matching for `Together`. | +| `table/engine.hpp` | `detail::JobInput::previousUndecided`, `detail::JobResult::undecided`; `Engine::undecidedRowCount()`, `undecidedRows()` (source rows), `onUndecidedChange(std::function)`; `Engine::distinctValues`; `ColumnLayout::setWidth`, `widths`, `setPinned`, `pinned`, `value`, `apply`, and `rebuild` placing a column by the fitting rule. | +| `table/query.hpp` | `TableQuery::columns`, `keys`, `distinct`, `retry`; `QueryLimits::maxColumns`, `maxKeys`, `maxSetValues`, `maxQuerySetValues`, `maxQuerySetBytes`, `maxCandidates`; `ColumnLevel` and the row type's `columnLevels()`; `LevelLoaded` as a trailing parameter of `RowsSource` and the typed `apply`; `LevelsNeeded`, `levelsNeeded`; `PageRows::totalKind`, `undecided`, `awaiting`, `failed`, `indices`; `UnloadedRow`, `UnloadedLevels`; `Page::unloadedRows`, `undecided`, `awaiting`, `failed`, `indices`, `distinct`; `DistinctRequest`, `DistinctValue`, `DistinctValues`, `DateBucket`, `distinctValues`; `FilterOp::OneOf`, `PlannedCondition::values`, `dimension`, `PlannedGroup::mode`, `match`, `exclude`; `PlanUse::Keys`, `Distinct`, `FilterSet`, `GroupCount`, `Measure`; `QueryPlan::keys`, `prefilter`; `Residual`, `finishPlan` over a `DataSource` and its typed overload; `changeKeys`, `rowIdOf`; `PageFetch::columns`, `revision`; `RowsFetch`, `Refresh`, `RefreshPlan`; `PageWindow`'s key function, `maxKeys` and `maxColumns`; `PageWindow::visible` with columns, `superseded`, `cancelled`, `refresh`, `owed`, `invalidate`, `patch`, `unloadedLevels`, `undecided`, `awaiting`, `failed`; the specialisation `PageWindow`. | +| `table/plan_conformance.hpp` | `ModelPage::undecided`, `unloadedRows`, `indices`; `standardCorpus` takes no operand from an unloaded cell and has keyed, `oneOf`, group, measure and distinct queries; `conformanceFailures` compares `undecided`, unloaded levels and keyed replies, and holds totals to the two-cause `AtLeast` rule; `ConformanceOptions::candidates` (row keys); the pushed-down and finished-in-C++ counts. | +| `table/edits.hpp` | `CellEdits::commit` refuses an unloaded cell. | +| `table/description.hpp` (new) | `TableColumn`, `ColumnGroupInfo`, `TableCapabilities`, `TableDescription`, `TableContext`, `PlannedTable`, `PlanRows`, `ServedTable`, `GenericRow`, `GenericPage`, `ChangedRow`, `ChangedRows`, `TableOf`, `table::typed`, `table::scoped`, `table::rowScoped`, `table::derivedFrom`; the generic page's writer and reader. | +| `table/level_cache.hpp` (new) | `LevelRead`, `LevelUse`, `LevelSnapshot`, `LevelSource`, `LevelCacheOptions`, `LevelCacheStats`, `LevelCache`, `CellLevelCache`. | +| `table/view_cache.hpp` (new) | `ViewCacheOptions`, `ViewCacheStats`, `ViewCache`. | +| `table/state_store.hpp` (new) | `TableStateKey`, `StoredTableState`, `SaveOutcome`, `SaveResult`, `ITableStateStore`, `MemoryTableStateStore`; the state's and the views' readers, writers, version migration and fitting. | +| `forms/views.hpp`, `forms/engine/collection.hpp` | The schema carries `x-level` from `columnLevels()` and `x-group` from `columnGroups()`; `deriveColumns` copies them; `CollectionColumn::level`. | +| `core/change_types.hpp` | `effects::IChangeObserver` (`changed(Changes const&)`, `bind(AnnounceSink)`); `effects::AnnounceSink`. | +| `core/remote.hpp` | `RemoteServer::announce`, `RemoteServer::attach(std::shared_ptr)`; `RemoteServerOptions::tableWork`, `views`. | +| `core/registry.hpp`, `core/effects.hpp` | `BRIDGE_REGISTER_TABLE` and its options; the actions `Table.` and `Table..describe`; the table's record; `effects::updatesTable`, `effects::changesTable`; the result walk's slot for `ChangedRows`, and its record of a holder's `openMembers()` and `partialMember()`; `ActionRecordInfo::open`, `partial`, `opaque`. `Page` declares both members. | +| `ui/table_rows.hpp` (new) | `SortChain`, `TableTotal`, `TableRowRef`, `TableCountsView`, `TableRows`, `EngineRows`, `WindowRows`, `ValueRowsSource`; `ui::parseMeasure`. | +| `ui/interpreter.hpp` | `Session::read`, `Session::watchRecord`, `WatchHolder`; the query cache's rules for open, partial and opaque entries; `SessionOptions::tableState`, `tableStateTimeout`, `tableStateDebounce`, `tableSettle`, `tableAwaitedRefetch`, `maxRangeRows`, `maxCopyRows`; the lowering of every table node onto `TableRows`; a table node's `status`, `persist`, a column's `kind` and `locked`. | +| `ui/view.hpp`, `ui/mount.hpp`, `ui/backend.hpp` | `Virtualization::columnThreshold`, `cellThreshold`, `columnOverscan`, `realizePerFrame`; `ColumnExtents`; the column window, cell pools and the frame budget; `Table::status`; `Backend::copyText`; the range-selection report by view rows. | +| `ui/table_state.hpp` (new), `ui/table_state_model.hpp` (new) | `PreferencesTableStateStore`; `TableStateModel` (`morph.tableState`), `TableStateOptions`. | +| `qt_quick` and the terminal backend | Cells and header cells for columns in the window only; pinned columns; header menus, drag, resize, the sort dialog, the column chooser, the filter menu, the status line. | + +A caller that passes nothing new gets what it gets without this design, with one exception, stated where it +applies: `ColumnLayout::rebuild` places a new column after its declared predecessor instead of last. + +## Tests + +Each case names the mutation it was seen to fail on, as the module's other tests do. + +| Condition | Test | Layer | +|---|---|---| +| (a) the first page arrives before a slow level is loaded | A recording `DataSource` logs the column of every `cell` and `readColumn` call and returns `Unloaded` for the slow level. `apply` returns the page, and the log holds only the sorted and filtered columns. Mutation: `apply` building keys for every column. Over the bridge: the list action answers while the loader is held on a latch the test never releases, and the reply's `unloadedRows` names exactly the unloaded rows and the level. Mutation: the model waiting for its loader. | `tests/table/test_table_unloaded_cells.cpp`; `tests/net/test_push_unloaded_rows.cpp` | +| (b) cells show as unloaded | `Page::unloadedRows` names exactly the unloaded rows and levels; `Engine::cellAt` returns `Unloaded`; `PageWindow::unloadedLevels` agrees after `accept`. Mutation: the typed `apply` dropping `unloadedRows`. | `test_table_unloaded_cells.cpp`, `test_table_server.cpp` | +| (b) unloaded cells do not match a filter | Per operator of every kind, `isEmpty` and `notEmpty` included, include and exclude, `any` and `all`, and a group: the row is undecided, the view lacks it, `totalKind` is `AtLeast`. A row `Unknown` at one stage and `No` at a later one is rejected. Mutation: `evaluate` returning `passes`. Property: invariant 1 over random tables, unloaded cells and completions. | `test_table_filter.cpp`, `test_table_unloaded_cells.cpp` | +| (b) unloaded cells do not sort as empty | A column with values, empties and unloaded cells, both directions: unloaded rows are the tail, after the empties. Mutation: `compareKeys` tying unloaded with empty. Property: invariant 2. | `test_table_sort.cpp` | +| (c) a level loads only when a shown, sorted or filtered column needs it | A table of `levelsNeeded` cases: a column of the level named, sorted, filtered, in a group, not named, `columns` absent, `columns` empty. A model with a recording loader: a query using no column of the level records nothing; one that names a column of it records the page's rows only; one that sorts on it records every candidate row. Mutation: `levelsNeeded` returning every level, which makes the first case record a load. | `test_table_query.cpp`; `test_push_unloaded_rows.cpp` | +| (d) exactly the arrived rows update, without polling | Engine: an undecided row arriving is one `Inserted` and one `onUndecidedChange`; an undecided row rejected is no view operation and one `onUndecidedChange`; a displayed row arriving is `Changed` for that row only, under `Immediate` and `Deferred`. Mutation: a repair that ignores `previousUndecided`. Window: `refresh` asks for the event's keys and one probe per held run; `patch` returns the replaced view rows only; a missing key, a moved row, an unheld key inside a held page and a changed total each invalidate; a keyed reply older than a later fetch for the same key is ignored. Bridge: over a real socket the server's action counter shows one page request, then none until `announce`, then one keyed request holding the announced keys and the window's probes; the test has no `refreshEvery` and advances no clock. Mutations: `refresh` asking for every held key; `refresh` dropping the keys the window does not hold, which leaves a row an edit moved into the window unseen. | `test_table_changes.cpp`, `test_table_server.cpp`, `test_push_unloaded_rows.cpp` | +| An event that precedes its page reply | The window is given `refresh(keys)` while the page fetch is in flight, then the reply with those rows unloaded: `owed` returns those keys. Mutation: `refresh` not recording keys in flight. | `test_table_server.cpp` | +| An ordinary edit on the same record | A held, fully loaded row's event yields a keyed fetch; a reply at another view row invalidates. Mutation: `refresh` skipping rows with no unloaded level. | `test_table_server.cpp` | +| Invalidation faster than a round trip | Invalidate, fetch, invalidate again, then accept the first reply: the page shows the reply's rows and is still stale; `visible` returns one fetch, not two. Mutation: `accept` ignoring every older revision. | `test_table_server.cpp` | +| `announce` | A derived record is included; more than `kMaxChangeKeys` keys become whole-type; an unregistered record is refused. `changeKeys` and `rowIdOf` round-trip both key kinds and agree with the record's key member. | `tests/net/test_push_unloaded_rows.cpp`, `test_table_query.cpp` | +| `TableQuery` on the wire | `columns` and `keys`, integer and text, round-trip through glaze; a query without them reads with both absent. An unknown column in `columns` is `UnknownColumn` and an unreadable one `ColumnNotReadable`; an empty `columns` gives a page of keys. | `test_table_query.cpp` | +| Translation of a plan keeps the semantics | The in-memory reference model handles unloaded cells and keyed plans; `conformanceFailures` runs over a table with unloaded cells in each kind and reports a model that calls an undecided total `Exact`. lims's two store-backed models refuse `PlanUse::Keys`, and their conformance runs skip the keyed query. | `test_table_database_pages.cpp`; `examples/lims/tests` | +| A value is loaded once | Two queries, then two clients over a socket, need the same 200 rows' level: `LevelCacheStats::loadedKeys` is 200 and the loader's own call log lists each key once. Mutation: `want` not checking the cache, which makes it 400 and 800. | `tests/table/test_table_level_cache.cpp`; `test_push_unloaded_rows.cpp` | +| Concurrent queries share a load | Eight threads call `want` for the same keys while the loader is held on a latch: one loader call. Mutation: no in-flight set. | `test_table_level_cache.cpp` | +| Reloaded once after invalidation | After `invalidate` of 10 keys the view lacks them; the next two queries load exactly those 10, once. An action declaring `updates` on the record does the same through the server with no call to `invalidate`. Mutation: `changed` ignoring the record, which leaves the old values served and the loader uncalled. | `test_table_level_cache.cpp`; `test_push_unloaded_rows.cpp` | +| A load in flight at invalidation | The loader is held; the row is invalidated; the loader returns the old value: the view does not hold it, `droppedLoads` is 1, and the next `want` loads again. Mutation: storing without the generation check, which serves the old value. | `test_table_level_cache.cpp` | +| Visible before told | The announce sink reads the cache when called and finds the value for every key it is given; on invalidation it finds none. Mutation: announcing before storing. | `test_table_level_cache.cpp` | +| Eviction | Over `maxBytes` the least recently used rows that no order lease holds go, the view shows them absent, and a query needing them loads them again; no event is sent for an eviction. | `test_table_level_cache.cpp` | +| No value crosses a scope | Two principals with different row scopes query one cached table: each reply holds only its rows, and the loader ran once for the shared rows. | `test_push_unloaded_rows.cpp` | +| An open entry hears every change | A cached page whose `awaiting` is not empty is marked by an event for a key it does not hold; one whose `awaiting` is empty is not. Mutation: the cache ignoring `open`. | `tests/test_ui_interpreter_cache.cpp` | +| A partial entry is not merged into | An `updates` reply for a row of a cached page with `unloadedRows` marks the entry and leaves its value; without `unloadedRows` it merges. Mutation: the cache ignoring `partial`, which shows an unloaded member's default as a value. | `tests/test_ui_interpreter_cache.cpp` | +| A table from a description | A model with no row type serves a table; a client given only the catalog draws its columns, sorts, filters and pages it. Mutation: a lowering that needs the node's `columns`. | `tests/table/test_table_description.cpp`; `tests/test_ui_table_model.cpp` | +| A model with several tables | Two tables of one model are authorised, cached and changed apart: a principal allowed one action is refused the other; an action declaring `updatesTable` on one marks only its watchers; an action that changes two tables returns two `ChangedRows`, and each table's watchers get that table's keys; `updatesTable` without its `ChangedRows` in the result is refused when the registries resolve. Mutation: one record for the model. | `tests/net/test_push_tables.cpp` | +| Columns by session | Two principals get different descriptions of one table; a page's `columns` are the caller's; a changed `revision` makes the client fetch the description again. | `tests/net/test_push_tables.cpp` | +| The generic page | Every kind of cell round-trips: empty, unloaded, a quantity with a factor and with none, a measure; a typed table served through `table::typed` writes the same page as the generic source of the same rows. | `test_table_description.cpp` | +| Column layout | A stored layout with unknown ids, a new column and every column hidden yields the fitted layout; hiding and moving create no cell widgets (gate D8). Mutation: `apply` ignoring `hidden`. | `tests/table/test_table_columns.cpp`; `tests/qt_quick` | +| Sort chains | `Shift`+click adds a key and flips one in the chain; a plain click replaces; a ninth key is refused. Mutation: `Shift` ignored. | `tests/test_ui_table_model.cpp`; `tests/qt_quick` | +| `oneOf` | It matches exactly the rows the OR of its `eq` entries matches, in include and exclude, with blanks beside it; an unloaded cell is undecided. Mutation: a `oneOf` that matches nothing. | `test_table_filter.cpp` | +| Distinct values | Counts are those of the rows the other filters leave; the column's own filter is ignored; counts, `blanks` and `invalid` sum to those rows when `partial` is false; a group's "(Blanks)" count is the rows its `isEmpty` entry admits; unloaded rows set `partial`. Mutation: counting over the view. | `tests/table/test_table_distinct.cpp` | +| Column groups | The truth table of [Column groups](#column-groups) row by row, including the row that tells `together` from `atLeast 2`; each quantifier with unloaded cells against its three-valued rule; an exclude. Mutation: `together` evaluated as `atLeast`. | `test_table_filter.cpp` | +| Values with units | 300 g sorts between 0.2 kg and 0.5 kg in both directions; `gt "250 g"` drops 0.3 % rows; 0.3 % and 0.3 kg are not equal; 0.3 kg and 300 g are one distinct value; an operand with no unit is a ratio. Mutation: a key that ignores the unit's factor. | `tests/table/test_table_measure.cpp` | +| A store's remainder | A model on `Residual::Refuse` never receives a prefilter plan; with `Allow`, `finishPlan` over the candidates equals `apply` over all rows; a prefilter returned as the answer is reported; more than `maxCandidates` rows is refused. | `test_table_database_pages.cpp` | +| Persisted state before the first query | A table mounts with a stored sort and filter in a `MemoryTableStateStore` whose load is held: no query is sent; released, one query goes out carrying the stored sort and filter. Mutation: a table that ignores its store sends the default query. | `tests/test_ui_table_model.cpp` | +| Stale state is fitted | A stored state naming a removed column, a removed group, an unknown unit, a filter that no longer compiles and every column hidden: the table shows, with one column at least, the rest applied, each drop reported once; what a session's description lacks is written back unchanged. Mutation: applying the state unfitted. | `tests/table/test_table_state.cpp` | +| Saves are coalesced | Fifty changes inside the debounce are one save holding the last arrangement; a change during a save in flight is one more; unmounting saves at once. Mutation: no debounce. | `tests/test_ui_table_model.cpp` | +| A slow or failed load | A load past the timeout starts the table from defaults and is applied when it arrives, unless the user changed something; a failed load is reported once and later changes still save. | `tests/test_ui_table_model.cpp` | +| The person is the store's | Two principals mounting one table through the preferences-backed store read and write separate state; a key naming a person does not exist to forge. | `tests/net/test_push_tables.cpp` | +| Bound values win | With `sort` bound, a stored sort is neither applied nor saved, and the stored layout still is; `persist: false` loads and saves nothing. | `tests/test_ui_table_model.cpp` | +| One order on every surface | A `rows` table, a collection and a model-backed table over the same rows show the same order for the same sort, text and measures included. Mutation: the interpreter's pipeline left in place for one surface. | `tests/test_ui_table_model.cpp` | + +| A cached view is the view | Property: for random sorts, filters and unloaded cells, a page, a keyed reply and a second page built from one cache entry equal `apply`'s over the same snapshots. An arrival on an order level makes the next request compute again; one on a display level does not. Mutation: an entry that ignores a level's generation, which serves the old order. | `tests/table/test_table_view_cache.cpp` | +| One sort for many watchers | Fifty requests for one view while it is computed: one computation, on a `tableWork` thread. Mutation: no single flight. | `test_table_view_cache.cpp`; `tests/net/test_push_tables.cpp` | +| A reply holds the named columns only | A page of forty named columns holds no other cell, and loads only the levels those columns have; a column that scrolls in is fetched for the held pages and merged by key; a column fetch whose keys differ invalidates. Mutation: a writer that ignores `columns`. | `test_table_description.cpp`, `test_table_server.cpp` | +| A row the window never held | An edit moves an unheld row into a held page, above it, and across it: the keyed reply's view row, a probe and the total each invalidate; an edit far below the pages with the total unchanged does not. Mutation: no probes, which leaves the held rows one view row off. | `test_table_server.cpp` | +| A scroll that passes | A drag across the table sends no fetch until the visible rows rest for `tableSettle`, and cancels what it left behind; a scroll into the next page fetches at once. Mutation: a fetch per `visible` call. | `tests/test_ui_table_model.cpp` | +| An awaited order is throttled | With `awaiting` set, ten events in 100 ms are one refetch at once and one after `tableAwaitedRefetch`. Mutation: a refetch per event. | `tests/test_ui_table_model.cpp` | +| A level's layout is the cache's | Two principals whose descriptions hold different columns of one level, in different orders, read one cached value: each reply has each cell under its own column, and a column `mayRead` refuses is absent. Mutation: reading the value by the caller's column order. | `test_table_level_cache.cpp`; `test_push_tables.cpp` | +| A cache's size is deep | Values of twenty text cells evict at `maxBytes` of their real size. Mutation: `sizeof(Value)`, under which nothing is ever evicted. | `test_table_level_cache.cpp` | +| A reader blocks no one | With a snapshot held, a store, an invalidation and the observer call complete, and the snapshot still reads its old values. Mutation: a shared lock held by the snapshot. | `test_table_level_cache.cpp` | +| Order uses are leased | Two order uses that do not fit together: the first loads once and keeps every row, the second is `Unavailable`; after `orderLease` on an injected clock the first's rows are evictable. A level whose values outgrow the estimate stops loading and announces without keys. Mutation: no lease, under which the loader's call count grows without end. | `test_table_level_cache.cpp` | +| A failed load | The loader returns no value for ten keys: their cells are unloaded, the reply names the level in `failed` and not in `awaiting`, the keys are announced once, and no request loads them until `retryAfter` on an injected clock, a `retry` query or an invalidation. A row with nothing at the level comes back empty and loaded. Mutation: a failed key wanted again by every request. | `test_table_level_cache.cpp`; `test_push_unloaded_rows.cpp` | +| A row-scoped table | A principal's watch receives an arrival without keys and refetches its pages; no event it receives names a key outside its scope. `rowScoped` with an empty `scope` is `SourceContract`. Mutation: the record registered without `privateKeys`. | `test_push_tables.cpp` | +| An opaque entry | A document query on `Table.` and a distinct-values entry are marked by a keyed change of the table's record that names none of their rows, and are not merged into. Mutation: the cache looking for a key member. | `tests/test_ui_interpreter_cache.cpp` | +| A read with no entry | `Session::read` makes no cache entry; `watchRecord` puts the record in the watch set while its holder lives and is told a whole-type change on resync; the first page fetch follows the watch. | `tests/test_ui_interpreter_cache.cpp` | +| A document's rows as a source | A fresh decode of the same rows notifies nothing; one changed row is one `Updated`; a change to what a `sortBy` reads updates that row; an unused column holds no cells; integers with decimals are one `Decimal` column and a text among them is one invalid cell. Mutation: a diff by row value. | `tests/test_ui_table_model.cpp` | +| Windows of the renderer | 1,999 × 300 is windowed both ways; cells and header cells alive match F2; a fling creates no cell once the pools are warm; a column growing left of the view keeps the columns in view where they are; pinned columns stay realized and narrow tables unpin for display only. Mutation: a column window that ignores `cellThreshold`. | `tests/test_ui_mount.cpp`; `tests/qt_quick` | +| Keyboard and focus under windowing | The focused cell survives a scroll of 10,000 rows; `Ctrl+End` over rows not fetched lands on the last row when it arrives; the reported row count is the total, or unknown for a lower bound. | `tests/test_ui_mount.cpp`; `tests/qt_quick` | +| Range selection in server mode | A range over rows not held resolves to the view's keys through pages that name no columns; over `maxRangeRows` it is refused with a status message and selects nothing. | `tests/test_ui_table_model.cpp` | +| Copy | Selected rows copy as tab-separated display text in layout order, in the chosen display units; rows not held are fetched; unloaded cells are empty and counted in the message; over `maxCopyRows` is refused. | `tests/test_ui_table_model.cpp` | +| The status line | Counts for exact, lower-bound and undecided totals; a failed level with "Retry", which sends `retry`; a refusal shown once; `status: false` sends the message to the session's notifications. Mutation: a refusal with no message. | `tests/test_ui_table_model.cpp` | +| State of any size | A state with a 20,000-value filter saves through `PreferencesTableStateStore` and reads back equal, as segments each within `segmentBytes`; a save stopped after its segments leaves the previous state readable; a missing segment is a failed load. Mutation: the store calling the preferences model, which refuses the value. | `tests/test_ui_table_state.cpp` | +| Views are not lost between windows | Two windows each add a view from the same loaded revision: the second save answers `Conflict`, reloads, and both views are stored. A `state` save overwrites. | `tests/test_ui_table_state.cpp` | +| Unit numbers across registries | A successor registry keeps every number and tombstones a removed unit; a level cache keeps its values under a successor and invalidates under another lineage; a unit id with a space is refused. | `tests/table/test_table_measure.cpp` | + +The arrival path on a screen (a model-backed `table`, its repaint and its frame cost) is tested and measured +with the lowering, as cases U6, F6 and F7 of [Performance](#performance). + +## What this does not do + +- It does not know how a level is read. The loader is the model's function; the level cache decides when it runs. +- It does not fetch one cell's detail on demand. A level that is never loaded in bulk and is fetched for one row + by a keyed query naming its column is expressible, and nothing here assumes a level is loaded for all rows. +- It does not add row attributes. `UnloadedRow` is a sidecar keyed by row index; another per-row sidecar on `Page` + does not conflict with it. +- It does not give `RowsSource` a row-level update: `setRows` notifies `Reset`. A client-mode table that + wants exact row updates on arrival uses a `VectorSource`. +- It does not page a forms collection or make its rows unloaded: a collection's rows arrive whole, and a + collection whose `v-query` takes a query is a design for the forms specs. +- **It does not edit cells generically.** There is no edit action per table, no `editable` column and no editor + chosen by a column's kind. A row is changed by an action of the application that declares `updatesTable`, + called from a column's `cell` template or from the screen; `CellEdits` stays the client-mode helper a + collection uses. A description-driven inline edit needs its own client vocabulary, validation path and + delivery step, and is a spec of its own beside this one. +- **It does not export.** Copy puts selected rows on the clipboard ([Copy](#copy)); writing a view to a file is + an application action over the query. +- It does not group rows, aggregate, or total a model-backed table: `group` and `rowSpans` are a `rows` table's. + Aggregate reports over a store are [`database_pages.md`](database_pages.md#aggregate-reports)'s. +- It does not offer top-N or ranking filters, or colour and icon filters. +- It does not keep loaded values across a server restart ([Caching layers](#a-persistent-tier)). +- It does not repair a cached view when a level's generation changes: the view is computed again, once for all + its watchers ([The view cache](#the-view-cache)). +- It does not cache a view for a table served through `planned`, or cache distinct values on the server. +- It does not invalidate a level cache in local mode, where no server derives changes: the application calls + `invalidate` itself. +- It does not lower a collection's or a wizard's chrome. +- It does not turn a `rows` table bound to a page into a model-backed one. +- It does not give a model-backed table a client-side select-all, grouping or row spans; the node refuses them. +- It does not sort or filter held rows on the client in server mode + ([Where sort and filter run](#where-sort-and-filter-run)). +- It does not change `forEach` or `grid`. +- It does not define the `layout` preference point. A table's arrangement is its persisted state, and the + `columns` and `saved` points that name a table edit that state. +- It does not merge two windows' arrangements or push a saved arrangement to a table that is already mounted. +- It does not limit the size of a table's stored state. + +## Alternatives considered + +- **A readiness mask per row and level instead of a `Cell` alternative.** It states the loader's unit directly and + costs one bit per row and level. But inside the engine the mask is a second call beside every cell: + `RowSnapshot` needs a new virtual, key building needs the mask beside each cell it is handed through + `ColumnSink`, `TableSnapshot`'s chunks and `withRow`, `VectorSource::updateRow` and `RowPatcher::patchRow` need + a mask argument, and every reader of `Engine::cellAt` must ask a second question or show an unloaded cell as + empty. The mask is used where a `Cell` cannot go, on the wire beside typed rows. +- **A `Loadable` member type in the row.** It would make the wire state impossible to ignore, but changes every + row type, its schema and every reader of its JSON, and says per member what is true per level. +- **Showing undecided rows in the view.** See [Unloaded cells in filters](#unloaded-cells-in-filters): it shows rows + the filter may reject and needs a + total that is an upper bound. +- **Two-valued filters, with an unloaded cell matching nothing.** An include then hides the row and an exclude shows + it, so the same unknown is answered both ways, and invariant 1 fails for excludes. +- **Unloaded tying with empty in the order.** Rows whose place is unknown would be scattered among the empties by + source row. +- **A separate "rows by key" action.** A second action per table repeats the table action's scope and + authorisation and can drift from it; `keys` on the query reuses both. +- **A keyed reply without view rows, patched in place.** It cannot tell an arrival from an edit that moves the + row, so a row edited in a sorted column would stay where it was. +- **Refreshing only rows with an unloaded level.** The record's events also carry edits; a loaded row would then + never refresh. +- **Dropping every reply older than the current revision, or cancelling fetches on `invalidate`.** With events + closer together than a round trip no reply is ever current, and the window never updates. +- **`announce` taking a ready-made `effects::Changes`.** It skips derived records and the key bound unless every + caller rebuilds them. +- **A level name in the event.** The server would decide per connection what a client needs; the client knows + what it shows and holds. +- **`ChangeKind::Loaded`.** An engine does the same work for it as for `Updated`. +- **Shown levels in the query instead of columns.** It keeps a query equal while columns of one level come and + go, which matters only where the query's body is a cache key. A window's fetches make no cache entry, and a + reply of every column is 1.27 MB at 300 columns, decoded on the painting thread in a browser, for the twenty + a user sees. +- **Asking only for the keys a window holds.** An edit that moves a row the window has never held into its + pages is then never seen, unless every action that might do so declares a whole-table change; and an action's + author cannot know how each client has sorted. +- **Sorting for every request.** Every page of a scroll, every keyed fetch and every watcher of an arrival pays + 33 to 46 ms at 100,000 rows, on a strand that answers nothing else meanwhile. +- **A view of the level cache that holds a shared lock.** A reply holds it for as long as its sort takes, and + the observer call that removes entries runs on the server strand and would wait for it. +- **Eviction by recency alone.** A row a view orders by is evicted by another query's loads, leaves the view + with no event, and is loaded again; two sorts on one level can then evict each other without end. +- **One untyped list of changed keys.** An action's keys are matched to a record by the type of the result's + slot, so one list cannot belong to two tables. +- **An edit action in the table's contract.** It needs client vocabulary, validation and effects of its own; + half of it in the contract is a capability no client can use. +- **`Pending` for the cell's state, renaming the table's other uses of the word.** A `table` element's `pending` + is read by documents, and the two facts differ. +- **Table state through the preferences model.** Its value bound is right for a preference and wrong for a + filter of 20,000 values; an exemption per caller would make the bound mean nothing. +- **A size limit on table state, dropping what does not fit.** A filter the user set would be forgotten + silently at the next mount. +- **Each model keeping its own store of loaded values.** Every model would re-derive deduplication, the bound, + the generation check and the order of storing and announcing, and a table would be as correct as its model's + copy of them. +- **Expiring cached values by age.** A slow value would be reloaded although nothing changed it, and a changed + one would be served until its age ran out. Invalidation says exactly when a value is wrong. +- **One cache per principal.** It multiplies the slowest work by the number of users. Scope is applied to rows + before the cache is read, so raw values can be shared. +- **An effect naming the levels it invalidates.** Effects name records; a second vocabulary beside them would + drift from the first. The cache's own `invalidatedBy` narrows instead. +- **Announcing an `order` level's arrival without keys for cache-driven screens.** It makes the loader know what + kind of client is listening. The cache's rule for open entries needs nothing from the server. +- **Sorting and filtering the held window on the client while the server's reply is in flight.** The held rows + are a slice of the old order, so their local order is not a slice of the new one: rows would move twice. +- **A row type as the contract.** The columns would be fixed when the server is compiled, so a table whose + columns depend on the session, or on data, could not be said, and every table would need a struct. +- **One generic table action addressed by a table id in its body.** See + [A model's tables](#a-models-tables): authorisation, the catalog, the cache key and effects are all per + action. +- **Column arrays on the wire.** The same bytes as row arrays, and every client structure is by row. +- **An unloaded marker in each cell.** Twenty markers a row for what one level name says. +- **A second, simpler pipeline for small tables.** Two orders on one screen, and the small table is the + one that grows. +- **A second engine for document values.** The engine's keys are typed arrays built once; ordering document + values directly is the per-comparison cost this replaces. +- **Detecting a paged `rows` expression and windowing it.** See + [Screens over a server's pages](#screens-over-a-servers-pages). +- **Virtualizing columns in the Qt Quick backend alone.** The mount would still build every cell's node; the + window has to be the mount's, as the row window is. +- **A list of `eq` entries for a value filter.** `QueryLimits::maxEntries` is 32, and 10,000 entries are 10,000 + comparisons a row. +- **One "all values" mode that lets two values share a column's match.** For values it is the same as + `together`, since a cell has one value; for conditions it is two `any` groups, which AND already. +- **Trying every assignment for `together`.** Sixteen entries over sixteen columns are 16! assignments; a matching + is found in a few hundred steps. +- **Translating `together` for a store.** Without a matching it is a count per subset of the entries, which + doubles with each entry, so a translation is practical for about six entries. In C++ the bound is the + group's size. +- **Refusing whatever a store cannot express.** The feature a user sees would depend on the table's storage. A + prefilter keeps every feature and bounds its cost. +- **Streaming the store's rows through the remainder in order.** The store must then sort, so an unsortable + column refuses again, and paging and counting are written twice. One `apply` over the candidates is the + oracle itself. +- **A prefilter by default.** A model written for whole plans would return the superset as its answer. +- **A quantifier on the existing `combine`.** `combine` says how entries meet in one cell; the quantifier counts + columns. One name for both leaves the fourth example unsayable. +- **Groups as a merged virtual column.** Sorting and showing the columns one by one would be lost, and "exactly + one" cannot be asked of a merged value. +- **One canonical unit for a mixed column.** There is no factor from kilograms to percent. +- **A per-cell dimension inside `QuantityCell`.** It grows every cell of every table by eight bytes; `Measure` is + smaller than the largest alternative. +- **Reversing the dimension groups with the direction.** The groups' order is a convention, not a magnitude. +- **Percent as its own dimension.** 50 % and 0.5 would be different values that mean the same. +- **A column default unit for an operand with none.** The same text would mean different amounts on different + columns; the menu supplies the unit instead. +- **Comparisons `Unknown` across dimensions.** `Unknown` means "not loaded yet"; a row would wait for a value + that will never come. +- **A separate `notIn` operator.** `exclude` already negates an entry, with the engine's rule for empty cells. +- **Distinct values over the view, the column's own filter included.** A value the user unticked would vanish + from the list and could not be ticked again. +- **A sibling action for distinct values.** It repeats the table action's scope and can drift from it. +- **A third click on a header clearing the sort.** It changes what a click does in documents that exist; the + header menu removes a key. +- **Persisting the layout through a bound `layout` and a handler in every document.** Every document would repeat + it, and the first page would be fetched before the handler ran. The binding stays for a document that wants + the value. +- **Table state as preference points.** A point's value is loaded after the screen mounts, which is after the + first query; and each application would need the points declared per table. One interface the table itself + calls loads before the query and needs no declaration. +- **Two stores, preferences and table state.** One arrangement could be written to both and read from either. + The default table store writes into the preferences store, so there is one. +- **The person in the key.** A client could ask for another person's state. +- **Merging concurrent saves.** See [Persisted state](#persisted-state). +- **Waiting for the state however long it takes.** A slow store would hold every table blank. +- **One `TableRows` implementation that holds either an engine or a window.** The two differ in whether a row can + be absent, which every reader would then test; two implementations of one interface keep that in one place. + +## Failure modes + +| Situation | Behaviour | +|---|---| +| A level never arrives | Its cells stay unloaded and its rows stay undecided; nothing times out in the engine. A load that failed is named in the reply's `failed`, and the status line offers "Retry". | +| A `Unloaded` cell in a base-level column | `SourceContract`, naming the column. | +| A `columnLevels()` entry naming no column | Refused where the row's columns are reflected. | +| An event without keys, a `resync`, or a key that is neither integer nor text | `PageWindow::invalidate`: the shown pages are fetched again. | +| An event that precedes the page reply holding its rows | The keys are recorded and `owed` asks for them after the reply. | +| A keyed reply misses a requested key, or returns it at another view row | The window invalidates. | +| A keyed reply after a later fetch for the same key | That row is ignored. | +| A page reply older than the current revision | Accepted when newer than what is held; the page stays stale. | +| More event keys and probes than `maxKeys` | The window invalidates instead of asking for rows. | +| A keyed query to a model that does not translate keys | `Unavailable`; the window invalidates. | +| `announce` for an unregistered record | Refused; nothing is sent. | +| `announce` after the server is gone | The cache's sink finds no server and does nothing. | +| The loader throws, or returns no value for a key it was given | The keys are `Failed`: their cells stay unloaded, replies name the level in `failed` and not in `awaiting`, `LevelCacheStats::failedLoads` counts them, `onLoadFailed` is told and the keys are announced once. They are loaded again after `retryAfter`, by a `retry` query, or after an invalidation. | +| A row is invalidated while its load is in flight | The load's result is dropped for that row. | +| The cache is over `maxBytes` | The least recently used values no order lease holds are evicted; their cells are unloaded again and reload on demand. | +| A sort or filter on a level whose candidate rows cannot fit in `maxBytes` | `want` finds the keys times the mean value size over the room left, and the framework refuses the query with `Unavailable` naming the column; nothing is loaded. | +| A change of a record in `invalidatedBy` other than the row's own | The whole level is invalidated and the table's record is announced without keys. | +| The server restarts | The cache is empty; every level is unloaded until it is asked for. | +| An edit of an unloaded cell | Refused on the cell with a `FieldError`; nothing is sent. | +| A filter on a level no cache serves | Every row is undecided and `awaiting` names the level. | +| A reader of `Page::rows` that ignores `unloadedRows` | It shows unloaded members as the server left them. | +| A `rows` table whose sort column holds values of different kinds | Those cells are invalid and sort last; reported once at the `sort` path. | +| A `rows` table over more rows or cells than the client-mode bound | A warning naming the table; it still shows. | +| A model-backed table whose action does not take a `TableQuery` | Refused when the document loads. | +| `pageSize`, `page`, `group` or `rowSpans` on a model-backed table | Refused when the document loads. | +| A row not fetched yet in a server-mode table | The row draws as a placeholder row of the estimated height. | +| A column scrolled into range | Its cells are realized for the realized rows; its width may grow once. | +| A fetch fails | `PageWindow::fail`: the table shows the error until the next reply. | +| A document names a table the model does not have | Refused when the document loads, against the catalog. | +| A reply's `revision` differs from the description held | The client fetches the description again and resets its window. | +| A description with two columns of one id, a key that is no column, a level or group naming no column | The table's action answers `SourceContract`; nothing is shown. | +| A principal may run a table's action but read none of its columns | A page with no columns; the table shows its empty state. | +| A stored layout names a column the table lacks, or hides every column | The id is dropped; the first declared column is shown. | +| A ninth sort key | Refused, with a message in the table's status line. | +| A `oneOf` over `maxSetValues` | `LimitExceeded`; the menu sends the smaller side, and disables the list when both exceed it. | +| A distinct request to a model that does not translate it | `Unavailable`; the menu offers conditions only. | +| A values list while the column's level loads | The loaded values with `partial`; the list refreshes on each arrival. | +| A saved view with a filter on a removed column | That filter is dropped when the view is applied. | +| No table state store is injected | Tables keep their state while mounted and persist nothing. | +| The state's load is slower than the timeout | The table starts from defaults; the state is applied when it arrives unless the user changed the arrangement. | +| The state's load or save fails | Reported once through the session; the table keeps working, from defaults or with the arrangement it has. | +| Stored state of a version the framework does not know, or that is not JSON | Not used; reported; overwritten at the next change. | +| A save over the operator's quota, or refused by the store | `Refused`: the status line says so, and the table keeps its arrangement, unsaved. | +| A stored state whose segment is missing | A failed load: reported, and the table starts from its defaults. | +| A `views` save over a revision another window replaced | `Conflict`: the table loads the views again, applies its one change and saves once more. | +| Stored state names what the caller's description lacks | Not applied; carried and written back unchanged. | +| A `Measure` cell whose unit the registry lacks, or whose dimension the column does not allow | An invalid cell: it sorts with the empty cells and matches nothing but `notEmpty`. | +| A `Measure` cell whose base amount does not fit | An invalid cell. | +| An operand with an unknown unit, a dimension the column does not allow, or two dimensions in a `between` | `InvalidValue`, naming the column. | +| An operand with no unit on a column that allows no ratio | `InvalidValue`. | +| A `Measure` column and no unit registry in the services | `SourceContract`. | +| A stored display unit the registry no longer has | Dropped; the cells show in their own units. | +| A group of columns of different kinds, of fewer than 2 or more than 16 columns, or sharing a column with another declared group | Refused where the columns are read. | +| A group filter naming a group the table lacks | `UnknownColumn`, naming the group. | +| `together` with fewer than 2 entries, or more entries than the group has columns | `InvalidSpec`. | +| `atLeast` or `exactly` with n below 0 or above the group's column count | `InvalidSpec`. | +| A part of a query the store cannot express, with `Residual::Allow` | A prefilter plan; `finishPlan` answers it. | +| The same with `Residual::Refuse` | `Unavailable`; the menu does not offer that part for the table. | +| A prefilter that returns more than `maxCandidates` rows | `Unavailable`, asking for a narrower filter. | +| A prefilter plan whose rows are returned as the answer | The model's conformance run reports rows the oracle lacks. | +| A user-formed group loses a column | The column leaves the group; under two columns the group and its filter are dropped. | +| A query's `columns` names a column the table lacks, or one the caller may not read | `UnknownColumn` or `ColumnNotReadable`; the client fetches the description again and asks once more. | +| A column fetch whose keys are not the held page's | The rows moved between the two fetches; the window invalidates. | +| A page reply whose total differs from the window's | The window adopts it and marks its other pages stale. | +| A keyed reply that shows a probe, or any held key, at another view row; an unheld key inside a held page; another total | The window invalidates. A check that fires because two replies crossed costs one page fetch. | +| A cell expression reads a column the window does not hold | The field reads as absent, the column is fetched, and the cell is evaluated again when it arrives. | +| An expression reads the row in a way the loader cannot follow | The table fetches every column; a warning names the node. | +| A fetch for a page the user scrolled away from | Cancelled; its reply, if it comes, is ignored. | +| `table::rowScoped` with an empty `scope` | `SourceContract` when the table is first served. | +| `updatesTable` on an action whose result holds no `ChangedRows` | A registration error when the registries resolve. | +| A second order use of a level that does not fit beside the first | `Unavailable` while the first's lease lives. | +| Leased values outgrow `maxBytes` while they load | The cache stops loading for order, drops the lease and announces the record without keys; the next query is refused. | +| A range selection over more than `maxRangeRows` rows, or a copy of more than `maxCopyRows` | Refused, with a message in the status line; nothing is selected or copied. | +| A unit id with white space, or one that begins like a number | Refused by `UnitRegistryBuilder`. | +| A level cache read under a registry of another lineage | The level is invalidated: its unit numbers mean nothing there. | +| A `rows` table that sets `persist` and has no `id` | Refused when the document loads. Without `persist` it shows, and persists nothing. | +| A refusal on a table whose document sets `status: false` | Shown through the session's notifications. | + +## Open questions + +- **Grouping text in a store.** Distinct values and group filters on text compare by the collator's fold. A store + that cannot group by the same fold answers through a prefilter or refuses; what a given store can express is + found by that model's conformance run. +- **A level cache in local mode.** With no server deriving changes, the application invalidates a cache itself. + A hook for it belongs with local mode's effects. +- **Frame timing in CI.** Whether `QQuickRenderControl` with the software backend gives stable step times on the + CI image is settled by the first baseline. +- **The shard count of a level cache.** 256 shards make a batch of 64 keys copy about a quarter of the cache's + key entries at 100,000 rows; L6 measures it, and the count is a constant to tune from that figure. + +## Neighbouring specs + +This document is the design. The specs below describe the code, so each takes its part when the step that +builds it lands, and until then says nothing of it; a link from here to one of them is to the file, not to a +section that does not exist yet. + +| Spec | Takes | With step | +|---|---|---| +| [`../core/registry.md`](../core/registry.md) | `BRIDGE_REGISTER_TABLE` and its options, the table's record and `TableOf`, the result walk's slot for `ChangedRows`, `openMembers`, `partialMember`, an opaque holder | 1, 2d | +| [`../core/backend.md`](../core/backend.md), "Fan-out" | `RemoteServerOptions::tableWork`; `RemoteServer::announce`, `attach`, `IChangeObserver`; the view cache | 1, 2c | +| [`../ui/interpreter.md`](../ui/interpreter.md), "Queries and the cache" | The rules for open, partial and opaque entries; `Session::read` and `watchRecord` | 2d | +| [`../ui/interpreter.md`](../ui/interpreter.md), "Not yet lowered" and the rows' pipeline | The entry for a model-backed `table` goes; the lowering onto `TableRows` is described; the "Rows' pipeline" and "Groups" rules keep what the lowering does around the engine's order | 4, 6 | +| [`../ui/document.md`](../ui/document.md), the `table` kind | `table`, `status`, `persist`, `layout`, `configurable`, a column's `kind` and `locked`; `filters` on every table; the element values this design names | 4, 6, 7 | +| [`../ui/preferences.md`](../ui/preferences.md) | That table state has its own model and is not bounded by `maxValueBytes`; the `columns` and `saved` points | 4 | +| [`../qt_quick/backend.md`](../qt_quick/backend.md) | The column window, cell pools, pinned columns, the status line; the menus and dialogs | 5, 7 | +| [`../forms/views.md`](../forms/views.md) | `x-level` and `x-group` on a schema property | 2a, 3 | +| [`engine.md`](engine.md) | Its "Not here yet" entries for the model-backed table and the single-threaded steps; `ColumnLayout`'s fitting rule | 4, 6, 7 | +| [`database_pages.md`](database_pages.md) | `PlannedTable`, prefilter plans and `finishPlan` | 1, 3 | + +## Delivery order + +The design lands in this order. Each step is shippable and green alone, and carries the gates named. A count +gate runs as an ordinary test; a timing is recorded in [Performance](#performance) when the step lands. + +| Step | Lands | Gates and records | +|---|---|---| +| 0 | The fixture's rows, with nothing that does not exist yet: the column-store `DataSource` and its generator (`bench::WideTableSource`), and a baseline of what the code does today. The table tests are built for WebAssembly and run under Node. | Gate: S3. Recorded: S1 and S2 with every cell loaded; S7 as ten `apply` calls; S8; B1 at 10,000, 20,000 and 100,000 rows; B2; B5; U1 to U5; W7 as a typed page; F2 to F5 on a `rows` table of 2,000 × 300, the size a mount without a column window holds | +| 1 | The generic table contract: `TableDescription`, `ServedTable` with `source` and `planned`, `BRIDGE_REGISTER_TABLE`, its options and its two actions, the table's record with `updatesTable`, `changesTable` and `ChangedRows`, the generic page with every cell encoding, `TableQuery::columns`, `table::typed`, requests answered on `tableWork`. The fixture becomes the registered model `bench::WideTableModel` with its three tables and no slow level. | W1, W2, W5, W7; W6's bytes and codec | +| 2a | The engine's part of slow columns: the `Unloaded` cell and its rank, the three-valued filter and the undecided set, `ColumnInfo::level`, `levelsNeeded`, `LevelLoaded`, `unloadedRows` on `PageRows` and the typed `Page`. | S1 to S6 | +| 2b | The wire's part: `unloadedRows`, `undecided`, `awaiting`, `failed` and `indices` on the generic page; `TableQuery::keys` and the keyed reply with its totals; `PlanUse::Keys` and the conformance corpus. | W3, W4 | +| 2c | The server's caches and the push: `LevelSource`, `LevelCache` with leases, failures and immutable snapshots; `RemoteServer::announce` and `attach`; the view cache. The fixture gains its slow level. | L1 to L8, S7, S9, C1 to C3 | +| 2d | The client's part: `PageWindow` under changes, with columns, probes and `superseded`; `Session::read` and `watchRecord`; the query cache's rules for open, partial and opaque entries. A table served over `apply` and read through a `PageWindow` then shows a first page before a slow level loads, shows its cells as unloaded, loads a level only when a named column needs it, and updates exactly the rows that arrived. | K1 to K4, K6, K7, E1 to E4 | +| 3 | The engine's configuring additions: `oneOf`, `distinctValues`, column groups with quantifiers and `together`, `Measure` and the unit registry, `ColumnLayout`'s widths, pinning, fitting rule and stored shape, `Residual` and `finishPlan`. | D1 to D6, G1 to G7, M1 to M5, X1 to X3 | +| 4 | `TableRows` and the lowering of a model-backed `table` onto `PageWindow`, from the description, with the status line's counts. The four behaviours of 2d then hold on a screen. `ITableStateStore`, the framework's stores and the load before the first rows land here, for sort and filters, so the rule is built into the first lowering. | U6 (E1 to E3 on a screen), K5, K8, F6, F7, T1, T3 to T5; W6's one request; F1 recorded | +| 5 | The column window in the mount and in both backends: extents, pinned columns, cell pools, the frame budget, focus, keys and counts under windowing. **Exit criterion:** U8, the same scrolls in the WebAssembly build in a browser, within F3's and F4's frame budgets. | F1 to F5, F8 to F10, U7, U8, D8 | +| 6 | `rows` tables on the engine through `ValueRowsSource`; the interpreter's own sort and page go. | U1 to U4; B1's 20,000-row step bound | +| 7 | The configuration interface: header and table menus, column order, visibility, pinning and width, persisted with the rest of the table's state; sort chains; the filter menu with values, conditions, groups and units; copy; range selection in server mode. Screenshots of existing screens change here, as intended. | D7, D9, B3, B4 | +| 8 | Collections through the same lowering. | U5 | +| 9 | Saved views, as the `views` part of the persisted state, with the conflict rule. | T2 | +| 10 | As measured: repairing a cached view in place of computing it again, local work when a window holds the whole view, a persistent tier under the level cache. | The figures that justify each | From 0ebb3ffe2c18fa96479f1ed5692e746b4e3da35f Mon Sep 17 00:00:00 2001 From: yaraslau Date: Tue, 17 Nov 2026 10:11:50 +0100 Subject: [PATCH 2/3] Specify acting on rows and editing in the table, and plan both table.md: how a column shows its cells (places, rounding, empty text, units, labels), tone, tooltip and lock from companion columns, and acting on rows: a selection of the whole view resolved on the server against its membership, the application's menu entries, rows an action changed kept in view, key changes, busy, failed and locked rows, a disabled header, shared views and a sort bound up to 32. Paging in a database leaves the design: a model's rows in a database are a DataSource the server sorts and filters with apply. editing.md (new): grids of a form's ObjectArray on the forms field model, derived values through evaluate and a derive action, cells of a model's table edited through CellEdit and RowEdit actions, the walk, cell selection, paste and fill, header bands, spans, summary rows and fixed grids. docs/superpowers/plans/2026-11-17-one-table.md: one implementation plan for both specs, task by task in their delivery order, including the removal of paging in a database. Co-Authored-By: Claude Opus 5.5 --- docs/spec/README.md | 4 +- docs/spec/table/editing.md | 625 +++++++++++++++ docs/spec/table/table.md | 711 ++++++++++++------ .../superpowers/plans/2026-11-17-one-table.md | 368 +++++++++ 4 files changed, 1484 insertions(+), 224 deletions(-) create mode 100644 docs/spec/table/editing.md create mode 100644 docs/superpowers/plans/2026-11-17-one-table.md diff --git a/docs/spec/README.md b/docs/spec/README.md index a3e6920a6..96525b775 100644 --- a/docs/spec/README.md +++ b/docs/spec/README.md @@ -139,7 +139,9 @@ server · lives in a database: `planQuery`, `Page::totalKind` and the conformance helper · [`table/table.md`](table/table.md) — the one table every surface uses: a table as a description any model can offer, the single pipeline, -unloaded cells and levels, the caches, configuration, and the budgets +unloaded cells and levels, the caches, configuration, acting on rows, and the budgets · +[`table/editing.md`](table/editing.md) — typing into a table: a grid of a form, cells of a model's table, the +cell editor, walking, paste, header bands, spans, summary rows and fixed grids **Reactive state and control** [`reactive/signals.md`](reactive/signals.md) · diff --git a/docs/spec/table/editing.md b/docs/spec/table/editing.md new file mode 100644 index 000000000..c25854a9c --- /dev/null +++ b/docs/spec/table/editing.md @@ -0,0 +1,625 @@ +# `morph::table` — editing and grids + +Design spec for typing into a table: a grid that is part of a record being edited, cells of a model's table +edited one at a time, the cell editor, walking, pasting and validation, and the layouts an entry grid needs and +a read-only table uses too (header bands, spans, summary rows, fixed grids). Read [`table.md`](table.md) first, +for the pipeline this builds on, and [`../forms/engine.md`](../forms/engine.md) for fields, drafts and exact +encoding, which this reuses and does not repeat. + +## Contents + +- [Goal](#goal) +- [Two homes for an edit](#two-homes-for-an-edit) +- [A grid of a form](#a-grid-of-a-form) +- [Cells of a model's table](#cells-of-a-models-table) +- [The cell](#the-cell) +- [Walking](#walking) +- [Selecting cells, clipboard and fill](#selecting-cells-clipboard-and-fill) +- [Rows of a grid](#rows-of-a-grid) +- [Layouts](#layouts) +- [Document vocabulary](#document-vocabulary) +- [Invariants](#invariants) +- [API surface](#api-surface) +- [Tests](#tests) +- [What this does not do](#what-this-does-not-do) +- [Alternatives considered](#alternatives-considered) +- [Failure modes](#failure-modes) +- [Neighbouring specs](#neighbouring-specs) +- [Delivery order](#delivery-order) + +## Goal + +A laboratory application enters most of its data in grids: the specimens of a test, the fractions of a sieve +analysis, the layers of a core, the limits of a specification. What such grids need, in the order it matters: + +- **An editor per cell that follows the record's types:** text, integers, exact decimals with fixed places, + quantities, nullable numbers, dates, choices whose options depend on the row, check boxes. The kind can differ + between the rows of one column. +- **Computed cells beside entered ones**, read-only, which the cursor skips, and which follow every entered value. + The computation is often the domain's, on the server, over the whole record. +- **Validation that keeps what was typed** and says why it is wrong, at the cell. +- **A keyboard walk** that enters a column of values without the mouse, and leaves the grid at its ends. +- **Layouts a printed form has:** headers in two or three bands with units, cells spanning rows or columns, + blocks of rows per specimen with a mean, a minimum and a deviation under each, grids whose rows are fixed by a + standard rather than by data. +- **Paste and fill** from a spreadsheet, and **a mark on what changed** since the record was loaded. + +Two things decide the design. + +- **An entry grid is a form.** Its values are members of one record being edited, saved together, checked + together and derived from each other. The forms engine already holds drafts, encodes them exactly, keeps what + was typed across a reload and submits; a grid is a way to lay out an `ObjectArray` field, not a second editor. +- **It is still a table.** The renderer, the windows, the keys, the copy, the status line and the arrangement are + [`table.md`](table.md)'s. A grid of 30 specimens and a table of 100,000 designs scroll, focus and copy alike. + +## Two homes for an edit + +| | A grid of a form | Cells of a model's table | +|---|---|---| +| The rows are | Members of one record: an `ObjectArray` field, or fields laid out as a fixed grid | Records of their own: the rows of a `Table.` | +| An edit goes | Into the form's draft; the form submits | To an action, per cell or per row, at once | +| Derived values | The form's evaluation and derive call, or the document's expressions | The action's result and the table's change events | +| Mode | Client: every row is held | Server: a model's table always is | +| Size | Tens to a few thousand rows | Any size the table has | + +A table node is in one home for its lifetime, decided by what it names, as its mode is. + +## A grid of a form + +### Binding + +A form's `ObjectArray` field is drawn as a grid, inside the form or anywhere in its scope: + +- inside a `form` node, the field's overlay `{"widget": "grid", "grid": {…}}`: `widget` chooses the grid as the + field's control, as `x-widget: grid` on the field's schema does, and `grid` holds the table's properties below; +- anywhere the form is in scope, a `table` node with `form` (the form node's id) and `field` (the field's path) + in place of `rows`. + +The grid's rows are the field's `RowState`s, keyed by their stable keys. `ui::FormRowsSource` is the +`DataSource` over them, as `ValueRowsSource` is over a document's list: a cell is the member's encoded value +when its draft encodes, and an invalid cell when it does not, so sorting and filtering a grid follow the table's +rules. Sorting or filtering a grid changes what is shown and never the array's order or its members: the body +holds the rows in the order they were entered. A row added while the grid is sorted or filtered is shown at +the end of the view, and passes the filter, until the sort or the filter next changes, so the row being entered +does not jump or vanish under the cursor. + +A grid of a form takes `group`, with its headings and its summaries ([Summary rows](#summary-rows)), as a table +over `rows` does: a grid of specimens grouped by sample is one declaration. + +### Columns are fields + +A grid's column names a member of the row type: `{"id": "mass", "field": "mass"}`. The member's `FieldSpec` +gives the column its kind, its places, its units, its options and its bounds, so a grid needs no column +declaration beyond layout. A column without `field` is a display column: its `cell` or `value` expression is +read in the row's scope and it takes no edit. + +**Per row.** A column's field part may be an expression in the row's scope, naming a different member per row: +`{"field": {"fn": "if", "args": [{"ref": "row.isChoice"}, "choice", "amount"]}}`. A row type that holds a value +of one kind for some rows and another kind for others has a member per kind, and the grid shows the one each row +uses. The kind of a value is the record's, never the grid's. + +A column also takes **overlays per row**, as a form field takes overlays: `decimals`, `unit`, `readonly`, +`options` (narrowing a choice's options), `placeholder`, each an expression in the row's scope. A row of a +specification grid that holds a value to one place and the next row to three says so here. + +On a grid column, `decimals` and `unit` are the field's overlays and mean what they mean on a `form` field: the +entry places, which are also the wire's `dp`, and the unit entered in. The column's display +([`table.md`](table.md#how-a-column-shows-its-cells)) follows them: a cell shows its value to the places its +field takes, so a grid never shows a value at places other than those it is entered with. The display-only +keys (`rounding`, `padPlaces`, `grouping`, `empty`, `labels`) keep their meaning on a grid. + +### Derived values + +A grid's derived cells come from three places, by who can compute them. + +- **The document's expressions**, for what the screen can compute from the values it holds: a column whose + `value` is `div(row.mass, row.volume)`, a summary row's `mean`. They are exact, follow every encoded draft at + once, and are display only: they are never sent. +- **Computed fields**, for a pure function of the record: members the action declares in `computedFields` + ([`../forms/forms.md`](../forms/forms.md#computed-fields)), on the action and, for a grid, on its row type. + The server recomputes them on every path and overwrites them on submit; the client asks for them while the + user is entering through the action's existing evaluation (`ActionDispatcher::evaluate`, the UI service's + `ui-evaluate`), which decodes a body, recomputes the computed fields and returns the recomputed body, with no + model instance, no journal entry and no state change ([`../core/registry.md`](../core/registry.md)). +- **Derived members**, for what needs the domain's data (a value looked up from the material, a limit from the + standard in force): members marked `derived()` in their type's field metadata, which the schema writes + `x-derived`. A computed field must be pure, so such a value cannot be one. A derived member is read-only in + the form, is not overwritten by `recomputeAll`, and is the model's to set when it executes the action: the + value the client holds is shown, never trusted. The form names its derive action with `derive`, an action of + the form's model that the application declares and registers as any action: + + ```cpp + /// The input of a derive action: the form's body as far as it encodes. + template + struct DeriveOf { Action body; }; + + /// The values of derived members, by path into the body that was sent. + /// Its JSON is {"values": [{"path": "specimens[3].density", "value": …}, …]}. + struct Derived { + struct Value { std::string path; glz::raw_json value; }; + std::vector values; + }; + + struct DeriveSieveAnalysis : forms::DeriveOf {}; + forms::Derived SieveModel::execute(DeriveSieveAnalysis const& derive); // `Loggable::No`, no effects + ``` + + It runs on the form's model instance (the form's `model` and `key`), declares no effect and is + `Loggable::No`, so a derive call is a read that leaves no trace. + +**The call.** After a cell's edit is committed ([The cell](#the-cell)), not per key, once the form's +`deriveDebounce` (a literal number of milliseconds on the `form` node, default 150) has passed with no further +commit, the form evaluates its body, and calls its derive action when it names one. + +- **The body as far as it encodes.** A member whose draft does not encode, or that is blank, is left out of the + body. A left-out member decodes to its default, as any body decodes; a computation that must tell "not + entered" from a value takes the member as `std::optional`, which is how the forms engine already represents a + nullable field. The domain computes what it can from what is there, as a lab sheet does with half its + fractions entered. +- **Loading the replies.** `FormSession::loadDerived` writes, from the evaluation's recomputed body, the computed + members, and from a derive reply, the derived members it names, and nothing else. Neither is a member the user + can edit, so a reply never replaces a draft the user typed, whatever was typed while it was in flight; and the + load moves no field's snapshot, so what the user typed stays an edit. A path is resolved against the rows as + they were when the call was sent (the session records the row keys of each `ObjectArray` at sending), so a + value lands on the row it was computed for even if rows were inserted or removed meanwhile, and a value for a + row since removed is dropped. The load is programmatic and causes no submission. +- **Order.** At most one evaluation and one derive call are in flight. A commit during them marks their replies + stale; the replies are still loaded, and then one more round is made with the newer body. Each round has a + sequence, and a reply older than one already loaded is ignored, so derived cells show at most one round trip + behind the last commit and never go back to an older result. +- While a round is in flight, derived cells draw muted. A failed round leaves the derived cells as they were, + marked stale, and the status line offers "Recalculate", which makes one more round. + +### Submission + +The grid submits with its form, in the form's mode. In explicit mode the form's submit first commits an open +cell editor, so a value typed and not yet confirmed is sent with the rest. In automatic mode a committed cell +whose form becomes ready submits the body. A failed submission's field errors land on the cells they name +(`specimens[3].mass`). Computed members are recomputed by the server and derived members set by the model, +whatever the body holds. + +## Cells of a model's table + +A column of a model's table is editable when its description names the action that changes it: + +```cpp +/// One cell's change: the row and the new value of one member. Its schema carries `x-cellEditOf` with the +/// table's record; which column it edits is the column's `EditInfo`. +template +struct CellEdit { + RowId key; + T value; +}; + +/// Several members of one row changed together. `Values` is the application's struct of the row's editable +/// members, each optional: a member present is changed, an absent one is left as it is. Its schema carries +/// `x-rowEditOf` with the table's record. +template +struct RowEdit { + RowId key; + Values values; +}; + +struct EditInfo { + std::string action; // the action id, taking a CellEdit or a RowEdit of this table + std::string member; // for a RowEdit, the member of `values` this column edits; empty for a CellEdit + std::string readonly; // the id of a companion column whose `true` makes this row's cell read-only, or empty +}; + +struct TableColumn { + // … table.md's members … + std::optional edit; +}; +``` + +- The cell's field is the `value` member of a `CellEdit` action's input schema, or the named member of a + `RowEdit`'s `values`, so its kind, places, bounds and options are the action's. A column whose kind and whose + field's kind differ, or a `member` that `values` does not have, is `SourceContract`. +- **The action is the application's.** It declares `updatesTable` and returns the `ChangedRows` of the row, as + any action that changes a table does ([`table.md`](table.md#a-models-tables)). A `FailedRow` for the row, or a + `FieldError` naming `value`, settles the cell with that message. +- **Committing** is `CellEdits`: edits to one row run one at a time, in order. A cell is stale from its commit + until the action replies. Then it shows the committed value, marked modified, until the window next fetches + the row, which shows the row as the server has it; an edit that changes nothing, or that moves the row out of + the view, leaves nothing waiting. A draft that does not encode, or one waiting to be committed, is held by row + key and column, not by the window, so a row that scrolls out of the window and back finds its draft and its + error where they were, and a key change by `from` moves them ([`table.md`](table.md#rows-an-action-changed)). + `errorCount` counts them wherever the row is. +- **A row at once.** A column whose `EditInfo` names a `RowEdit` collects the row's edits while focus stays in the + row and commits them together when focus leaves it, or on `Ctrl+Enter`. A refusal puts each `FieldError` on + its cell and keeps focus in the row. This is for a record whose members are checked together, and + `editMode: "row"` on the table chooses it for all its columns. +- **Who may edit.** The action's authorisation is the rule: a principal the authoriser refuses for the action + sees the column read-only, which the client learns from the catalog's permitted actions. A row is read-only + per cell through `EditInfo::readonly`, and entirely while it is locked + ([`table.md`](table.md#how-a-column-shows-its-cells)). `TableColumn::edit` absent is read-only. +- A cell that is unloaded, or in a busy row ([`table.md`](table.md#rows-at-work-rows-that-failed)), takes no + edit: an edit of a value the user has not seen is an overwrite of whatever arrives. +- **No row is held still.** An edit in a sorted or filtered column moves its row, or takes it out of the view, + when the window next fetches it. The walk moves by view position at the moment of the commit + ([Walking](#walking)), so entering a column of a sorted table continues down the rows that were below, and + does not follow the edited row. + +## The cell + +### The editor + +A cell's editor is the forms control for its field's kind, drawn inside the cell: a text input, a number input +with its places and unit, a check box, a dropdown for an `Enum` or a `Choice`, a multi-select for an `Array` of +options, a date or date-time input. A grid adds no control kind and changes none; a cell is a field with a +smaller frame. + +- **Choices that depend on the row.** A `Choice` field's `x-optionsDependsOn` names members of the same row + first, then the form's top-level fields. The options are fetched when the editor opens and cached by the + values they depend on, so a column of 300 rows over five materials makes five requests. +- **A check box** toggles with `Space` or a click and commits at once; it opens no editor. +- **Nullable numbers** are blank: a blank optional cell is no value and is left out of its row object, as the + forms engine encodes it. `Delete` on a focused cell makes it blank. + +### Entering and leaving + +| Gesture | Effect | +|---|---| +| Typing a character on a focused cell | Opens the editor with that character replacing the value | +| `F2`, `Enter`, double click on an editable cell | Opens the editor holding the value, caret at its end | +| `Enter`, double click on a read-only cell | Runs the table's `onActivate`, as on any table | +| `Enter` in the editor | Commits and moves by `enterMoves` ([Walking](#walking)) | +| `Tab`, `Shift+Tab` in the editor | Commits and walks forward or back | +| Arrow keys in the editor | Move the caret; `Alt+Down` opens a dropdown | +| `Escape` in the editor | Closes it and restores the draft the cell had when it opened | +| Focus leaving the table | Commits | + +A cell that is read-only opens no editor; typing on it says so in the status line. + +### Validation + +- **What was typed stays.** A draft that does not encode keeps its text, draws the error mark and shows the + forms error (`malformed`, `too-precise`, `below-minimum` and the rest) as its tooltip and in the status line + while the cell has focus. No dialog is opened and focus is not held: the user may walk on and come back. +- In a grid of a form the form is not ready while a cell does not encode, as for any field. In a model's table a + cell that does not encode is not committed. +- Rules beyond one value (`x-rules`, instance constraints, the server's checks) report through the form or + through the action's `FieldError`, on the cells they name. + +### Read-only and computed + +A cell is read-only when its field is `x-readonly`, `x-computed` or `x-derived`, a `readonlyWhen` rule holds, +the column's `readonly` overlay is true for its row, its `EditInfo::readonly` companion is true, the column has +no field or no edit action, the row is busy or locked, or the cell is unloaded. A read-only cell: + +- draws on the theme's read-only cell surface, so computed values read as computed; +- is skipped by the walk (`Tab`, `Enter`) and still reached by arrows and by a click, so it can be read, + selected and copied; +- takes no paste and no fill; a pasted value that lands on it is dropped and counted + ([Selecting cells](#selecting-cells-clipboard-and-fill)). + +Read-only, its surface and its place in the walk are one consequence of one fact, so they cannot disagree. + +### What changed + +A cell whose value differs from what the record held when it was loaded draws the theme's modified mark, and its +row's header draws the row's modified mark while any of its cells does. + +- In a grid of a form the comparison is per cell: the forms engine keeps, for each member of each row of an + `ObjectArray`, the draft it held when the form was last loaded, by row key (`FormSession::modified(path)`), as + it keeps a top-level field's. A row added since the load is modified as a whole. `refill` keeps comparing + top-level fields as it does; the per-cell record is what the marks and "Revert cell" read. +- In a model's table a cell is modified from its commit until the window next fetches its row. +- "Revert cell" in the cell's menu restores the loaded value (`FormSession::revert(path)`); "Revert row" restores + the row's. The element values `modifiedCount` and `errorCount` count such cells, and `modifiedRows` the rows. + +## Walking + +- **`enterMoves`**: `down` (the default: a column of measurements is entered top to bottom), `right`, or + `none`. +- **`walk`**: the order `Tab` follows, `rows` (along a row, then the next row; the default) or `columns` (down a + column, then the next column). The walk visits editable cells only, from the view position of the cell it + leaves. +- **The ends.** `Tab` on the last editable cell, and `Shift+Tab` on the first, leave the table to the next or + previous control of the screen's focus chain, so a grid sits in a form like any field. With `appendRows` a + `Tab` on the last cell, and `Enter` moving down from the last row, add a row instead. +- Arrow keys move to any cell, editable or not, and `Ctrl` with an arrow to the last cell in that direction, as + in [`table.md`](table.md#keyboard-focus-and-assistive-technology). + +**What the walk needs from the view tree.** Leaving the table moves focus to another node, which the view tree +cannot do yet: it is why the forms engine lists a grid widget as not supported. The view tree gains a focus +chain in document order and `focusNext`/`focusPrevious` from a node ([Neighbouring specs](#neighbouring-specs)); +step 1 of this design lands it. + +## Selecting cells, clipboard and fill + +A grid selects **cells**: `selectionMode: "cells"` is a rectangle of the view, extended with `Shift` and the +arrows or a drag, and the whole grid with `Ctrl+A`. A row header click selects its row, a column header's +selection gesture its column. It is the default for a grid of a form and allowed on any table held whole on the +client; a model's table, which is in server mode, selects rows, and refuses `cells`. + +- **Its value.** `selection` and `onSelectionChange`'s `event` are `{"anchor": {"key", "column"}, "focus": + {"key", "column"}}`, the rectangle's two corners by row key and column id; the rectangle is the rows between + the two keys' view positions and the columns between the two ids' places in the layout. `null` is no + selection. A sort or a column move keeps the corners and so may change the rectangle, as a spreadsheet does. +- **Copy** writes the rectangle as [`table.md`](table.md#copy) writes rows: tab-separated display text. +- **Paste** (`Ctrl+V`) takes tab-separated text and writes it from the focused cell right and down, the block + keeping its shape. Each value is read as the cell's draft, in the user's locale, and encoded by the cell's + field, so a pasted "12,5" in a comma locale is 12.5 and a pasted "abc" in a number column is an invalid draft, + kept. A value that lands on a read-only or covered cell is dropped, and the status line says how many were. + Rows the block runs past are added when `appendRows` allows, up to the field's `maxItems`, and dropped + otherwise, said in the status line. +- **Fill down** (`Ctrl+D`) copies the first row of the rectangle into its other rows, editable cells only. +- **Clear** (`Delete`) blanks the rectangle's editable cells. +- A paste, a fill or a clear in a grid of a form is one change of the form: the round of derived values it + causes is one. + +## Rows of a grid + +A grid of a form adds and removes rows of its field. + +- `appendRows` (a literal flag) adds a row by the walk past the end, and offers "Add row" under the grid. +- The cell's and the table's menus offer "Insert row above", "Insert row below" and "Delete row" (or "Delete + rows" for a selection); `Ctrl+Shift+=` inserts below and `Ctrl+-` deletes, as in a spreadsheet. A deletion of + rows that hold values asks once, in the table's status line, with "Undo" until the next change. +- The field's `minItems` and `maxItems` bound them: the entries are disabled at a bound, and say why. +- `rowsFixed` (a literal flag) offers none of them: the rows are the record's, such as one per sieve of a + standard series. + +Rows of a model's table are added and removed by the application's actions, through its menu entries +([`table.md`](table.md#the-applications-menu-entries)). + +## Layouts + +Each layout here is drawn by the mount and holds for any table it fits, editable or not. + +### Header bands + +```cpp +struct BandInfo { + std::string label; + std::string unit; // shown after the label: "Mass retained [g]"; empty for none + std::vector columns; // the column ids under the band + std::vector bands; // bands one level down, each over a subset of `columns` +}; +``` + +`bands` (`TableDescription::bands`, or the document's parts of the same shape) are headings over columns, +nested up to three levels, drawn above the column headers. A band one level down covers a subset of its +parent's `columns`. A column belongs to at most one band per level. + +- **Units.** A band's `unit` is shown after its label as a column's is, and a column under it whose unit is the + same does not repeat it, so a band "Mass retained [g]" over six sieve columns shows the unit once. +- **Under rearranging.** When the layout keeps a band's columns side by side, the band is drawn once over them. + When the user moves or hides columns so that they are not, the band is drawn over each run of its columns that + stands together, so a band never covers a column that is not its own. A band whose columns straddle the + pinned edge is drawn as two parts, one pinned, one scrolling. +- A band with no visible column is not drawn. A user does not form bands; a user-formed column group draws as a + band of one level, and so does a declared column group ([`table.md`](table.md#column-groups)). +- A band's accessible name is read before each of its columns' headers: "Mass retained, sieve 4 mm". +- Bands are windowed with their columns: a band exists for the columns in the column window. + +### Spans + +A cell may cover cells beside it and below it. A span part is `{column, rows, columns}`: the cell in `column` +covers `rows` rows, itself included, downwards, and `columns` columns, itself included, to the right in the +layout; either count defaults to 1. + +- A table over rows gives its spans through `rowSpans` + ([`table.md`](table.md#the-document-table-node-on-the-engine)), whose parts take `columns` beside `rows`. A + fixed grid gives them on its cells ([below](#fixed-grids)), as `rows` and `columns` of the cell part. +- A span across a column the user has moved away or hidden is cut at that column. +- Spans are drawn only while the table is unsorted and under its windowing thresholds, as row spans are; a + fixed grid is always both. +- A covered cell is not focused, selected alone, walked or pasted into; the walk passes over it. + +### Summary rows + +`summaries` are parts `{key, label, position, cells}`: rows drawn after the table's rows (`position: "bottom"`, +the default) or before them (`"top"`), outside the order, the filter and the selection. + +- `cells` maps column ids to expressions read in a scope with `rows`, the records of the view's rows in view + order, and `column`: `{"fn": "mean", "args": [{"map": {"ref": "rows"}, "as": "r", "to": {"ref": "r.mass"}}]}`. + The expression language's `count`, `sum`, `mean`, `min`, `max`, `stdev(list, dp, mode?)` and + `rsd(list, dp, mode?)` are the aggregates; its exact arithmetic and its rounding modes are theirs. +- `rows` are the view's rows: a summary follows the filter, so it summarises what the user sees. +- A summary cell shows by its column's display and draws on the theme's summary surface. It is read-only, is + copied when a copy covers it, and is never pasted into. +- **Per group.** A `group` part takes `summaries` too: drawn after each run of the group, in a scope whose + `rows` are that run's. A grid of specimens grouped by sample shows each sample's mean, minimum, maximum and + deviation under its rows, and the table's own summaries under all of them. +- A summary row is a table over held rows' feature: a model-backed table refuses `summaries` and group + summaries, as it refuses `group`. A total over a whole model's table is the application's report. + +### Fixed grids + +Some grids have rows fixed by a standard or a specification, each row a different quantity: rows of tests and +columns of minimum, maximum and declared value; a key-value sheet of a sample's parameters. A **fixed grid** is a +table node with `layoutRows` in place of `rows`, and `form`, the id of the form node whose fields its cells +edit (absent for a fixed grid that shows values only): + +- `layoutRows` are parts `{key, label, cells}`, in order; keys are unique in the grid. `cells` maps column ids to + a cell part: `{field}` (a path of that form's fields, `limits.density.min`), `{value}` (an expression), or + `{text}`, with the column overlays, and `rows` and `columns` for a span. +- Each cell is its own field, so each takes its own kind, places and editor: one row's minimum is a decimal to + three places and the next row's a choice. +- A fixed grid is not sorted, filtered or rearranged by rows; its columns may still be hidden and resized. It + walks, selects, pastes and validates as any grid. +- **Blocks.** A part `{"forEach": "specimens", "rows": […]}` repeats its rows for each row of that + `ObjectArray`, in the block's own scope, where `row` is the specimen and a cell's `{field}` path is relative to + it. A block row's key is the specimen's row key and the part's key joined (`"/"`), so it is + stable while specimens are added and removed. A block of several fixed rows per specimen is one declaration. + +## Document vocabulary + +| Key | On | Meaning | +|---|---|---| +| `form`, `field` | `table` | A grid of a form's `ObjectArray` field, in place of `rows` | +| `{"widget": "grid", "grid": {…}}` | a `form` field overlay | The field drawn as a grid, with the table's properties | +| `layoutRows`, with `form` | `table` | A fixed grid, in place of `rows`; `form` names the form its cells edit | +| a column's `field` | grid column | The member the column edits; an expression for one member per row | +| a column's `value` | grid column | A display column's expression in the row's scope | +| a column's `decimals`, `unit`, `readonly`, `options`, `placeholder` | grid column | Overlays per row, expressions in the row's scope | +| `derive`, `deriveDebounce` | `form` | The form's derive action; milliseconds of no commit before a round, default 150 | +| `editMode` | `table` | `cell` (the default) or `row`, for a model's table | +| `enterMoves`, `walk` | `table` | The walk ([Walking](#walking)) | +| `selectionMode: "cells"` | `table` | A rectangle of cells | +| `appendRows`, `rowsFixed` | `table` | Which row operations a grid offers | +| `bands` | `table` | Header bands | +| a span part's `columns` | `rowSpans` part, fixed-grid cell | A span across columns | +| `summaries` | `table`, `group` | Summary rows | +| Element values `modifiedCount`, `modifiedRows`, `errorCount`, `editing` (the cell being edited, `{key, column}` or `null`) | `table` | | + +Every key is optional, and a table that names none of them is what [`table.md`](table.md) describes. + +## Invariants + +1. **One field model.** A cell's draft, encoding, error and bounds are its field's, from the forms engine; a + grid holds no value the form does not. +2. **Typed text survives.** No commit, evaluation, derive reply, refill or change event replaces a draft the + user changed since it was loaded, until that draft is submitted, committed or reverted. +3. **Derived values never run backwards.** A reply older than one already loaded is not loaded, and a loaded + value lands on the row it was computed for or nowhere. +4. **Read-only is one rule.** A cell is skipped by the walk, refuses an edit, a paste and a fill, and draws as + read-only exactly when [Read-only and computed](#read-only-and-computed) says it is read-only. +5. **The record's order is the record's.** Sorting or filtering a grid of a form never changes the order or + the content of the field's array. +6. **No edit of an unseen value.** No edit is committed for an unloaded cell, a busy or locked row. +7. **A paste is its cells.** A paste writes exactly the drafts a user would have typed into the editable cells + under the block, and says how many values it dropped. +8. **A span covers what it says.** A covered cell is never focused, walked, selected alone or pasted into. +9. **The server decides derived values.** A computed or derived member the client sends is never the value the + server keeps. + +## API surface + +| Header | Holds | +|---|---| +| `table/edits.hpp` | `CellEdit` and `x-cellEditOf`, `RowEdit` and `x-rowEditOf`, `EditInfo`; `CellEdits` committing a `RowEdit` per row in `row` mode, and holding drafts by row key and column, moved by a key change. | +| `table/description.hpp` | `TableColumn::edit`; `TableDescription::bands` and `BandInfo` ([`table.md`](table.md#api-surface)). | +| `forms/derive.hpp` (new) | `DeriveOf`, `Derived`; the `derived()` field metadata and `x-derived`. | +| `forms/engine` | `FormSession::loadDerived`, the round's debounce, sequence and row-key mapping, over `evaluate` and the derive action; per-cell snapshots of `ObjectArray` rows, `modified(path)`, `revert(path)`; `computedFields` on a row type, recomputed by `recomputeAll`; options of a `Choice` depending on its row. | +| `ui/form_rows.hpp` (new) | `FormRowsSource`: a `DataSource` over an `ObjectArray` field's rows. | +| `ui/grid.hpp` (new) | The cell editor host, the walk, cell-range selection and its value, paste, fill and clear, row operations; summary rows and fixed grids' lowering, with blocks. | +| `ui/view.hpp`, `ui/mount.hpp`, `ui/backend.hpp` | The focus chain and `focusNext` / `focusPrevious`. | +| `ui/interpreter.hpp` | The lowering of a grid, a fixed grid and an editable model column; the keys of [Document vocabulary](#document-vocabulary) and its element values. | +| `ui/view.hpp`, `ui/mount.hpp` | Bands, spans and summary rows in the mount; the read-only, modified, stale and locked cell and row marks. | +| `qt_quick` and the terminal backend | Editors in cells; the marks; bands and spans. | + +## Tests + +| Condition | Test | Layer | +|---|---|---| +| A grid is a form | A grid over an `ObjectArray` of three rows: typing "12,5" in a comma locale gives the body `12.5` exactly; a draft with too many places is kept, marked and keeps the form unready. Mutation: a grid that encodes with its own parser. | `tests/test_ui_grid.cpp` | +| Read-only is one rule (grid) | A computed member, a derived member and a `readonlyWhen` cell are each skipped by `Tab`, refuse typing, drop a pasted value and draw the read-only surface. Mutation: the walk testing `x-readonly` only. | `tests/test_ui_grid.cpp` | +| Read-only is one rule (model table) | An unloaded cell, a busy row, a locked row and an `EditInfo::readonly` cell refuse an edit and are skipped by the walk. Mutation: the busy test left out of the rule. | `tests/test_ui_table_model.cpp` | +| The walk | `enterMoves: down` fills a column; `Tab` on the last editable cell moves focus to the next control; with `appendRows` it adds a row; `walk: columns` goes down first; after an edit that moves its row in a sorted model table, `Enter` lands on the row that was below. Mutation: `Tab` wrapping to the first cell, which never leaves the grid. | `tests/test_ui_grid.cpp`; `tests/qt_quick` | +| Derived values | A commit evaluates once after `deriveDebounce` and calls the derive action with the body as far as it encodes; a commit during the round yields one more; computed and derived members load, onto the rows they were computed for after a row was inserted above them, and a member typed meanwhile stays as typed and modified; an older reply is ignored; a failed round marks the cells stale and "Recalculate" runs one more. Mutation: `loadDerived` resolving paths against the current rows, which writes a value onto the wrong row. | `tests/test_ui_grid.cpp` | +| The server decides | A submitted body with a forged computed and derived member: the server keeps `recomputeAll`'s value and the model's. Mutation: `recomputeAll` skipping row members. | `tests/forms/test_forms_derive.cpp` | +| Kind per row | A column whose `field` expression names `choice` for one row and `amount` for the next opens a dropdown and a number input, and encodes each by its own field. Mutation: the column's field resolved once for every row. | `tests/test_ui_grid.cpp`; `tests/qt_quick` | +| Choices per row | Options depending on a row member are fetched once per distinct value. Mutation: a fetch per row. | `tests/test_ui_grid.cpp` | +| Paste, fill, clear | A 3 × 4 block over a read-only column keeps its shape, drops and counts those values, adds rows to `maxItems` and reports the rest; fill down and clear touch editable cells only; one paste is one derive round. Mutation: a paste that shifts values past a read-only cell. | `tests/test_ui_grid.cpp` | +| Cell selection | `selection` is the two corners by key and column; `Shift`+arrows extend it; `cells` on a model's table is refused. Mutation: corners stored as view positions, which a sort moves to other rows. | `tests/test_ui_grid.cpp` | +| Rows of a grid | Insert and delete at `minItems` and `maxItems` are disabled; a deletion of rows with values offers "Undo"; `rowsFixed` offers none. Mutation: bounds unchecked. | `tests/test_ui_grid.cpp` | +| A sorted grid keeps its record | Sorting a grid by a column and editing a row submits the array in its entered order with the edit in place; an added row shows at the end until the sort changes. Mutation: a source that reorders the field's rows. | `tests/test_ui_grid.cpp` | +| Submission | An explicit submit with a cell editor open sends the typed value; a field error `specimens[3].mass` lands on that cell. Mutation: submit before the editor commits. | `tests/test_ui_grid.cpp` | +| What changed | Typing in one cell marks that cell and its row, not its neighbours; "Revert cell" restores the loaded value; a derive reply moves no snapshot. Mutation: a mark per `ObjectArray`, which marks every cell. | `tests/test_ui_grid.cpp` | +| Cells of a model's table | An edit commits through the column's action, is stale until the reply and modified until the row's next fetch, and shows a `FieldError` on refusal; `row` mode commits once when focus leaves the row; a draft survives its row scrolling out and back, and follows a `from`; a principal refused the action sees the column read-only. Mutation: drafts held by the window, which loses them on scroll. | `tests/test_ui_table_model.cpp`; `tests/net/test_push_tables.cpp` | +| Bands | A band over three columns draws once; moving one away draws it over two runs; across the pinned edge it draws as two parts; hiding all three draws none; the band's unit is not repeated by its columns. Mutation: a band drawn over its first and last column's span, which covers a foreign column. | `tests/test_ui_mount.cpp`; `tests/qt_quick` | +| Spans | A cell spanning two rows and three columns covers them; the walk passes over covered cells; a paste onto a covered cell drops the value; moving a covered column cuts the span. Mutation: covered cells left focusable. | `tests/test_ui_mount.cpp`; `tests/test_ui_grid.cpp` | +| Summary rows | A grid's `mean` and `stdev` follow every committed value exactly and follow the filter; group summaries are per run; a model-backed table refuses `summaries`. Mutation: summaries over the source rows, which ignore the filter. | `tests/test_ui_grid.cpp` | +| Fixed grids | A `layoutRows` grid of four parameters with different kinds per cell walks, pastes and validates per cell; a `forEach` block repeats per specimen with keys that survive inserting a specimen above. Mutation: block keys by position, which move every row's state on insertion. | `tests/test_ui_grid.cpp` | + +## What this does not do + +- **It does not lay out printed reports.** A table drawn on a report page, filled by the report and split across + pages, is a report's design, not an interactive table's. +- **It has no formula cells.** No cell holds an expression a user typed, and there are no references by cell + address. Computation is the server's (computed fields and derived members) or the document's (an expression + column or a summary row). +- **It does not undo.** "Revert cell" and "Revert row" restore the loaded value, a deleted row offers "Undo" + until the next change, and the form's `reset` restores the record; a history of edits is not kept. +- It does not paste into, fill or clear a model's table, which selects rows, not cells: a bulk change of a + model's rows is an action over a selection ([`table.md`](table.md#acting-on-rows)). +- It does not derive values for a model's table on the client: its derived values arrive with the row. +- It does not hold an edited row still in a model's table. +- It does not let a user form header bands. + +## Alternatives considered + +- **A grid with its own cell store**, filled and read back by the screen. Every screen then writes the copy in + and out, the encoding is the screen's, and the dirty state is a second bookkeeping beside the form's. The + form already holds the record. +- **One derive mechanism for both pure and domain values.** A computed field must be pure and is overwritten on + submit; a value from the domain's data needs a model and is the model's to set. Folding both into computed + fields breaks the purity the server relies on; folding both into a derive action makes every form with a sum + declare an action. `evaluate` already serves the pure half. +- **Recomputing on the client by rules in the schema.** A lab computation is the domain's code, with its + standards' rounding, and is needed on the server to save anyway. The server runs it; the document's + expressions cover what is display only. +- **Deriving on every key.** A request per character, and values that change under the cursor while a number is + half typed. A commit is the moment a value is meant. +- **Loading derived values with `refill`.** `refill` keeps or replaces whole top-level fields, and a grid is one + such field: once any cell is typed, every derived member of every row would be kept as it was. Writing only + members the user cannot edit needs no comparison at all. +- **A cell kind chosen by the grid**, overriding the record's type. The same value would then encode one way in + the grid and another in the body. A member per kind keeps the record honest; the grid chooses which to show. +- **Holding focus on an invalid cell**, or a dialog on a refusal. The user loses the place they were going to, + and a dialog per cell turns a column of entries into a column of confirmations. +- **Skipping read-only cells by a per-cell flag.** Separate flags for read-only, its colour and its place in the + walk disagree the first time one is forgotten. +- **Pasting around read-only cells**, shifting values into the next editable cell. The pasted block then lands + in cells it did not come from. +- **Pasting into a model's table, one commit per cell.** Thousands of serial round trips, edits of rows the + client does not hold, and a refetch per change event; a bulk change is an action over a selection. +- **Holding an edited row still in a server-mode table.** The window would show a row at a place the server no + longer reports, which [`table.md`](table.md#invariants)'s window rules forbid; moving the walk by position + gives the entry behaviour without it. +- **Cell-selection corners by view position.** A sort would move the selection onto other rows. +- **Summary rows as rows of the data.** They would be sorted, filtered, selected and submitted. +- **Bands that move columns together.** A user reordering columns would drag several at once without having + chosen to; a band follows the layout instead. + +## Failure modes + +| Situation | Behaviour | +|---|---| +| A draft that does not encode | Kept, marked, its error in the tooltip and the status line; the form is not ready, or the cell is not committed. | +| An evaluation or a derive call fails | Derived cells keep their values, marked stale; the status line offers "Recalculate". | +| A reply arrives after a newer one | Ignored. | +| A derive reply names a path that is not a derived member | That value is dropped and reported once. | +| A column's kind and its edit field's kind differ, or a `member` that `values` lacks | `SourceContract` when the description is built. | +| A column `field` naming no member of the row type | Refused when the document loads; an expression naming none is reported once and the cell is read-only. | +| An edit of an unloaded cell, a busy or a locked row | Refused at the cell; nothing is sent. | +| A `FieldError` or `FailedRow` for an edited cell | The cell keeps the committed draft and shows the message. | +| A paste onto read-only or covered cells, or past the last row without `appendRows` | Those values are dropped; the status line says how many. | +| Insert or delete past `maxItems` or `minItems` | The entry is disabled and says why. | +| `summaries` or a group's summaries on a model-backed table | Refused when the document loads. | +| `selectionMode: "cells"` on a model's table | Refused when the document loads. | +| `editMode` on a table that is not a model's table | Refused when the document loads. | +| A span over a column the user moved or hid | Cut at that column. | +| A band nested past three levels, a band naming no column, a child band over a column its parent lacks, or a column in two bands of one level | Refused where the bands are read. | +| Two `layoutRows` parts with one key | Refused when the document loads. | +| `form` and `rows` both on a table, `layoutRows` with `rows` or `field`, or a `{field}` cell in a fixed grid without `form` | Refused when the document loads. | + +## Neighbouring specs + +This document is the design. The specs below describe the code, so each takes its part when the step that +builds it lands, and until then says nothing of it. + +| Spec | Takes | With step | +|---|---|---| +| [`table.md`](table.md) | Points here for editing; `TableColumn::edit` and `TableDescription::bands` among its members | 1 | +| [`../ui/view_tree.md`](../ui/view_tree.md), [`../ui/backend_contract.md`](../ui/backend_contract.md) | The focus chain and `focusNext` / `focusPrevious` | 1 | +| [`../forms/engine.md`](../forms/engine.md) | The grid widget leaves "Not supported"; per-cell snapshots, `modified`, `revert` | 1 | +| [`../forms/engine.md`](../forms/engine.md) | Rounds of derived values and `FormSession::loadDerived`; `deriveDebounce`; a `Choice` whose options depend on its row | 2 | +| [`../forms/forms.md`](../forms/forms.md) | `computedFields` on a row type, recomputed on every path, as an exception to "computed fields are top-level only"; the `derived()` metadata and `x-derived`, which `recomputeAll` does not touch | 2 | +| [`../core/registry.md`](../core/registry.md) | `DeriveOf` and `Derived`; `evaluate` recomputing row members | 2 | +| [`../ui/document.md`](../ui/document.md), [`../ui/interpreter.md`](../ui/interpreter.md) | The keys of [Document vocabulary](#document-vocabulary) on `table`, `form`, `group` and a field overlay, and their lowering | 1 to 6 | +| [`../qt_quick/backend.md`](../qt_quick/backend.md) | Editors inside cells, the cell and row marks, bands and spans | 1, 4 | + +## Delivery order + +| Step | Lands | Gates | +|---|---|---| +| 1 | The focus chain in the view tree. `FormRowsSource`; a grid of an `ObjectArray` field with the forms editors in cells, validation at the cell, read-only as one rule, the walk, per-cell modified marks, submission. Needs [`table.md`](table.md)'s steps 4, 6 and 9a. | "A grid is a form", "Read-only is one rule (grid)", "The walk" (its grid cases), "A sorted grid keeps its record", "Submission", "What changed" | +| 2 | Derived values: rounds over `evaluate` and the derive action, `loadDerived`, `computedFields` on row types, `derived()`; choices depending on their row; kind and overlays per row. | "Derived values", "The server decides", "Kind per row", "Choices per row" | +| 3 | Cell-range selection, paste, fill, clear; row operations. | "Cell selection", "Paste, fill, clear", "Rows of a grid" | +| 4 | Bands, spans, summary rows and group summaries. | "Bands", "Spans", "Summary rows" | +| 5 | Fixed grids, with blocks per row of an `ObjectArray`. | "Fixed grids" | +| 6 | Cells of a model's table: `CellEdit`, `RowEdit`, `row` mode, stale, modified and refused cells, drafts held by key. Needs [`table.md`](table.md)'s steps 4 and 9b (busy and locked rows, `FailedRow`). | "Cells of a model's table", "Read-only is one rule (model table)", "The walk" (its model-table case) | diff --git a/docs/spec/table/table.md b/docs/spec/table/table.md index 793deee39..83b32f8e7 100644 --- a/docs/spec/table/table.md +++ b/docs/spec/table/table.md @@ -3,8 +3,7 @@ Design spec for the one table every surface of morph uses: what a table is, how a model offers one or many, the single pipeline from a model's rows to the screen, slow columns and the caches that hold them, where sort and filter run, what a user can configure, and the performance all of it is held to. Read [`engine.md`](engine.md) -first, for the engine this builds on, and [`database_pages.md`](database_pages.md) for a model that pages in a -store. +first, for the engine this builds on. ## Contents @@ -25,8 +24,8 @@ store. - [Caching layers](#caching-layers) - [Where sort and filter run](#where-sort-and-filter-run) - [Configuring a table](#configuring-a-table) +- [Acting on rows](#acting-on-rows) - [Values with units](#values-with-units) -- [What a store cannot express](#what-a-store-cannot-express) - [Performance](#performance) - [Invariants](#invariants) - [API surface](#api-surface) @@ -94,6 +93,9 @@ struct TableColumn { bool sortable = true; bool filterable = true; bool shown = true; // the default visibility + ColumnDisplay display; // how a cell's value becomes its text; see "How a column shows its cells" + std::string tone; // the id of a companion column holding this cell's tone, or empty + std::string tooltip; // the id of a companion column holding this cell's tooltip, or empty }; struct ColumnGroupInfo { @@ -105,6 +107,7 @@ struct ColumnGroupInfo { struct TableCapabilities { bool keys = true; // keyed queries bool distinct = true; // distinct values + std::uint32_t maxSortKeys = 8; // set by the framework from `ServedTable::limits` }; struct TableDescription { @@ -113,8 +116,11 @@ struct TableDescription { std::vector columns; // in the declared order std::string key; // the id of the column that is the row's key std::vector groups; + std::vector bands; // header bands over columns: {label, columns, bands} (editing.md) std::vector levels; // every level a column names, base excluded std::vector units; // the units its `Measure` columns may hold + std::string rowTone; // the id of a companion column holding each row's tone, or empty + std::string rowLock; // the id of a companion column holding why a row is locked, or empty TableCapabilities capabilities; std::string revision; // set by the framework: a hash of everything above }; @@ -134,29 +140,10 @@ struct TableContext { session::Context const& session; // who asks }; -/// A table paged in a store, in place of rows held in memory (database_pages.md). -struct PlannedTable { - /// Runs a plan. For a whole plan `rows` is the page in order; for a prefilter plan it is the candidates. - std::function(QueryPlan const&)> run; - /// What the store can push down; see "What a store cannot express". - std::function translatable; - Residual residual = Residual::Refuse; -}; - -struct PlanRows { - std::shared_ptr rows; // columns as the description's - std::optional total; // a whole plan's count - TotalKind totalKind = TotalKind::Exact; - std::vector indices; // a keyed plan: each row's view row - std::optional distinct; // a pushed-down distinct plan -}; - struct ServedTable { TableDescription description; - /// The rows the caller may see, for a table held in memory or read lazily: `apply` runs over it. + /// The rows the caller may see, held in memory or read lazily: `apply` runs over it. std::shared_ptr source; - /// In place of `source`, for a table paged in a store. - std::optional planned; /// The slow levels, one per level the description names. std::vector> levels; /// Whether the caller may read a column; empty allows every column of the description. @@ -190,8 +177,8 @@ BRIDGE_REGISTER_TABLE(lims::DesignModel, "recipes", &lims::DesignModel::recipes, - **Services are the caller's.** `services` is what makes text order by the caller's locale. Left empty it is the default `Services`, whose collator orders by code point; a table with text columns that users sort sets a collator. Two callers with different collators have different `scope`s. -- **The row scope is the model's, and it is applied before the framework sees a row.** `source` and - `planned.run` return only rows the caller may see. Nothing downstream filters rows by principal: a level +- **The row scope is the model's, and it is applied before the framework sees a row.** `source` holds only + rows the caller may see. Nothing downstream filters rows by principal: a level cache, the view cache and a reply all work on the rows they are given, which is why a cached value cannot cross a scope. `scope` names the scope so that callers who see the same rows share a view. - **What a row scope costs.** A model whose callers all see the same rows returns one shared source. A model @@ -199,16 +186,21 @@ BRIDGE_REGISTER_TABLE(lims::DesignModel, "recipes", &lims::DesignModel::recipes, `table::scoped(source, keys)` makes one from a shared source and the scope's row keys, without copying a cell. Building a source per request at 100,000 rows is a full pass per request and defeats the view cache, whose key holds the source's snapshot. +- **Rows in a database are a source too.** A model whose rows live in a database serves them as a `DataSource` + it reads, whole or lazily by column, and the server sorts, filters, pages and counts them with `apply`. The + database is never asked to order or filter a view: one order on every surface needs one comparator, the + table's, and slow levels, unloaded cells, column groups and values with units are the engine's. - **A table is always C++ code.** Every table is a registered function of a C++ model; nothing on the server defines a table from a document or from data alone. What is generic is the table's shape, which that function decides when it runs. A model's set of table ids is fixed when it is registered; a table's columns are not. -**Two actions per table, made by the registration.** `BRIDGE_REGISTER_TABLE(Model, "designs", …)` registers: +**Actions per table, made by the registration.** `BRIDGE_REGISTER_TABLE(Model, "designs", …)` registers: | Action id | Takes | Returns | |---|---|---| | `Table.designs` | `TableQuery` | `GenericPage`: a page, a keyed reply or distinct values, as the query asks | | `Table.designs.describe` | nothing | `TableDescription` | +| `Table.designs.resolve` | `ResolveRequest` (a `RowSelection` and a `SelectionPolicy`) | `SelectedRows` ([Acting on rows](#acting-on-rows)) | One generic action taking a table id in its body was the alternative, and loses on every count that matters: @@ -224,10 +216,9 @@ A typed list action that takes a `TableQuery` and returns a `Page` remains with the description derived from the row type. **Changing a row is the application's action.** The registration makes no edit action. A model changes a row -through an action of its own, which declares its effect on the table and returns the keys it changed; a cell -editor on a screen is a column's `cell` template calling that action. A generic, description-driven cell edit -(an edit action per table, editable columns, an editor per kind) is a design of its own and is not in this -contract ([What this does not do](#what-this-does-not-do)). +through an action of its own, which declares its effect on the table and returns the keys it changed. A column +whose description names such an action (`TableColumn::edit`) is edited in place with an editor for its kind, +as [`editing.md`](editing.md#cells-of-a-models-table) specifies; the table's own actions change nothing. **The table's record.** Each registered table is a keyed record named `.
` (`DesignModel.designs`), registered by `BRIDGE_REGISTER_TABLE` beside the table's actions. A record is @@ -250,11 +241,16 @@ every watcher without keys, and each refetches its visible pages instead of its watcher the authoriser admits. ```cpp -struct ChangedRow { RowId key; }; +struct ChangedRow { RowId key; std::optional from; }; // `from`: a key the action changed +struct FailedRow { RowId key; std::string message; }; -/// The rows of one table that an action changed. Its JSON is {"rows": [{"key": …}, …]}. +/// The rows of one table that an action changed, and those it failed to. Its JSON is +/// {"rows": [{"key": …, "from": …}, …], "failed": [{"key": …, "message": …}, …]}. template -struct ChangedRows { std::vector rows; }; +struct ChangedRows { + std::vector rows; + std::vector failed; +}; struct RenameDesignResult { table::ChangedRows<&lims::DesignModel::designs> designs; @@ -272,6 +268,9 @@ BRIDGE_REGISTER_EFFECTS(lims::ImportDesigns, ::morph::effects::changesTable<&lim an action whose result has no `ChangedRows` is refused when the registries are resolved. - **An action that changes two tables returns two members**, one `ChangedRows` per table, and declares both effects. One untyped list of keys could not say which table a key belongs to. +- `from` and `failed` are what a screen shows of an action's outcome per row + ([Acting on rows](#acting-on-rows)). The walk collects a `from` as a changed key beside its `key`; a failed + row's key is not collected, since the action did not change it. - Watches, admission, fan-out, coalescing and `announce` are the record machinery's: a table is one more record. @@ -367,13 +366,101 @@ A page of a generic table is row arrays under one header: `DataSource` with every possible column, narrowed by `mayRead`. The level caches hold raw values for a column list that does not depend on the caller ([The server level cache](#the-server-level-cache)) and are shared. +### How a column shows its cells + +A cell crosses the wire as its exact value; the client turns it into text, in the session's locale, by the +column's display. Order, filters and distinct values always use the exact value, never the text. + +```cpp +/// The expression language's two modes and names (`"halfEven"`, `"halfUp"`), so a value a document rounds and +/// the same value a column shows agree. +enum class Rounding : std::uint8_t { + HalfEven, // 0.125 at 2 places is 0.12 + HalfUp, // half away from zero: 0.125 at 2 places is 0.13, the rule most test standards prescribe +}; + +struct ValueLabel { + std::string operand; // a value as a filter entry takes it: "true", "3", "V" + std::string text; // what the cell shows for it +}; + +struct ColumnDisplay { + /// Places shown for an `Integer`, `Decimal`, `Quantity` or `Number` cell. Absent shows the value's own places + /// (a `Rational`'s declared places, a `double`'s shortest round trip). + std::optional decimals; + Rounding rounding = Rounding::HalfEven; + /// With `decimals`, whether fewer places are padded with zeros: 2 places shows 1.5 as "1.50". + bool padPlaces = true; + /// Digit grouping by the locale: 12,500. + bool grouping = false; + /// What an empty cell shows: "–", "n/a". Empty shows nothing. + std::string empty; + /// The unit of a plain numeric column, shown in its header; a `Measure` column's units are its cells'. + std::string unit; + /// A `Date` or `DateTime` column's pattern; empty takes the session's. + std::string date; + /// Text for particular values: codes, booleans in words. A value no entry names shows as `decimals` says. + std::vector labels; +}; +``` + +- **Where it is formatted.** On the client, by `ui::formatCell(cell, column, locale)`, the one function every + backend, the copy, the status line and the filter menu call. The server sends no text for a value it can + send exactly; a model that formats on the server sends a `Text` column, which then sorts and filters as text. +- **Rounding is for showing.** 0.125 shown at two places under `HalfUp` is "0.13", and a filter + `eq 0.13` does not match it: the filter compares the exact value. A filter menu's condition field accepts + more places than the column shows for that reason. +- **The values list** formats each `DistinctValue` by the column's display. Two values that show alike (0.124 + and 0.123 at two places) stay two entries, because ticking one must not match the other; the menu shows the + value's full places for entries whose texts collide. +- **`labels`** change the text and nothing else. A column of `Bool` shows "available" and "not available", and + its values list shows those texts; it still orders false before true. Ordering by a label's text is a `sortBy` + over a column that holds it. +- **Empty, unloaded and invalid differ on screen.** `empty` is shown for an empty cell only. An unloaded cell + draws its placeholder ([Cells](#cells)), an invalid one an error mark whose tooltip says the value could not be + read. +- **A `Measure` column** shows each cell in its unit and to that unit's decimals ([Display](#display)), so + `decimals` and `unit` on it are a `SourceContract` error. +- **The header** shows `unit` after the label, as the backend draws a unit ("Mass [kg]" or the unit on a second, + muted line); the header's accessible name is "Mass, kg", and "Copy with headers" writes "Mass [kg]". +- **A document's column parts** `decimals`, `rounding`, `padPlaces`, `grouping`, `empty`, `unit`, `date` and + `labels` override the description's for that column, and are the only source on a table over `rows`. + +**Tone and tooltip.** A cell may carry a tone (`neutral`, `info`, `ok`, `warn`, `err`) and a tooltip, and a +row a tone. A model that decides them names **companion columns**: ordinary `Text` columns of the description, +usually `shown: false`, whose cell for a row is the tone's name or the tooltip's text. + +- `TableColumn::tone` and `tooltip` name a cell's companions, `TableDescription::rowTone` the row's. A + companion that names no column, or one that is not `Text`, is `SourceContract`. +- **They travel with their column.** A query that names a column names its companions too, and one that draws + any row names the `rowTone` companion, so a tone or a tooltip costs a cell of the rows in view and nothing for + columns out of the window. A companion may have a level, which makes an expensive tooltip a `display` level + loaded for the rows in view only. +- **A document overrides them.** A column's parts `tone` and `tooltip` are expressions in the row's scope; the + node's `rowTone` is one already. They win over the description's companions, whose columns are then not + fetched for them. A range rule ("weight ratio above 0.5 is `err`") is one of these two places: the model's + companion or the document's expression. The description holds no rule language of its own. +- **Drawing.** A cell with a tone is tinted with the theme's tone surface, and keeps its selection and focus + marks over it. A tooltip shows on hover after the theme's delay and on keyboard focus at once, and is the + cell's accessible description. A cell with a tooltip draws the theme's mark for "more here", so a user knows to + hover. +- **Companions are not columns a user arranges.** A column that the description names as a `tone`, `tooltip`, + `rowTone` or `rowLock` companion is never shown and is not listed in the column chooser or the header menus; + it sorts and filters only where its own `sortable` and `filterable` say so, through a document's `sort` or + `filters`. +- **An unknown tone name** shows no tone and is reported once for the column, as `rowTone` is. +- **A locked row.** A row whose `rowLock` companion holds text is locked: it draws the theme's locked hatch, its + text is the row's tooltip ("Being edited by another user"), and it takes no cell edit + ([`editing.md`](editing.md)). It can be selected and copied; whether an action accepts a locked row is the + action's rule, and a `FailedRow` says so when it does not. + ## The pipeline ```text in-document rows a collection's rows a model's table (a list of values) (the session's reply) (description; TableQuery → GenericPage) │ │ │ - ValueRowsSource VectorSource ViewCache, LevelCache, apply / planQuery ← server + ValueRowsSource VectorSource ViewCache, LevelCache, apply ← server │ │ │ pages, keyed rows, change events └────────── table::Engine ───┘ PageWindow ← client (client mode: sort, filter, (server mode: the pages around @@ -500,8 +587,11 @@ columns of which twenty are drawn converts twenty cells. expressions read: - the columns in the renderer's column window ([The renderer](#the-renderer)); -- for each of those, the columns its `cell` expression reads as `row.`, found when the document loads; -- the columns `rowTone`, `rowLabel`, `onActivate` and a row action read, for every row; +- for each of those, the columns its `cell`, `tone` and `tooltip` expressions read as `row.`, found when the + document loads, or, where the document gives none, its `tone` and `tooltip` companions + ([How a column shows its cells](#how-a-column-shows-its-cells)); +- the columns `rowTone` (or the description's `rowTone` companion), `rowLabel`, `onActivate` and the table's menu + entries read, for every row; - every column, when an expression reads the row in a way the loader cannot follow (`row[name]`, or the row passed whole to a function); the loader reports that once as a warning naming the node, since it turns the column window off for the wire. @@ -600,10 +690,10 @@ Windowing must not change what a keyboard or a screen reader can reach. (`setExistingKeys`), which a server-mode table does not have. There the backend reports the gesture as two view rows and the lowering resolves it through `TableRows::keysOf`: rows the window holds give their keys, and the rest are fetched as pages that name no columns, which carry keys alone. A range of more than - `SessionOptions::maxRangeRows` (10,000) rows is refused with a message in the status line. `Ctrl+A` is not - offered in server mode. -- **Selection in server mode** is a set of keys. It is not pruned by the filter, which the client cannot - evaluate; `selectedCount` counts the keys. + `SessionOptions::maxRangeRows` (10,000) rows is refused with a message in the status line. `Ctrl+A` selects + the view as a whole ([Acting on rows](#acting-on-rows)), which needs no keys. +- **Selection in server mode** is a set of keys, or the view less a set of keys. A set of keys is not pruned by + the filter, which the client cannot evaluate; `selectedCount` counts the keys, or the view's total less them. ## Every surface on the pipeline @@ -633,7 +723,9 @@ over `model`. | `pageSize`, `page` | A slice of the engine's view | Refused on a model-backed table, whose window pages. | | `group` (`by`, `heading`) | The run index is a hidden leading sort key computed by `ValueRowsSource`; headings are placed by the lowering after paging | Rows table only. | | `rowSpans` | The lowering, on the page's rows | Drawn only unsorted and below the windowing thresholds. Rows table only. | -| `selectionMode`, `selection`, `onSelectionChange`, `selectColumn`, `selectAllLabel`, `rowLabel` | The lowering, by key; select-all selects the view through `Selection::selectAll` | A model-backed table has ranges through `keysOf` and no select-all. | +| `selectionMode`, `selection`, `onSelectionChange`, `selectColumn`, `selectAllLabel`, `rowLabel` | The lowering, by key; select-all selects the view through `Selection::selectAll` | A model-backed table has ranges through `keysOf`, and its select-all is a selection of the view ([Acting on rows](#acting-on-rows)). | +| `cellMenu`, `headerMenu`, `retainChanged`, `headerEnabled`, `maxSortKeys` | The lowering ([Acting on rows](#acting-on-rows)) | Served on every table; `retainChanged` has no effect on a table that cannot retain ([Rows an action changed](#rows-an-action-changed)), and `maxSortKeys` is a `rows` table's only. | +| A column's `decimals`, `rounding`, `padPlaces`, `grouping`, `empty`, `unit`, `date`, `labels`, `tone`, `tooltip` | `ui::formatCell` and the mount ([How a column shows its cells](#how-a-column-shows-its-cells)) | Override the description's per column. | | `rowTone`, `onActivate` | The row's scope | Their columns are fetched for every row. | | `empty` | Shown while the view is empty | On a model-backed table it waits for the first reply, and is not shown while rows are undecided. | | `filters`, `onFiltersChange`, a column's `filterable` | `TableRows::setFilter`, as a `FilterSpec`; set from the header menu ([Filters from the header](#filters-from-the-header)) | Served on every table. | @@ -641,7 +733,7 @@ over `model`. | `status` | The status line ([The status line](#the-status-line)) | A literal: `true`, `false`, or absent for the default. | | `persist` | The table's persisted state ([Persisted state](#persisted-state)) | A literal. | | `highlightKey` | A keyed query for the row's view index, then `visible` around it | Model-backed only. | -| Element values `selection`, `selectedCount`, `sort`, `viewCount`, `pending`, `page`, `pageCount`, `pageFirst`, `pageLast` | `TableRows::counts` and the page slice | On a model-backed table `viewCount` is the reply's total, with `viewCountKind` (`exact` or `atLeast`) and `undecided` beside it. Every table has `layout`, `visibleColumns`, `hiddenCount`, `filters`, `awaiting` and `failed`. | +| Element values `selection`, `selectedCount`, `sort`, `viewCount`, `pending`, `page`, `pageCount`, `pageFirst`, `pageLast` | `TableRows::counts` and the page slice | On a model-backed table `viewCount` is the reply's total, with `viewCountKind` (`exact` or `atLeast`) and `undecided` beside it. Every table has `layout`, `visibleColumns`, `hiddenCount`, `filters`, `awaiting`, `failed`, `selectionSet` and `busyCount` ([Acting on rows](#acting-on-rows)). | A document that uses none of `layout`, `status`, `persist`, a column's `kind` or `locked` loads and orders as it does without them, with two visible differences that are intended: its tables have header and table menus, and @@ -760,8 +852,7 @@ the engine's rule in one sentence: a cell matches an operator only when the cell bound and `totalKind` is `AtLeast`. `AtLeast` therefore has two causes, the cap and undecided rows, and `Page::undecided` tells them apart: a client shows "100,000 or more" for the first and "12 so far, 40 loading" for the second. `apply` still omits `total` above `QueryLimits::totalCap`. A sort on an unloaded column changes no -membership and leaves the total `Exact`. The rule a store-backed model and the conformance helper follow is in -[`database_pages.md`, "Totals"](database_pages.md#totals). +membership and leaves the total `Exact`. ### Unique values @@ -875,6 +966,8 @@ struct TableQuery { /// The user asked for failed loads to be tried again: the level caches load the failed keys this query /// needs without waiting out their retry delay. bool retry = false; + /// Rows the view keeps although the filter rejects them (see "Acting on rows"). + std::vector retain; }; ``` @@ -963,6 +1056,10 @@ struct Page { std::vector indices; /// A distinct-values reply only; `rows` is then empty. std::optional distinct; + /// The indices of rows the view retains although the filter rejects them (see "Acting on rows"). + std::vector unmatched; + /// Changes whenever the view's membership may have changed; a selection of the view records it. + std::string viewRevision; }; ``` @@ -1000,7 +1097,7 @@ incomplete: an unloaded row is a row the server has sent. A query with `keys` set asks for specific rows of the view through the table's own action, so the model's scope and authorisation apply to it as to a page. -- The view is the one a page of the same sort and filter has. The reply's `rows` are the view's rows whose key +- The view is the one a page of the same sort, filter and `retain` has. The reply's `rows` are the view's rows whose key is in `keys`, in view order, with the query's `columns`; `indices` holds each one's view row; `unloadedRows` describes them. A key that is not in the view (the row was removed, is rejected, or is undecided) has no row in the reply. @@ -1014,11 +1111,7 @@ scope and authorisation apply to it as to a page. ([The window under changes](#the-window-under-changes)). - The server answers a keyed query from the view it has cached for the sort and filter ([The view cache](#the-view-cache)): a key's view row is a lookup, not a sort. -- **A store-backed model** pushes a keyed query down when its predicate accepts `translatable("", PlanUse::Keys)`: - it translates `QueryPlan::keys` and reports each row's view row. Otherwise the query is answered over the - store's candidates in C++, or refused when the model does not allow that - ([What a store cannot express](#what-a-store-cannot-express)). `standardCorpus` has a keyed query. A client - whose keyed fetch is refused with `Unavailable` falls back to `invalidate`. +- A client whose keyed fetch is refused with `Unavailable` falls back to `invalidate`. ## Arrival @@ -1391,8 +1484,10 @@ struct ViewCacheOptions { ``` - **Key.** The table's record, `ServedTable::scope`, and the validated query's sort and filter in canonical - form. `page`, `columns`, `keys` and `distinct` are not in it: every page, every column set and every keyed - fetch of one sort and filter read one entry. + form. `page`, `columns`, `keys`, `distinct` and `retain` are not in it: every page, every column set and every + keyed fetch of one sort and filter read one entry. A query with `retain` merges its retained rows into the + entry's view per request ([Rows an action changed](#rows-an-action-changed)), so sessions that retain + different rows still share one sort. - **Valid while** the table's source has the snapshot the entry was computed from and each of the view's order levels has the generation it read. A request finds an entry, compares those, and uses it or computes again. An entry is never repaired in place. @@ -1409,7 +1504,6 @@ struct ViewCacheOptions { - **What it does not hold.** Cells: a reply reads them from the source and the level snapshots. Distinct values: a values list runs the filter again without the column's own entry, and its reply is cached by the client's query cache. -- A table served through `planned` has no entry: the store orders it. A scoped table whose `scope` is left empty would share views between callers who see different rows. The framework cannot see that from the outside, so `table::rowScoped` and an empty `scope` together are refused @@ -1536,7 +1630,7 @@ from the action's shape: | The source | Mode | Sort, filter, totals, distinct values, the undecided set | Selecting all | |---|---|---|---| -| A model's table: its action takes a `TableQuery` (the schema carries `x-table`) | Server | On the server, once for each view and state of its data | Not offered on the client; an action over the query does it | +| A model's table: its action takes a `TableQuery` (the schema carries `x-table`) | Server | On the server, once for each view and state of its data | A selection of the view, which an action resolves on the server ([Acting on rows](#acting-on-rows)) | | Rows the client holds entirely: a document's `rows`, a collection's reply | Client | An `Engine` on the client | `Selection::selectAll` | A table is in server mode, and must be, when any of these holds: @@ -1655,8 +1749,11 @@ The engine, `TableQuery` and the document's `sort` take a list of keys; what a u down and remove, and "Add key". - Every header in the chain shows its direction; with more than one key each also shows its place in the chain, from 1. A header's accessible description says "sorted ascending, key 2 of 3". -- A chain has at most `QueryLimits::maxSortKeys` (8) keys in either mode. At the bound, adding a key is refused - with a message in the status line, not by dropping the first key. +- A chain has at most `maxSortKeys` keys: 8 by default, and up to 32 where a model's table's `limits` raise it, + which the description carries as `TableCapabilities::maxSortKeys` so the client knows the bound; a table over + `rows` takes it as the literal `maxSortKeys` on the node. A key costs one comparison only between rows that tie on every key before it, so a long + chain costs little more than a short one; the default is what a header shows legibly. At the bound, adding a + key is refused with a message in the status line, not by dropping the first key. `sort` and `onSortChange` keep their shapes. `Table::sort` carries the whole chain to the backend. @@ -1704,10 +1801,6 @@ struct FilterEntry { of one query together, and `maxQuerySetBytes` (1 MiB) their text; over any of them is `LimitExceeded`. The entry bound alone would allow a query of millions of values across its columns. When both the ticked and the unticked side exceed it, the menu disables the list and says so: the user narrows with a condition. -- In a plan it is `FilterOp::OneOf` with `PlannedCondition::values`; a store that accepts `FilterSet` translates - it as a set membership test, and otherwise it is applied in C++ - ([What a store cannot express](#what-a-store-cannot-express)). `standardCorpus` adds a `oneOf` query for each - column, in include and in exclude. **Distinct values.** @@ -1761,10 +1854,7 @@ distinctValues(DataSource const& source, TableQuery const& query, DistinctReques - **Levels.** The column's level is an `order` level of the request: it is needed for every candidate row. - **Client mode.** `Engine::distinctValues(request)` answers from the key columns the engine holds. - **Server mode.** `TableQuery::distinct` carries the request through the table's own action, so the model's - scope applies; the reply is a `Page` with no rows and `Page::distinct` set. A store-backed model pushes it down - as a grouped count or answers it over its candidates - ([What a store cannot express](#what-a-store-cannot-express)); where it refuses, the menu offers conditions - only. + scope applies; the reply is a `Page` with no rows and `Page::distinct` set. - **Validation.** `validate` holds a distinct request to the query's rules. Exactly one of `column` and `group` is set. The column, or every column of the group, exists and the caller may read it (`UnknownColumn`, `ColumnNotReadable`): a values list is a read of the column. `search` is at most `maxTextLength` bytes and @@ -1908,8 +1998,7 @@ with its own filter button. Its menu is a column's, with two additions: are one `oneOf` entry with a `match`; the last is one `eq` entry per ticked value in `together` mode, and is offered while at least 2 values are ticked and no more than the group has columns. -**On a server.** Over rows in memory, `apply` evaluates every form above. A model that pages in a store is -covered by the next section. +**On a server.** `apply` evaluates every form above. ### The status line @@ -1920,7 +2009,7 @@ by each backend from `Table::status`, and it is where every refusal this documen |---|---|---| | Counts, at the leading edge | The view's size and what is still coming, from `TableRows::counts` | "9,093 rows"; "100,000 or more"; "12 so far, 40 loading"; "3 of 9,093 selected" | | State | A level that failed, with "Retry"; a fetch that failed, with its error | "Recipe data could not be loaded. Retry" | -| Message, at the trailing edge | The latest refusal or notice, until the user's next gesture on the table or for six seconds | "A table sorts by at most 8 columns"; "Copied 200 rows, 12 cells were not loaded" | +| Message, at the trailing edge | The latest refusal or notice, until the user's next gesture on the table or for six seconds | "A table sorts by at most 8 columns" (the table's bound); "Copied 200 rows, 12 cells were not loaded" | - "Retry" sends the table's query again with `TableQuery::retry` set, which makes the level cache load the failed keys the reply needs at once ([The server level cache](#the-server-level-cache), rule 11). @@ -1966,7 +2055,7 @@ struct TableStateKey { std::string model; // the model's type; empty for a table over a document's rows std::string table; // the table's id; for a `rows` table, the node's id std::string qualifier; // the screen's id, or what the table's `persist` names - std::string part; // "state" or "views" + std::string part; // "state", "views" or "shared" }; struct StoredTableState { @@ -1991,6 +2080,8 @@ public: virtual async::Completion save(TableStateKey const& key, std::string json, std::string expected = {}) = 0; virtual async::Completion remove(TableStateKey const& key) = 0; + /// Whether the store keeps the `shared` part for everyone rather than for one person. + [[nodiscard]] virtual bool shares() const { return false; } }; ``` @@ -2013,7 +2104,7 @@ public: server ([The framework's stores](#the-frameworks-stores)); a save over a quota is `Refused`, and the table says so in its status line and keeps its arrangement. -**What is stored.** Two JSON documents per table, each with a version: +**What is stored.** JSON documents per table, each with a version: the arrangement and the saved views below, and the shared views, of the views' shape ([Saved views](#saved-views)): ```json { "version": 1, @@ -2026,11 +2117,12 @@ public: ``` ```json -{ "version": 1, "default": "Open orders", +{ "version": 1, "default": "Open orders", "defaultOnMount": false, "views": [{ "name": "Open orders", "layout": {}, "sort": [], "filters": {} }] } ``` -- `part: "state"` is the arrangement in use; `part: "views"` is the saved views. +- `part: "state"` is the arrangement in use; `part: "views"` is the saved views; `part: "shared"` is the views + published for everyone ([Saved views](#saved-views)). - `view` names the saved view the arrangement was taken from, or is absent. The menu shows the name, and shows it as modified when the arrangement differs from that view; nothing more is stored for it. - **Versions.** A document of an older version is brought forward by the framework, one version at a time, when @@ -2143,6 +2235,221 @@ and removes one, and marks one as the default. Applying one sets the three value they are bound, and records its name in the state's `view`. The view the `default` member names is applied when the table mounts and no `state` is stored for it. A stored view is fitted to the table as its state is. +- **The default first.** With `"defaultOnMount": true` in the views document, the default view's sort and + filters are applied at every mount, over those of the stored `state`; the stored layout still applies. It is + for a user who wants each session to start from the same order, and the menu sets it as "Start from the + default view". +- **Shared views.** A user may publish a view, under a name, for everyone who uses the table. Published views + are a third part, `part: "shared"`, of the same shape as `views`, which the store keeps for the application + and not for one person: a `TableStateKey` still names no person, and the store decides what the part means. + The table's menu lists them under the user's own, applies one as it applies one of the user's, and copies one + into the user's views with "Save as my view". + - **Who may publish.** Reading the part is the `morph.tableState` model's `load`, open to everyone who may use + the table. Writing it is a separate action of the model, `morph.tableState.publish`, carrying one operation + (add, replace, rename or remove one shared view), so the application's authoriser admits or refuses + publishing by action id, without reading a body. A user's own `save` of the `shared` part is refused. + - A shared view is changed under the `Conflict` rule, as a user's views are. The shared document has no + `default` and no `defaultOnMount`; they are the user's choice, kept in the user's own views document. + - **Which stores share.** `ITableStateStore::shares()` (default `false`) says whether a store keeps the + `shared` part for everyone. A table offers publishing only over a store that shares, so a host's own store + that does not never keeps a "shared" view for one person. `PreferencesTableStateStore` shares: it keeps the + part under the application's own scope in the preferences store, with no person. + +## Acting on rows + +A table is where a user picks the rows an application's action works on, and where they see what it did: a +selection that is the whole view, the application's own entries in the table's menus, rows an action changed +that stay in view, a row whose key an action changed, and rows an action is working on or failed on. + +### A selection of the view + +On a model-backed table the client holds a window, not the view, so "every row I see" cannot be a list of keys. +It is a description of the view, which the server turns into keys when an action runs. + +```cpp +/// What a user selected in one table. Its JSON is {"keys": […]}, or {"view": {…}, "except": […]}. +template +struct RowSelection { + struct View { + FilterSpec filters; // the view's filter, as the table queried it + std::vector retained; // the rows the view kept although they no longer match + std::string revision; // the view's `viewRevision` when the user selected it + }; + std::vector keys; // without `view`: exactly these rows + std::optional view; // with it: every row of that view, less `except` + std::vector except; +}; + +enum class SelectionPolicy : std::uint8_t { + Exact, // refuse when the view's rows may have changed since the user selected it + Current, // act on the view as it is when the action runs +}; + +struct SelectedRows { + std::vector keys; // in the table's source order + std::int64_t dropped = 0; // keys the caller may no longer see, left out +}; + +/// Resolves on the table executor; a Task handler awaits it. +template +[[nodiscard]] async::Completion> +resolve(Model const& model, RowSelection const& selection, TableContext const& context, + SelectionPolicy policy); + +/// The body of `Table..resolve`, for an action of another model or a task off the model's strand. +template +struct ResolveRequest { + RowSelection selection; + SelectionPolicy policy = SelectionPolicy::Exact; +}; +``` + +`RowSelection` is the value an action receives; `table::Selection`, the set of selected keys +([`engine.md`](engine.md)), is what a client-mode table holds while the user selects. + +- **Making one.** On a model-backed table `Ctrl+A`, the + select-all box and "Select all" in the table's menu select the view: no row is fetched. `Ctrl`+click then + toggles a row in `except`, and a `Shift` range adds or removes a range of `except` through `keysOf`. A plain + click replaces the selection with that one key. A table held whole on the client selects all by keys, since it + holds them. +- **Its count.** `selectedCount` is the view's total less the keys of `except` the window still sees in the + view; a key of `except` that has left the view is dropped from it when the window learns so. The count is + exact when the total is and no key of `except` is unaccounted for; the element value `selectedCountKind` + (`exact` or `approximate`) says which, and the status line shows an approximate count with "about". +- **Its life.** A view selection is of one filter. A change of the filter, or of the retained rows by "Show only + matching rows", ends it: the selection becomes empty. A change of the sort keeps it, since the rows are the + same. +- **Not while membership is open.** A view whose filter waits for a level does not yet know which rows are in + it, and "the rows I saw" says nothing about them. While `undecided` is above zero, or `awaiting` names a + level a filtered column needs, select-all is disabled and the status line says why: "Select all is available + once 40 rows are decided", or "… once recipe data has loaded" when no row is undecided yet. A level only the + sort needs does not disable it, since order is not membership. +- **The view's revision.** Every page reply carries `viewRevision`, which names the view's membership: two + replies with the same revision describe the same rows, and the sort is not part of it. It is a digest of the + keys of the view's rows, taken in source order when the view is computed, a pass the computation makes + anyway; the view cache keeps it with the entry, and the retained rows merged into a reply are added to it. An + edit that leaves every row in or out of the view as it was, the session's own previous action included, + leaves the revision as it was; an edit that brings a row in or takes one out changes it. +- **Resolving.** An action of the table's model takes a `table::RowSelection<&Model::designs>` member, and its + Task handler ([`../core/coroutines.md`](../core/coroutines.md#task-handlers)) awaits + `resolve(*this, selection, context, policy)`, which runs on the table executor. `resolve` calls the table's + function on that model for the caller, so the caller's row scope and `mayRead` apply, and computes the view's + membership: the filter's rows with an empty sort, which the view cache holds under that key when another + request computed it, and the retained rows merged in. An action of another model, or a server task, sends + `Table..resolve` with the selection and the policy instead, under its own principal. A key selection + resolves to the keys the caller can still see, counting the others in `dropped`. A view of more than + `QueryLimits::maxSelectedRows` (1,000,000) rows is `LimitExceeded`. +- **When the rows changed.** Under `Exact`, `resolve` refuses with `TableErrorCode::ViewChanged` when the + revision differs. The client then takes the selection again from the view as it is now (its filter, its + current revision and retained rows, the same `except`), says "The rows changed since you selected them: 9,120 + rows now. Run on these?", and sends the action again with the new selection when the user confirms. +- **The action's input says which table.** `RowSelection`'s schema carries `x-selectionOf` with the + table's record, so the client knows which table an action works on + ([Rows at work](#rows-at-work-rows-that-failed)). +- **One filter language.** A server task that works on "the rows matching a filter" builds a view selection and + resolves it under `Current`, so the rows a user filters to in a header and the rows a task selects on the + server are chosen by the same code. +- **Copying a selection of the view** fetches the view's rows as pages naming the visible columns, in view + order, less `except`, for at most `maxCopyRows` rows; over that the copy is refused, as any copy is + ([Copy](#copy)). + +In a document a table's element value `selectionSet` is the selection's JSON on every table, for an action's +body. `selection` is the list of keys, and is empty while the selection is a view. + +### The application's menu entries + +A table's built-in menus ([Configuring a table](#configuring-a-table)) take the application's entries after +their own: + +- `cellMenu` and `headerMenu` are parts of the `menu` node's entry shape: `{label, icon, keys, checked, enabled, + onClick, items, description, visible}` or `{separator: true}`. +- **A cell's menu** opens on a right-click, the menu key or a long press on a cell. It focuses the cell, and + selects its row alone when the row is not already selected. It shows the table's entries ("Copy", "Copy with + headers", "Filter by this value", "Exclude this value"), a separator, then `cellMenu`, read in the row's scope + with `column` (the column's id) and `value` (the cell) beside `row`. +- **A header's menu** shows its built-in entries, a separator, then `headerMenu`, read in a scope with `column`. + An entry's `visible` decides which columns it appears on: `{"fn": "eq", "args": [{"ref": "column"}, "kind"]}` + shows it on one. +- **"Filter by this value"** sets the column's filter to a `oneOf` of the cell's operand in `include`, replacing + that column's entries; "Exclude this value" adds the operand to the column's `exclude`. On an empty cell they + use `isEmpty`; on an unloaded or invalid cell they are disabled. On a column of a declared group the entry + "Filter by this value in any " sets the group's filter instead. +- An entry's `keys` work while the table has focus. The columns its expressions read are fetched for every row, + as `rowTone`'s are. + +### Rows an action changed + +**Kept in view.** A user who changes the rows they filtered to (replaces a component in the designs that hold +it) expects to see the result, not to watch the rows vanish because they no longer match. With `retainChanged` +(a literal flag, default `false`) a row an action of this session changed stays in the view while it no longer +passes the filter. + +- The retained rows are the keys of the `ChangedRows` the session's own actions returned for this table, since + the filter last changed. A change of the filter, or "Show only matching rows" in the status line, clears them. +- `TableQuery::retain` carries them, at most `QueryLimits::maxRetained` (10,000) in all. An action whose rows + would take the set past the bound adds none of them, and the status line says so; the rows retained before + stay. +- **The view.** The rows that pass the filter, and the retained rows the caller may see, in the sort's order. A + retained row whose filter is `Unknown` is in the view, not undecided. `Page::unmatched` holds the indices, + into the reply's `rows`, of its retained rows that the filter rejects. +- **On the server** the base view of the sort and filter is the view cache's shared entry; the retained rows not + in it are ordered by the sort's keys and merged into it per request, a cost of the retained rows and not of + the table. A keyed query sees the merged view; distinct values ignore `retain`, being the filter's. +- **In client mode** `Engine::retain(keys)` does the same; its filter run admits those keys. +- **Ordering with the event.** The action's change event can reach the client before the action's reply, whose + `ChangedRows` adds the keys. On a table with `retainChanged`, while an action of this session that declares an + effect on the table is in flight, the window holds back the refetches the table's change events cause, for at + most `SessionOptions::retainHold` (2 s), and sends one when the action settles or the hold runs out, with the + retained rows added. Within the hold a row never leaves the view and comes back; past it, other users' + changes are not held up by a long action. +- A retained row that does not match draws muted, with a tooltip saying it no longer matches the filter, and + the status line counts them: "3 rows shown that no longer match. Show only matching rows". +- Retained rows are not stored with the table's state. A view selection names them, so it resolves to the rows + the user saw. + +**A key that changes.** An action may give a row a new key: saving a modified design as a new one. The action +returns the row in its `ChangedRows` with `from`, the key the row had ([A model's tables](#a-models-tables)). + +- The result walk collects both keys, so the change event names both. +- The session that ran the action moves what it holds by key from `from` to `key`: the row's membership of the + selection, the focus, its place among the retained rows, its busy or failed state and its cell drafts + ([`editing.md`](editing.md)). Both keys are then fetched, since the old one may still be a row (the original + design, unchanged). +- Another session's change event names both keys without saying which became which, so there a selected or + focused old key behaves as any row that left the view: it is dropped from the selection, and focus moves as + [Keyboard, focus and assistive technology](#keyboard-focus-and-assistive-technology) says. + +### Rows at work, rows that failed + +While the session runs an action whose input names rows of a table on screen, the rows it names are **busy**: +for a `RowSelection` member, the keys of a key selection or the held rows of a view selection that `except` does +not name; the row of a `CellEdit` or a `RowEdit` ([`editing.md`](editing.md)); and the row of a +`table::RowRef<&Model::designs>` member, a row key whose schema carries `x-rowOf` with the table's record, for an +action that works on one row. + +- A busy row draws the theme's busy mark and takes no cell edit ([`editing.md`](editing.md)). It can still be + selected, copied and scrolled past. +- The element value `busyCount` counts them, and the status line says "Saving 40 rows". +- When the action settles, its rows stop being busy, whatever its outcome. + +An action that changes some rows and not others names the others in `ChangedRows::failed`, each with its +reason ([A model's tables](#a-models-tables)). + +- A failed row draws with the `err` tone and its message as the row's tooltip, until the next action of this + session on it, until a change event that names it arrives after the action settled (a failed row's key is not + among the keys the action's own event names), or until the user chooses "Clear marks" in the status line. A + row-scoped table's events name no keys, so there the mark lasts until the next action on the row or "Clear + marks". +- The status line says "12 rows could not be saved", with "Select them", which makes them the selection. + +### The header while the application is busy + +`headerEnabled` (a condition, default `true`): while it is false, the header's sort, filter and arrangement +gestures, and the menu entries that change them, are disabled. A gesture made anyway is refused with a message +in the status line. Selecting, scrolling, copying and the application's own entries stay available. A screen +that runs a long operation over a table's rows sets it, so the rows do not reorder or leave the view under the +operation. + ## Values with units A column can hold amounts in different units, and of different kinds of unit: 0.3 kg in one row, 200 g in the @@ -2285,93 +2592,6 @@ order and filters always use the exact amount. A `Measure` column aligns to the an `end` column does, with the symbol after the number, so amounts in one unit line up. Changing the display unit changes the cells' text and may widen the column once. -### Units on a server - -Over rows in memory `apply` does all of it. A store that keeps a base amount and a dimension beside each value -can order and filter by them: it accepts `translatable("", PlanUse::Measure)`, and a condition reaches it as the -operand's dimension and base amount (`PlannedCondition::dimension`, and `value` as a `Rational`). A store that -does not is answered through a prefilter ([What a store cannot express](#what-a-store-cannot-express)). -`standardCorpus` adds, for a `Measure` column, operands in a unit other than the cell's and of another -dimension. - -## What a store cannot express - -A store-backed model answers a query in two parts: the store narrows, and `apply` decides. `planQuery` pushes -down what the store can express; when something is left over, the model fetches the rows the pushed part admits -and the server runs the whole query over them in C++. A model that allows this (`Residual::Allow`) refuses -nothing merely because its store cannot say it. - -```cpp -enum class Residual : std::uint8_t { - Refuse, // a query the store cannot express whole is `Unavailable` - Allow, // it is answered by a prefilter and `finishPlan` -}; - -[[nodiscard]] std::expected -planQuery(TableQuery const& query, std::span columns, QueryLimits const& limits = {}, - std::function const& mayRead = {}, Services const& services = {}, - std::function const& translatable = {}, - Residual residual = Residual::Refuse); - -struct QueryPlan { - // … sort, filters, groups, offset, limit, totalCap, keys … - /// The plan is a prefilter: its rows are a superset of the view, unordered and unpaged, and `finishPlan` must - /// be called on them. - bool prefilter = false; -}; - -/// Over the candidates as a `DataSource`: what the framework calls with the rows a `PlannedTable` returned. -[[nodiscard]] std::expected -finishPlan(QueryPlan const& plan, TableQuery const& query, DataSource const& candidates, - ApplyOptions const& options = {}); - -/// The typed convenience: the rows through a `RowsSource`, the result copied into a `Page`. -template -[[nodiscard]] std::expected, TableError> -finishPlan(QueryPlan const& plan, TableQuery const& query, std::vector const& candidates, RowKey key, - ApplyOptions const& options = {}, /* kinds, comparators, loaded as for apply */ ...); -``` - -- **What `translatable` means.** It says what the store can push down: `translatable(column, use)` for `Sort` - and `Filter`, and for the uses a translator written for those two would not recognise (`Keys`, `Distinct`, - `FilterSet` for a `oneOf`, `GroupCount` for a `match` other than `any` or a group `exclude`, `Measure` for a - sort or filter on a column of values with units) a first question - with no column, `translatable("", use)`. A predicate that only looks at column names refuses an empty one, and - an empty predicate accepts none of them. -- **A whole plan.** When the store accepts every part of the query, the plan is as - [`database_pages.md`](database_pages.md) specifies: the store sorts, filters, pages and counts. -- **A prefilter plan.** When any part is not accepted and `residual` is `Allow`, the plan holds only the column - filters and `any` groups the store accepted, has no sort, `offset` 0 and `limit` of - `QueryLimits::maxCandidates + 1`, and `prefilter` is set. For a `together` group the plan adds a necessary - condition the store does know: one `any` group per entry, since every entry must match some column. The model - runs it and the rows go to `finishPlan`, called by the framework for a `PlannedTable`, which is `apply` over - them with the whole query: the parts the - store already applied are applied again, which changes nothing, so the answer is the oracle's by construction. - Sort, paging, totals, undecided rows, keyed replies and distinct values all come from that one call. -- **The bound.** `QueryLimits::maxCandidates` (100,000, the total cap) bounds the rows a prefilter may return. - `finishPlan` given more refuses with `Unavailable`, whose message asks for a narrower filter; a model may count - the prefilter first and refuse without fetching. Under the bound the total is exact, because every candidate - was seen. -- **Cost.** Each request reads the candidates from the store and runs `apply` over them: at 100,000 candidates - that is the store's read of 100,000 rows plus the figures of S1, S2 and G1 to G4, tens of milliseconds for the - C++ part (X1, measured over the fixture's rows in memory). The default bound is the size at which that part - stays within S1 and G4. A store's own read of the candidates comes on top, so a store-backed model whose reads - are slow sets a lower `maxCandidates`. A deep page costs what the first does; nothing is kept between requests. -- **When a model still refuses.** When its rows are costly to materialise by the hundred thousand, it keeps - `Residual::Refuse`, or lowers `maxCandidates`, and the menu does not offer what it refuses. -- **Distinct values** follow the same rule and meet the bound sooner: the column's own filter is removed, so with - no other filter the candidates are the whole table. A values list over more than `maxCandidates` rows is - refused and the menu offers conditions only; a store that can group pushes `Distinct` down instead. -- **A translator that knows only whole plans is never wrong.** `residual` defaults to `Refuse`, so a model that - does not ask for prefilters receives either a plan made only of what its predicate accepted, or `Unavailable`. - It never receives a - prefilter plan, and so never returns a superset as an answer. -- **Conformance.** `conformanceFailures` holds a model to `apply` whichever way it answered. For a prefilter - plan it also checks that the candidates contain every row of the oracle's view - (`ConformanceOptions::candidates`, the keys of the rows the model's prefilter returned), and it reports, per - use, how many queries - were pushed down and how many finished in C++, so a model that pushes nothing down is seen. - ## Performance The target is a table of 100,000 rows and 300 columns that stays fluent on the screen. Every layer has a named @@ -2600,14 +2820,6 @@ browser is measured when the frontend's cost is decided and not after everything | M4 | `distinctValues` on the column | ≤ 1.2 × D1 on a decimal column (P) | Grouping by shown text instead of key | | M5 | Registry lookups per cell while building keys | One, by id (gate) | The same as M1 | -### Benchmarks: a store's remainder - -| Id | Measures | Budget | Catches | -|---|---|---|---| -| X1 | `finishPlan` over 100,000 candidate rows of the fixture with a `together` group and a sort | ≤ S1 + G4 (P) | Work beyond one `apply` | -| X2 | Rows a prefilter plan admits for a `together` of two values, the plan applied to the fixture's rows, against the view's size | Recorded; the per-entry `any` groups are in the plan (gate) | A prefilter that narrows nothing | -| X3 | Queries pushed down and finished in C++, per use, over the corpus | Reported; a model's expected split is asserted in its own test | A model that silently stops pushing down | - ### Benchmarks: every surface The budgets above are for the pipeline, and every surface is on it, natively and in the browser. These cases hold @@ -2640,8 +2852,8 @@ pipeline, recorded once in step 0 and kept beside the budget it is held to after row `apply` puts in the view is in the view of the completed table, and a row it rejects is rejected there. 2. **Soundness of the order.** Two rows with no unloaded cell in any sort column have the same relative order before and after any completion. -3. **Counts.** view rows + undecided rows + rejected rows = source rows. `totalKind` is `AtLeast` whenever - `undecided > 0`. +3. **Counts.** view rows + undecided rows + rejected rows = source rows, a retained row counting in the view and + not as rejected. `totalKind` is `AtLeast` whenever `undecided > 0`. 4. **Unloaded is not empty.** `isEmpty` and `notEmpty` are `Unknown` on an unloaded cell; `displayText(Unloaded{})` is the empty string and is never used as a key. A cell whose load failed is unloaded. 5. **No unloaded cell at the base level.** One is refused as `SourceContract`. @@ -2649,7 +2861,8 @@ pipeline, recorded once in step 0 and kept beside the budget it is held to after cells only of the columns its query names; `levelsNeeded` names no level outside the query's sorted, filtered and named columns. 7. **Absent means loaded and whole.** A source that never produces `Unloaded`, columns with no `level`, a query - without `columns` and `keys`, and a `Page` without `unloadedRows`, `undecided`, `awaiting` and `failed` are a + without `columns`, `keys` and `retain`, and a `Page` without `unloadedRows`, `undecided`, `awaiting`, `failed` + and `unmatched` are a fully loaded table of every column: no cell is unloaded, no row is undecided, and every result is the one [`engine.md`](engine.md) specifies. 8. **A window's totals come from page replies only**, and a window never shows a row at a view row the server did @@ -2691,24 +2904,30 @@ pipeline, recorded once in step 0 and kept beside the budget it is held to after 24. **A group filter is its definition.** For every row with no unloaded cell, a `Columns` filter passes exactly when the count of satisfying columns meets `match`, and a `Together` filter exactly when an assignment of entries to distinct columns exists; `{columns, combine, include}` alone is the `any` case. -25. **A prefilter only narrows.** The rows of a prefilter plan contain every row of the view, and the answer is - `apply`'s over them. A model that did not ask for a prefilter never receives one. -26. **Units do not change an amount.** Two `Measure` cells of one dimension order and compare as their amounts in +25. **Units do not change an amount.** Two `Measure` cells of one dimension order and compare as their amounts in the base unit do, exactly; two of different dimensions are never equal and never ordered by amount. A unit's numeric id means the same unit in every registry of one lineage. -27. **A stored arrangement never breaks a table.** Whatever a store holds, the table shows at least one +26. **A stored arrangement never breaks a table.** Whatever a store holds, the table shows at least one column and applies only keys and filters on columns it has. -28. **State before rows.** A table with a store shows its first rows only after its stored state is applied or +27. **State before rows.** A table with a store shows its first rows only after its stored state is applied or its load has timed out or failed; a table whose load completes in time sends exactly one first query. -29. **One store.** Every table's arrangement is read and written through the injected `ITableStateStore` and +28. **One store.** Every table's arrangement is read and written through the injected `ITableStateStore` and nowhere else, no key it is given names a person, and no layer shortens what it is given. -30. **A cached view is the view.** A reply built from a view cache entry equals the reply `apply` would build +29. **A cached view is the view.** A reply built from a view cache entry equals the reply `apply` would build from the same source snapshot and level snapshots. -31. **A shift is seen.** After a keyed reply is patched, either every held row is at the view row the server +30. **A shift is seen.** After a keyed reply is patched, either every held row is at the view row the server has it at and the window's totals are the server's, or the window is invalidated. -32. **A refusal is shown.** Every refusal of a user's gesture (a ninth sort key, a set too large, a range or a - copy too long, a save refused) appears in the status line or, where a document hides it, in the session's - notifications. +31. **A refusal is shown.** Every refusal of a user's gesture (a sort key past the bound, a set too large, a + range or a copy too long, a save refused, a select-all while rows are undecided, a gesture on a disabled + header) appears in the status line or, where a document hides it, in the session's notifications. +32. **Text never orders.** A column's display (places, rounding, labels, empty text) changes what a cell shows + and nothing a sort, a filter, a distinct value or a selection uses. +33. **A selection of the view is the view.** Under `Exact`, `resolve` returns exactly the rows of the view the + user selected, less `except`, or refuses; under either policy it returns no row outside the caller's scope. +34. **A revision that holds means rows that held.** Two page replies with the same `viewRevision` describe views + of the same rows. +35. **Retaining adds, never removes.** A view with `retain` holds every row it holds without it, and the retained + rows the caller may see; `unmatched` names exactly the retained rows the filter rejects. ## API surface @@ -2720,21 +2939,24 @@ Headers marked (new) do not exist before the step that lands them ([Delivery ord | `table/sort.hpp` | `CellState::Unloaded`; `KeyColumn::read` maps `Unloaded`; `compareKeys` ranks valid, then empty and invalid, then unloaded. | | `table/filter.hpp` | `enum class Match { No, Yes, Unknown }`; `CompiledFilter::evaluate(Column const&, KeyColumn const&, std::uint32_t) -> Match`; `passes` is `evaluate == Yes`; `FilterRun::takeUndecided() &&`; `FilterEntry::oneOf`; `GroupMode`, `ColumnCount`; `GroupFilter::group`, `mode`, `match`, `exclude`; matching for `Together`. | | `table/engine.hpp` | `detail::JobInput::previousUndecided`, `detail::JobResult::undecided`; `Engine::undecidedRowCount()`, `undecidedRows()` (source rows), `onUndecidedChange(std::function)`; `Engine::distinctValues`; `ColumnLayout::setWidth`, `widths`, `setPinned`, `pinned`, `value`, `apply`, and `rebuild` placing a column by the fitting rule. | -| `table/query.hpp` | `TableQuery::columns`, `keys`, `distinct`, `retry`; `QueryLimits::maxColumns`, `maxKeys`, `maxSetValues`, `maxQuerySetValues`, `maxQuerySetBytes`, `maxCandidates`; `ColumnLevel` and the row type's `columnLevels()`; `LevelLoaded` as a trailing parameter of `RowsSource` and the typed `apply`; `LevelsNeeded`, `levelsNeeded`; `PageRows::totalKind`, `undecided`, `awaiting`, `failed`, `indices`; `UnloadedRow`, `UnloadedLevels`; `Page::unloadedRows`, `undecided`, `awaiting`, `failed`, `indices`, `distinct`; `DistinctRequest`, `DistinctValue`, `DistinctValues`, `DateBucket`, `distinctValues`; `FilterOp::OneOf`, `PlannedCondition::values`, `dimension`, `PlannedGroup::mode`, `match`, `exclude`; `PlanUse::Keys`, `Distinct`, `FilterSet`, `GroupCount`, `Measure`; `QueryPlan::keys`, `prefilter`; `Residual`, `finishPlan` over a `DataSource` and its typed overload; `changeKeys`, `rowIdOf`; `PageFetch::columns`, `revision`; `RowsFetch`, `Refresh`, `RefreshPlan`; `PageWindow`'s key function, `maxKeys` and `maxColumns`; `PageWindow::visible` with columns, `superseded`, `cancelled`, `refresh`, `owed`, `invalidate`, `patch`, `unloadedLevels`, `undecided`, `awaiting`, `failed`; the specialisation `PageWindow`. | -| `table/plan_conformance.hpp` | `ModelPage::undecided`, `unloadedRows`, `indices`; `standardCorpus` takes no operand from an unloaded cell and has keyed, `oneOf`, group, measure and distinct queries; `conformanceFailures` compares `undecided`, unloaded levels and keyed replies, and holds totals to the two-cause `AtLeast` rule; `ConformanceOptions::candidates` (row keys); the pushed-down and finished-in-C++ counts. | +| `table/query.hpp` | `TableQuery::columns`, `keys`, `distinct`, `retry`; `QueryLimits::maxColumns`, `maxKeys`, `maxSetValues`, `maxQuerySetValues`, `maxQuerySetBytes`; `ColumnLevel` and the row type's `columnLevels()`; `LevelLoaded` as a trailing parameter of `RowsSource` and the typed `apply`; `LevelsNeeded`, `levelsNeeded`; `PageRows::totalKind`, `undecided`, `awaiting`, `failed`, `indices`; `UnloadedRow`, `UnloadedLevels`; `Page::unloadedRows`, `undecided`, `awaiting`, `failed`, `indices`, `distinct`; `DistinctRequest`, `DistinctValue`, `DistinctValues`, `DateBucket`, `distinctValues`; `changeKeys`, `rowIdOf`; `PageFetch::columns`, `revision`; `RowsFetch`, `Refresh`, `RefreshPlan`; `PageWindow`'s key function, `maxKeys` and `maxColumns`; `PageWindow::visible` with columns, `superseded`, `cancelled`, `refresh`, `owed`, `invalidate`, `patch`, `unloadedLevels`, `undecided`, `awaiting`, `failed`; the specialisation `PageWindow`. | | `table/edits.hpp` | `CellEdits::commit` refuses an unloaded cell. | -| `table/description.hpp` (new) | `TableColumn`, `ColumnGroupInfo`, `TableCapabilities`, `TableDescription`, `TableContext`, `PlannedTable`, `PlanRows`, `ServedTable`, `GenericRow`, `GenericPage`, `ChangedRow`, `ChangedRows`, `TableOf`, `table::typed`, `table::scoped`, `table::rowScoped`, `table::derivedFrom`; the generic page's writer and reader. | +| `table/description.hpp` (new) | `TableColumn`, `ColumnGroupInfo`, `TableCapabilities`, `TableDescription`, `TableContext`, `ServedTable`, `GenericRow`, `GenericPage`, `ChangedRow`, `ChangedRows`, `TableOf`, `table::typed`, `table::scoped`, `table::rowScoped`, `table::derivedFrom`; the generic page's writer and reader. `ColumnDisplay`, `Rounding`, `ValueLabel`; `TableColumn::display`, `tone`, `tooltip`, `edit` and `EditInfo` ([`editing.md`](editing.md)); `TableDescription::rowTone`, `rowLock`, `bands` and `BandInfo`; `TableCapabilities::maxSortKeys`; `ResolveRequest` and the `Table..resolve` action; `ChangedRow::from`, `FailedRow`, `ChangedRows::failed`. | +| `table/row_selection.hpp` (new) | `RowSelection` and its schema's `x-selectionOf`, `SelectionPolicy`, `SelectedRows`, `resolve`, `ResolveRequest`; `RowRef` and its schema's `x-rowOf`. | +| `table/data_source.hpp`, `table/query.hpp`, `table/engine.hpp` (additions) | `TableErrorCode::ViewChanged`; the view cache entry's membership digest; `QueryLimits::maxSelectedRows`, `maxRetained`; `TableQuery::retain`, `Page::unmatched`, `Page::viewRevision` and their generic-page members; `Engine::retain`. | +| `table/query.hpp`, `table/filter.hpp` (removals) | `QueryPlan` and the planned types, `PlanUse`, `planQuery`, and `plan_conformance.hpp` whole: no table is paged in a database. | +| `ui/cell_format.hpp` (new) | `ui::formatCell`, the one formatter of a cell's text from its column's display and the session's locale. | | `table/level_cache.hpp` (new) | `LevelRead`, `LevelUse`, `LevelSnapshot`, `LevelSource`, `LevelCacheOptions`, `LevelCacheStats`, `LevelCache`, `CellLevelCache`. | | `table/view_cache.hpp` (new) | `ViewCacheOptions`, `ViewCacheStats`, `ViewCache`. | -| `table/state_store.hpp` (new) | `TableStateKey`, `StoredTableState`, `SaveOutcome`, `SaveResult`, `ITableStateStore`, `MemoryTableStateStore`; the state's and the views' readers, writers, version migration and fitting. | +| `table/state_store.hpp` (new) | `TableStateKey`, `StoredTableState`, `SaveOutcome`, `SaveResult`, `ITableStateStore` with `shares()`, `MemoryTableStateStore`; the state's and the views' readers, writers, version migration and fitting. | | `forms/views.hpp`, `forms/engine/collection.hpp` | The schema carries `x-level` from `columnLevels()` and `x-group` from `columnGroups()`; `deriveColumns` copies them; `CollectionColumn::level`. | | `core/change_types.hpp` | `effects::IChangeObserver` (`changed(Changes const&)`, `bind(AnnounceSink)`); `effects::AnnounceSink`. | | `core/remote.hpp` | `RemoteServer::announce`, `RemoteServer::attach(std::shared_ptr)`; `RemoteServerOptions::tableWork`, `views`. | -| `core/registry.hpp`, `core/effects.hpp` | `BRIDGE_REGISTER_TABLE` and its options; the actions `Table.` and `Table..describe`; the table's record; `effects::updatesTable`, `effects::changesTable`; the result walk's slot for `ChangedRows`, and its record of a holder's `openMembers()` and `partialMember()`; `ActionRecordInfo::open`, `partial`, `opaque`. `Page` declares both members. | -| `ui/table_rows.hpp` (new) | `SortChain`, `TableTotal`, `TableRowRef`, `TableCountsView`, `TableRows`, `EngineRows`, `WindowRows`, `ValueRowsSource`; `ui::parseMeasure`. | -| `ui/interpreter.hpp` | `Session::read`, `Session::watchRecord`, `WatchHolder`; the query cache's rules for open, partial and opaque entries; `SessionOptions::tableState`, `tableStateTimeout`, `tableStateDebounce`, `tableSettle`, `tableAwaitedRefetch`, `maxRangeRows`, `maxCopyRows`; the lowering of every table node onto `TableRows`; a table node's `status`, `persist`, a column's `kind` and `locked`. | +| `core/registry.hpp`, `core/effects.hpp` | `BRIDGE_REGISTER_TABLE` and its options; the actions `Table.` and `Table..describe`; the table's record; `effects::updatesTable`, `effects::changesTable`; the result walk's slot for `ChangedRows`, collecting each row's `key` and `from` and no `failed` key, and its record of a holder's `openMembers()` and `partialMember()`; `ActionRecordInfo::open`, `partial`, `opaque`. `Page` declares both members. | +| `ui/table_rows.hpp` (new) | `SortChain`, `TableTotal`, `TableRowRef`, `TableCountsView`, `TableRows`, `EngineRows`, `WindowRows`, `ValueRowsSource`; `ui::parseMeasure`; a selection of the view, retained, busy, failed and locked rows, and key moves by `from`, in both `TableRows`; `TableCountsView::retained`, `busy`; `SessionOptions::retainHold`; the element value `selectedCountKind`. | +| `ui/interpreter.hpp` | `Session::read`, `Session::watchRecord`, `WatchHolder`; the query cache's rules for open, partial and opaque entries; `SessionOptions::tableState`, `tableStateTimeout`, `tableStateDebounce`, `tableSettle`, `tableAwaitedRefetch`, `maxRangeRows`, `maxCopyRows`; the lowering of every table node onto `TableRows`; a table node's `cellMenu`, `headerMenu`, `retainChanged`, `headerEnabled` and element values `selectionSet`, `busyCount`; a column's display parts, `tone` and `tooltip`; a table node's `status`, `persist`, a column's `kind` and `locked`. | | `ui/view.hpp`, `ui/mount.hpp`, `ui/backend.hpp` | `Virtualization::columnThreshold`, `cellThreshold`, `columnOverscan`, `realizePerFrame`; `ColumnExtents`; the column window, cell pools and the frame budget; `Table::status`; `Backend::copyText`; the range-selection report by view rows. | -| `ui/table_state.hpp` (new), `ui/table_state_model.hpp` (new) | `PreferencesTableStateStore`; `TableStateModel` (`morph.tableState`), `TableStateOptions`. | +| `ui/table_state.hpp` (new), `ui/table_state_model.hpp` (new) | `PreferencesTableStateStore`; `TableStateModel` (`morph.tableState`) with its `publish` action for shared views, `TableStateOptions`. | | `qt_quick` and the terminal backend | Cells and header cells for columns in the window only; pinned columns; header menus, drag, resize, the sort dialog, the column chooser, the filter menu, the status line. | A caller that passes nothing new gets what it gets without this design, with one exception, stated where it @@ -2757,7 +2979,6 @@ Each case names the mutation it was seen to fail on, as the module's other tests | Invalidation faster than a round trip | Invalidate, fetch, invalidate again, then accept the first reply: the page shows the reply's rows and is still stale; `visible` returns one fetch, not two. Mutation: `accept` ignoring every older revision. | `test_table_server.cpp` | | `announce` | A derived record is included; more than `kMaxChangeKeys` keys become whole-type; an unregistered record is refused. `changeKeys` and `rowIdOf` round-trip both key kinds and agree with the record's key member. | `tests/net/test_push_unloaded_rows.cpp`, `test_table_query.cpp` | | `TableQuery` on the wire | `columns` and `keys`, integer and text, round-trip through glaze; a query without them reads with both absent. An unknown column in `columns` is `UnknownColumn` and an unreadable one `ColumnNotReadable`; an empty `columns` gives a page of keys. | `test_table_query.cpp` | -| Translation of a plan keeps the semantics | The in-memory reference model handles unloaded cells and keyed plans; `conformanceFailures` runs over a table with unloaded cells in each kind and reports a model that calls an undecided total `Exact`. lims's two store-backed models refuse `PlanUse::Keys`, and their conformance runs skip the keyed query. | `test_table_database_pages.cpp`; `examples/lims/tests` | | A value is loaded once | Two queries, then two clients over a socket, need the same 200 rows' level: `LevelCacheStats::loadedKeys` is 200 and the loader's own call log lists each key once. Mutation: `want` not checking the cache, which makes it 400 and 800. | `tests/table/test_table_level_cache.cpp`; `test_push_unloaded_rows.cpp` | | Concurrent queries share a load | Eight threads call `want` for the same keys while the loader is held on a latch: one loader call. Mutation: no in-flight set. | `test_table_level_cache.cpp` | | Reloaded once after invalidation | After `invalidate` of 10 keys the view lacks them; the next two queries load exactly those 10, once. An action declaring `updates` on the record does the same through the server with no call to `invalidate`. Mutation: `changed` ignoring the record, which leaves the old values served and the loader uncalled. | `test_table_level_cache.cpp`; `test_push_unloaded_rows.cpp` | @@ -2772,12 +2993,11 @@ Each case names the mutation it was seen to fail on, as the module's other tests | Columns by session | Two principals get different descriptions of one table; a page's `columns` are the caller's; a changed `revision` makes the client fetch the description again. | `tests/net/test_push_tables.cpp` | | The generic page | Every kind of cell round-trips: empty, unloaded, a quantity with a factor and with none, a measure; a typed table served through `table::typed` writes the same page as the generic source of the same rows. | `test_table_description.cpp` | | Column layout | A stored layout with unknown ids, a new column and every column hidden yields the fitted layout; hiding and moving create no cell widgets (gate D8). Mutation: `apply` ignoring `hidden`. | `tests/table/test_table_columns.cpp`; `tests/qt_quick` | -| Sort chains | `Shift`+click adds a key and flips one in the chain; a plain click replaces; a ninth key is refused. Mutation: `Shift` ignored. | `tests/test_ui_table_model.cpp`; `tests/qt_quick` | +| Sort chains | `Shift`+click adds a key and flips one in the chain; a plain click replaces; a key past the table's `maxSortKeys` is refused. Mutation: `Shift` ignored. | `tests/test_ui_table_model.cpp`; `tests/qt_quick` | | `oneOf` | It matches exactly the rows the OR of its `eq` entries matches, in include and exclude, with blanks beside it; an unloaded cell is undecided. Mutation: a `oneOf` that matches nothing. | `test_table_filter.cpp` | | Distinct values | Counts are those of the rows the other filters leave; the column's own filter is ignored; counts, `blanks` and `invalid` sum to those rows when `partial` is false; a group's "(Blanks)" count is the rows its `isEmpty` entry admits; unloaded rows set `partial`. Mutation: counting over the view. | `tests/table/test_table_distinct.cpp` | | Column groups | The truth table of [Column groups](#column-groups) row by row, including the row that tells `together` from `atLeast 2`; each quantifier with unloaded cells against its three-valued rule; an exclude. Mutation: `together` evaluated as `atLeast`. | `test_table_filter.cpp` | | Values with units | 300 g sorts between 0.2 kg and 0.5 kg in both directions; `gt "250 g"` drops 0.3 % rows; 0.3 % and 0.3 kg are not equal; 0.3 kg and 300 g are one distinct value; an operand with no unit is a ratio. Mutation: a key that ignores the unit's factor. | `tests/table/test_table_measure.cpp` | -| A store's remainder | A model on `Residual::Refuse` never receives a prefilter plan; with `Allow`, `finishPlan` over the candidates equals `apply` over all rows; a prefilter returned as the answer is reported; more than `maxCandidates` rows is refused. | `test_table_database_pages.cpp` | | Persisted state before the first query | A table mounts with a stored sort and filter in a `MemoryTableStateStore` whose load is held: no query is sent; released, one query goes out carrying the stored sort and filter. Mutation: a table that ignores its store sends the default query. | `tests/test_ui_table_model.cpp` | | Stale state is fitted | A stored state naming a removed column, a removed group, an unknown unit, a filter that no longer compiles and every column hidden: the table shows, with one column at least, the rest applied, each drop reported once; what a session's description lacks is written back unchanged. Mutation: applying the state unfitted. | `tests/table/test_table_state.cpp` | | Saves are coalesced | Fifty changes inside the debounce are one save holding the last arrangement; a change during a save in flight is one more; unmounting saves at once. Mutation: no debounce. | `tests/test_ui_table_model.cpp` | @@ -2809,6 +3029,21 @@ Each case names the mutation it was seen to fail on, as the module's other tests | State of any size | A state with a 20,000-value filter saves through `PreferencesTableStateStore` and reads back equal, as segments each within `segmentBytes`; a save stopped after its segments leaves the previous state readable; a missing segment is a failed load. Mutation: the store calling the preferences model, which refuses the value. | `tests/test_ui_table_state.cpp` | | Views are not lost between windows | Two windows each add a view from the same loaded revision: the second save answers `Conflict`, reloads, and both views are stored. A `state` save overwrites. | `tests/test_ui_table_state.cpp` | | Unit numbers across registries | A successor registry keeps every number and tombstones a removed unit; a level cache keeps its values under a successor and invalidates under another lineage; a unit id with a space is refused. | `tests/table/test_table_measure.cpp` | +| A column's display | 0.125 at two places shows "0.12" under `HalfEven` and "0.13" under `HalfUp`; `eq 0.13` matches neither; an empty cell shows `empty`, an unloaded one its placeholder; a `Bool` with labels orders false first; `decimals` on a `Measure` column is `SourceContract`; copy writes the formatted text. Mutation: a sort over the formatted text, which orders "10.00" before "9.00". | `tests/test_ui_cell_format.cpp`; `test_table_description.cpp` | +| Companion columns travel with their column | A page naming a column names its `tone` and `tooltip` companions and the `rowTone` one; a column out of the window fetches neither; a document `tone` expression stops the companion being fetched. Mutation: companions fetched for every column. | `test_table_description.cpp`; `tests/test_ui_table_model.cpp` | +| A selection of the view | `Ctrl+A` on a model-backed table fetches no row; `resolve` under `Exact` returns the view's keys less `except` and refuses `ViewChanged` after an edit that moves a row into the view; under `Current` it returns the new view; a principal's resolution holds no row outside its scope; select-all is refused while rows are undecided. Mutation: `viewRevision` that ignores the source snapshot, which resolves the edited view under `Exact`. | `tests/table/test_table_selection.cpp`; `tests/net/test_push_tables.cpp` | +| Retained rows | After an action changes three rows so that the filter rejects them, they stay in the view, listed in `unmatched` and counted; a change of the filter clears them; an action over `maxRetained` retains none and says so; a client-mode engine does the same. Mutation: the merged view written into the shared view cache entry, which serves one session's retained rows to another. | `test_table_selection.cpp`; `tests/test_ui_table_model.cpp` | +| A key that changes | A `ChangedRow` with `from` moves the selection, the focus and a failed mark to the new key and fetches both. Mutation: the walk ignoring `from`, which leaves the old row's watchers untold. | `tests/test_ui_table_model.cpp`; `test_push_tables.cpp` | +| Busy and failed rows | While an action over a selection runs, its rows draw busy and refuse an edit; a `failed` entry draws `err` with its message until the row's next change; "Select them" selects exactly the failed keys. Mutation: busy state cleared only on success. | `tests/test_ui_table_model.cpp` | +| The application's menu entries | `cellMenu` entries read `row`, `column` and `value`; a right-click on an unselected row selects it alone; "Filter by this value" sets a one-member `oneOf` and on an empty cell `isEmpty`; a `visible` on `column` shows an entry on one header only. Mutation: the cell menu read in the table's scope. | `tests/test_ui_table_model.cpp`; `tests/qt_quick` | +| Resolving from elsewhere | `Table..resolve` from another model's action returns the caller's keys and counts unseen keys in `dropped`; a view over `maxSelectedRows` is `LimitExceeded`; after `ViewChanged` the client offers the view as it is and resends on confirmation; a change of the filter ends a view selection. Mutation: `resolve` run under the server's principal, which returns rows outside the caller's scope. | `tests/net/test_push_tables.cpp`; `tests/test_ui_table_model.cpp` | +| The view revision is membership | An edit that changes a cell of a row in the view without moving it in or out leaves `viewRevision` as it was; one that brings a row in changes it. Mutation: the revision taken from the snapshot's version, which refuses the first case. | `test_table_selection.cpp` | +| Retained rows and the event | With `retainChanged`, an event that precedes its action's reply causes no refetch until the reply or `retainHold`. Mutation: no hold, under which a retained row leaves the view and comes back. | `tests/test_ui_table_model.cpp` | +| Locked rows and companions | A row whose `rowLock` companion holds text draws locked with that tooltip and refuses a cell edit; companion columns are absent from the column chooser and the header menus; an unknown tone is reported once. Mutation: the chooser listing every column of the description. | `tests/test_ui_table_model.cpp` | +| A raised sort bound | A table whose `limits` set 32 accepts a 32-key chain from the header and refuses the 33rd; the description carries the bound; `limits` above 32 is `SourceContract`. Mutation: the client holding 8 whatever the description says. | `tests/test_ui_table_model.cpp`; `test_table_description.cpp` | +| Shared views | A view published through `morph.tableState.publish` is read by a second principal and copied by "Save as my view"; a principal the authoriser refuses `publish` is refused; a store whose `shares()` is false offers no publishing; `defaultOnMount` applies the default view's sort and filters over the stored state at mount. Mutation: the shared part saved through `save`, which an authoriser cannot tell from a user's own state. | `tests/test_ui_table_state.cpp`; `tests/net/test_push_tables.cpp` | +| Copy of a selection of the view | A copy of a view selection less two keys writes the view's rows in order without them, fetched as pages; over `maxCopyRows` it is refused. | `tests/test_ui_table_model.cpp` | +| A disabled header | With `headerEnabled` false a header click, a filter menu and a column drag change nothing and each is refused in the status line; selection and copy still work. Mutation: `headerEnabled` read once at mount, which leaves the header enabled after the condition turns false. | `tests/test_ui_table_model.cpp` | The arrival path on a screen (a model-backed `table`, its repaint and its frame cost) is tested and measured with the lowering, as cases U6, F6 and F7 of [Performance](#performance). @@ -2824,25 +3059,28 @@ with the lowering, as cases U6, F6 and F7 of [Performance](#performance). wants exact row updates on arrival uses a `VectorSource`. - It does not page a forms collection or make its rows unloaded: a collection's rows arrive whole, and a collection whose `v-query` takes a query is a design for the forms specs. -- **It does not edit cells generically.** There is no edit action per table, no `editable` column and no editor - chosen by a column's kind. A row is changed by an action of the application that declares `updatesTable`, - called from a column's `cell` template or from the screen; `CellEdits` stays the client-mode helper a - collection uses. A description-driven inline edit needs its own client vocabulary, validation path and - delivery step, and is a spec of its own beside this one. +- **It does not edit cells.** Editing a cell in place, the grid that lays out a fixed form as rows and columns, + spans, header bands and summary rows are [`editing.md`](editing.md)'s, which builds on this pipeline. A row + of a model's table is changed by an action of the application that declares `updatesTable`. +- It does not format a value on the server for a client: the client formats from the exact value + ([How a column shows its cells](#how-a-column-shows-its-cells)). - **It does not export.** Copy puts selected rows on the clipboard ([Copy](#copy)); writing a view to a file is an application action over the query. - It does not group rows, aggregate, or total a model-backed table: `group` and `rowSpans` are a `rows` table's. - Aggregate reports over a store are [`database_pages.md`](database_pages.md#aggregate-reports)'s. -- It does not offer top-N or ranking filters, or colour and icon filters. +- It does not page, sort or filter a table in a database. A model with rows in a database reads them into a + `DataSource`; a report that aggregates over a database is the application's. +- It does not offer top-N or ranking filters, or colour and icon filters. A tone is shown, never filtered on; + a filter on what decides it is a filter on that column. - It does not keep loaded values across a server restart ([Caching layers](#a-persistent-tier)). - It does not repair a cached view when a level's generation changes: the view is computed again, once for all its watchers ([The view cache](#the-view-cache)). -- It does not cache a view for a table served through `planned`, or cache distinct values on the server. +- It does not cache distinct values on the server. - It does not invalidate a level cache in local mode, where no server derives changes: the application calls `invalidate` itself. - It does not lower a collection's or a wizard's chrome. - It does not turn a `rows` table bound to a page into a model-backed one. -- It does not give a model-backed table a client-side select-all, grouping or row spans; the node refuses them. +- It does not give a model-backed table grouping or row spans; the node refuses them. Its select-all is not + a client-side one: it is a selection of the view, resolved on the server. - It does not sort or filter held rows on the client in server mode ([Where sort and filter run](#where-sort-and-filter-run)). - It does not change `forEach` or `grid`. @@ -2938,15 +3176,14 @@ with the lowering, as cases U6, F6 and F7 of [Performance](#performance). `together`, since a cell has one value; for conditions it is two `any` groups, which AND already. - **Trying every assignment for `together`.** Sixteen entries over sixteen columns are 16! assignments; a matching is found in a few hundred steps. -- **Translating `together` for a store.** Without a matching it is a count per subset of the entries, which - doubles with each entry, so a translation is practical for about six entries. In C++ the bound is the - group's size. -- **Refusing whatever a store cannot express.** The feature a user sees would depend on the table's storage. A - prefilter keeps every feature and bounds its cost. -- **Streaming the store's rows through the remainder in order.** The store must then sort, so an unsortable - column refuses again, and paging and counting are written twice. One `apply` over the candidates is the - oracle itself. -- **A prefilter by default.** A model written for whole plans would return the superset as its answer. +- **Paging in the database**, the model translating each query into the database's own query and the server + answering only what the database cannot say. The table's order would then be the database's collation + wherever the query was pushed down and the engine's where it was not, so two tables, or one table before and + after a filter, would order the same text differently. Every feature (column groups, `together`, values with + units, unloaded cells and undecided rows, distinct values by fold) would depend on what each store can + translate, and every model would need a translator and a conformance run to show it agrees with `apply`. + A `DataSource` read from the database and `apply` over it give one order and every feature, at the cost of + holding a column's keys on the server while it is sorted or filtered, which the 100,000-row budget allows. - **A quantifier on the existing `combine`.** `combine` says how entries meet in one cell; the quantifier counts columns. One name for both leaves the fourth example unsayable. - **Groups as a merged virtual column.** Sorting and showing the columns one by one would be lost, and "exactly @@ -2977,6 +3214,20 @@ with the lowering, as cases U6, F6 and F7 of [Performance](#performance). - **The person in the key.** A client could ask for another person's state. - **Merging concurrent saves.** See [Persisted state](#persisted-state). - **Waiting for the state however long it takes.** A slow store would hold every table blank. +- **Text formatted on the server.** It cannot follow the user's locale or display unit without a request per + change, and a column sorted by its text orders "10.00" before "9.00". The exact value crosses; the client + formats it. +- **A rule language for tones in the description** (ranges, thresholds, colours). It would be a second expression + language beside the document's, evaluated on a client that may lack the values a rule needs. A companion + column computed by the model, or a document expression, says the same with what exists. +- **Select-all as a list of keys in server mode.** It fetches every key of up to 100,000 rows before the user + has done anything with them, and is stale by the time the action runs. The view's filter and revision are + what the user selected. +- **Retained rows frozen at their place.** A retained row that the sort would move is a row whose edit the + user cannot see take effect; retaining exempts it from the filter and leaves the order truthful. Holding a row + still is `ReorderPolicy::Deferred`'s, for the time an edit is in flight. +- **A rename as a removal and an insertion.** The selection, the focus and a failed mark would be lost on the + row the user is looking at. - **One `TableRows` implementation that holds either an engine or a window.** The two differ in whether a row can be absent, which every reader would then test; two implementations of one interface keep that in one place. @@ -3017,7 +3268,8 @@ with the lowering, as cases U6, F6 and F7 of [Performance](#performance). | A description with two columns of one id, a key that is no column, a level or group naming no column | The table's action answers `SourceContract`; nothing is shown. | | A principal may run a table's action but read none of its columns | A page with no columns; the table shows its empty state. | | A stored layout names a column the table lacks, or hides every column | The id is dropped; the first declared column is shown. | -| A ninth sort key | Refused, with a message in the table's status line. | +| A sort key past the table's `maxSortKeys` | Refused, with a message in the table's status line. | +| A table whose `limits` set `maxSortKeys` above 32 | `SourceContract` when the table is first served. | | A `oneOf` over `maxSetValues` | `LimitExceeded`; the menu sends the smaller side, and disables the list when both exceed it. | | A distinct request to a model that does not translate it | `Unavailable`; the menu offers conditions only. | | A values list while the column's level loads | The loaded values with `partial`; the list refreshes on each arrival. | @@ -3040,10 +3292,6 @@ with the lowering, as cases U6, F6 and F7 of [Performance](#performance). | A group filter naming a group the table lacks | `UnknownColumn`, naming the group. | | `together` with fewer than 2 entries, or more entries than the group has columns | `InvalidSpec`. | | `atLeast` or `exactly` with n below 0 or above the group's column count | `InvalidSpec`. | -| A part of a query the store cannot express, with `Residual::Allow` | A prefilter plan; `finishPlan` answers it. | -| The same with `Residual::Refuse` | `Unavailable`; the menu does not offer that part for the table. | -| A prefilter that returns more than `maxCandidates` rows | `Unavailable`, asking for a narrower filter. | -| A prefilter plan whose rows are returned as the answer | The model's conformance run reports rows the oracle lacks. | | A user-formed group loses a column | The column leaves the group; under two columns the group and its filter are dropped. | | A query's `columns` names a column the table lacks, or one the caller may not read | `UnknownColumn` or `ColumnNotReadable`; the client fetches the description again and asks once more. | | A column fetch whose keys are not the held page's | The rows moved between the two fetches; the window invalidates. | @@ -3061,12 +3309,24 @@ with the lowering, as cases U6, F6 and F7 of [Performance](#performance). | A level cache read under a registry of another lineage | The level is invalidated: its unit numbers mean nothing there. | | A `rows` table that sets `persist` and has no `id` | Refused when the document loads. Without `persist` it shows, and persists nothing. | | A refusal on a table whose document sets `status: false` | Shown through the session's notifications. | +| A companion column that names no column, or is not `Text`; `decimals` or `unit` on a `Measure` column | `SourceContract` when the description is built. | +| A companion's tone that is not a tone name | No tone; reported once for the column. | +| `resolve` under `Exact` after the view's membership changed | `ViewChanged`; the client keeps the selection and asks the user to confirm on the view as it is. | +| A selection of the view larger than `maxSelectedRows` | `LimitExceeded`. | +| Select-all while rows are undecided or a level is awaited | Refused, with a message in the status line. | +| An action that retains more rows than `maxRetained` | None of its rows are retained; the status line says so. | +| A `ChangedRow::from` the client does not hold | The new key is fetched as any changed key; nothing moves. | +| A gesture on a header while `headerEnabled` is false | Refused, with a message in the status line. | +| A `publish` the authoriser refuses, or a store whose `shares()` is false | Refused with a message; over a store that does not share, publishing is not offered. | +| A user's own `save` of the `shared` part | Refused. | +| A `rowLock` companion that names no column, or is not `Text` | `SourceContract` when the description is built. | ## Open questions -- **Grouping text in a store.** Distinct values and group filters on text compare by the collator's fold. A store - that cannot group by the same fold answers through a prefilter or refuses; what a given store can express is - found by that model's conformance run. +- **Retaining the rows of an action over a whole view.** `retain` carries keys, so an action that changes more + than `maxRetained` rows (a replacement over every visible design) retains none of them. Carrying a set the + server holds for the session, named in the query, would lift the bound; whether that is needed is settled by + the first application that meets it. - **A level cache in local mode.** With no server deriving changes, the application invalidates a cache itself. A hook for it belongs with local mode's effects. - **Frame timing in CI.** Whether `QQuickRenderControl` with the software backend gives stable step times on the @@ -3086,12 +3346,13 @@ section that does not exist yet. | [`../core/backend.md`](../core/backend.md), "Fan-out" | `RemoteServerOptions::tableWork`; `RemoteServer::announce`, `attach`, `IChangeObserver`; the view cache | 1, 2c | | [`../ui/interpreter.md`](../ui/interpreter.md), "Queries and the cache" | The rules for open, partial and opaque entries; `Session::read` and `watchRecord` | 2d | | [`../ui/interpreter.md`](../ui/interpreter.md), "Not yet lowered" and the rows' pipeline | The entry for a model-backed `table` goes; the lowering onto `TableRows` is described; the "Rows' pipeline" and "Groups" rules keep what the lowering does around the engine's order | 4, 6 | -| [`../ui/document.md`](../ui/document.md), the `table` kind | `table`, `status`, `persist`, `layout`, `configurable`, a column's `kind` and `locked`; `filters` on every table; the element values this design names | 4, 6, 7 | +| [`../ui/document.md`](../ui/document.md), the `table` kind | `table`, `status`, `persist`, `layout`, `configurable`, a column's `kind` and `locked`; `filters` on every table; `cellMenu`, `headerMenu`, `retainChanged`, `headerEnabled`; a column's display parts, `tone` and `tooltip`; the element values this design names | 4, 6, 7, 9a, 9b | | [`../ui/preferences.md`](../ui/preferences.md) | That table state has its own model and is not bounded by `maxValueBytes`; the `columns` and `saved` points | 4 | | [`../qt_quick/backend.md`](../qt_quick/backend.md) | The column window, cell pools, pinned columns, the status line; the menus and dialogs | 5, 7 | | [`../forms/views.md`](../forms/views.md) | `x-level` and `x-group` on a schema property | 2a, 3 | | [`engine.md`](engine.md) | Its "Not here yet" entries for the model-backed table and the single-threaded steps; `ColumnLayout`'s fitting rule | 4, 6, 7 | -| [`database_pages.md`](database_pages.md) | `PlannedTable`, prefilter plans and `finishPlan` | 1, 3 | +| [`engine.md`](engine.md), `database_pages.md` | `planQuery`, `QueryPlan` and the conformance helper leave `engine.md`; `database_pages.md` is removed | 0a | +| [`editing.md`](editing.md) | Editing cells in place, grids, spans, header bands and summary rows, on this pipeline | its own order | ## Delivery order @@ -3101,16 +3362,20 @@ gate runs as an ordinary test; a timing is recorded in [Performance](#performanc | Step | Lands | Gates and records | |---|---|---| | 0 | The fixture's rows, with nothing that does not exist yet: the column-store `DataSource` and its generator (`bench::WideTableSource`), and a baseline of what the code does today. The table tests are built for WebAssembly and run under Node. | Gate: S3. Recorded: S1 and S2 with every cell loaded; S7 as ten `apply` calls; S8; B1 at 10,000, 20,000 and 100,000 rows; B2; B5; U1 to U5; W7 as a typed page; F2 to F5 on a `rows` table of 2,000 × 300, the size a mount without a column window holds | -| 1 | The generic table contract: `TableDescription`, `ServedTable` with `source` and `planned`, `BRIDGE_REGISTER_TABLE`, its options and its two actions, the table's record with `updatesTable`, `changesTable` and `ChangedRows`, the generic page with every cell encoding, `TableQuery::columns`, `table::typed`, requests answered on `tableWork`. The fixture becomes the registered model `bench::WideTableModel` with its three tables and no slow level. | W1, W2, W5, W7; W6's bytes and codec | +| 0a | Store paging goes: `QueryPlan`, `planQuery`, `PlanUse` and `plan_conformance.hpp`, their tests, and the example models that page in SQLite, which serve their rows as a `DataSource` instead. | The table, net and example suites green without them | +| 1 | The generic table contract: `TableDescription`, `ServedTable` over a `source`, `BRIDGE_REGISTER_TABLE`, its options and its two actions, the table's record with `updatesTable`, `changesTable` and `ChangedRows`, the generic page with every cell encoding, `TableQuery::columns`, `table::typed`, requests answered on `tableWork`. The fixture becomes the registered model `bench::WideTableModel` with its three tables and no slow level. | W1, W2, W5, W7; W6's bytes and codec | | 2a | The engine's part of slow columns: the `Unloaded` cell and its rank, the three-valued filter and the undecided set, `ColumnInfo::level`, `levelsNeeded`, `LevelLoaded`, `unloadedRows` on `PageRows` and the typed `Page`. | S1 to S6 | -| 2b | The wire's part: `unloadedRows`, `undecided`, `awaiting`, `failed` and `indices` on the generic page; `TableQuery::keys` and the keyed reply with its totals; `PlanUse::Keys` and the conformance corpus. | W3, W4 | +| 2b | The wire's part: `unloadedRows`, `undecided`, `awaiting`, `failed` and `indices` on the generic page; `TableQuery::keys` and the keyed reply with its totals. | W3, W4 | | 2c | The server's caches and the push: `LevelSource`, `LevelCache` with leases, failures and immutable snapshots; `RemoteServer::announce` and `attach`; the view cache. The fixture gains its slow level. | L1 to L8, S7, S9, C1 to C3 | | 2d | The client's part: `PageWindow` under changes, with columns, probes and `superseded`; `Session::read` and `watchRecord`; the query cache's rules for open, partial and opaque entries. A table served over `apply` and read through a `PageWindow` then shows a first page before a slow level loads, shows its cells as unloaded, loads a level only when a named column needs it, and updates exactly the rows that arrived. | K1 to K4, K6, K7, E1 to E4 | -| 3 | The engine's configuring additions: `oneOf`, `distinctValues`, column groups with quantifiers and `together`, `Measure` and the unit registry, `ColumnLayout`'s widths, pinning, fitting rule and stored shape, `Residual` and `finishPlan`. | D1 to D6, G1 to G7, M1 to M5, X1 to X3 | +| 3 | The engine's configuring additions: `oneOf`, `distinctValues`, column groups with quantifiers and `together`, `Measure` and the unit registry, `ColumnLayout`'s widths, pinning, fitting rule and stored shape. | D1 to D6, G1 to G7, M1 to M5 | | 4 | `TableRows` and the lowering of a model-backed `table` onto `PageWindow`, from the description, with the status line's counts. The four behaviours of 2d then hold on a screen. `ITableStateStore`, the framework's stores and the load before the first rows land here, for sort and filters, so the rule is built into the first lowering. | U6 (E1 to E3 on a screen), K5, K8, F6, F7, T1, T3 to T5; W6's one request; F1 recorded | | 5 | The column window in the mount and in both backends: extents, pinned columns, cell pools, the frame budget, focus, keys and counts under windowing. **Exit criterion:** U8, the same scrolls in the WebAssembly build in a browser, within F3's and F4's frame budgets. | F1 to F5, F8 to F10, U7, U8, D8 | | 6 | `rows` tables on the engine through `ValueRowsSource`; the interpreter's own sort and page go. | U1 to U4; B1's 20,000-row step bound | | 7 | The configuration interface: header and table menus, column order, visibility, pinning and width, persisted with the rest of the table's state; sort chains; the filter menu with values, conditions, groups and units; copy; range selection in server mode. Screenshots of existing screens change here, as intended. | D7, D9, B3, B4 | | 8 | Collections through the same lowering. | U5 | -| 9 | Saved views, as the `views` part of the persisted state, with the conflict rule. | T2 | +| 9 | Saved views, as the `views` part of the persisted state, with the conflict rule; shared views and the default first. | T2 | +| 9a | A column's display and `ui::formatCell`; companion columns for tone and tooltip; header units. It may land before step 4; if it lands after, step 4's lowering formats through a stand-in that 9a replaces. | The display and companion tests of [Tests](#tests) | +| 9b | Acting on rows: `RowSelection`, `resolve` and `Table..resolve`, `viewRevision`, retained rows, `ChangedRow::from`, busy, failed and locked rows, the application's menu entries, `headerEnabled`. Needs step 4. | The selection, store selection, retained, key-change, busy, locked, menu and disabled-header tests of [Tests](#tests) | +| 9c | `maxSortKeys` configurable up to 32, carried in the description. | "A raised sort bound" | | 10 | As measured: repairing a cached view in place of computing it again, local work when a window holds the whole view, a persistent tier under the level cache. | The figures that justify each | diff --git a/docs/superpowers/plans/2026-11-17-one-table.md b/docs/superpowers/plans/2026-11-17-one-table.md new file mode 100644 index 000000000..b2209388c --- /dev/null +++ b/docs/superpowers/plans/2026-11-17-one-table.md @@ -0,0 +1,368 @@ +# The One Table, and Editing in It — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the table of `docs/spec/table/table.md`, a table as a run-time description any model offers, one pipeline for every surface, slow columns and their caches, configuration, persisted state and acting on rows, and the editing of `docs/spec/table/editing.md`, grids of forms, cells of a model's table and the layouts entry grids need. Along the way, remove paging in a database. + +**Architecture:** The table engine (`include/morph/table`) gains the description, unloaded cells, levels, the level and view caches, column groups, values with units and row selections; the UI layer (`include/morph/ui`) gains `TableRows` with its two implementations, cell formatting, the column window, the configuration menus, persisted state and grids; the forms engine gains derived values and per-cell snapshots. Every task follows the spec's **Delivery order** (table.md steps 0 to 9c, then editing.md steps 1 to 6), and each lands with the tests and mutations the spec's **Tests** tables name. + +**Tech Stack:** C++23, CMake (Ninja, MSVC preset `cl-debug`), header-only morph, glaze, Catch2, Qt 6 Quick (offscreen test executables only), the terminal backend, Qt WebAssembly (single-threaded) for the browser gates. + +**Spec:** `docs/spec/table/table.md` and `docs/spec/table/editing.md`. Each task quotes the section it implements; the executor reads that section before writing a line. Where this plan and the spec differ, the spec wins and the plan is corrected. + +## Global Constraints + +- AGENTS.md "Comments and documentation": a comment or spec states what the code does now and why; no history, no issue numbers, no commit hashes. +- Every public symbol has complete Doxygen `@param`/`@tparam`/`@return`; the docs build runs `WARN_AS_ERROR = FAIL_ON_WARNINGS` (`cmake -S . -B build/doc -G Ninja -DMORPH_BUILD_DOCUMENTATION=ON -DMORPH_BUILD_TESTS=OFF -DMORPH_BUILD_EXAMPLES=OFF && cmake --build build/doc --target doc`). +- morph is public: no company, product or customer names in code, tests, specs or commit messages; examples use the neutral `lims` domain. +- Commits are authored `yaraslau ` and end with the session's attribution lines. +- No GUI application is launched by hand or by an agent; Qt Quick behaviour is tested through `morph_qt_quick_tests` (offscreen). +- AGENTS.md "Verify rather than assert": every test names the mutation it was seen to fail on, and the task runs that mutation once and records the failing output in its commit message body. +- Limits are the spec's, verbatim: `Virtualization{threshold = 2000, overscan = 10, columnThreshold = 40, cellThreshold = 20'000, columnOverscan = 4, realizePerFrame = 240}`; `QueryLimits::maxColumns = 512`, `maxKeys = 1000`, `maxSetValues = 10'000`, `maxQuerySetValues = 20'000`, `maxQuerySetBytes = 1 MiB`, `maxGroupColumns = 16`, `maxSortKeys = 8` (up to 32 by `limits`), `maxSelectedRows = 1'000'000`, `maxRetained = 10'000`; `SessionOptions::maxRangeRows = 10'000`, `maxCopyRows = 10'000`, `maxPasteCells = 10'000`, `tableStateTimeout = 300 ms`, `tableStateDebounce = 500 ms`, `retainHold = 2 s`; `TableStateOptions::segmentBytes = 12 KiB`; `PageWindow(pageSize = 200, maxPages = 8, maxKeys = 1000, maxColumns = 96)`; a form's `deriveDebounce` default 150 ms. +- A spec that describes code (the "Neighbouring specs" tables of both documents) takes its part in the task whose step its row names, and says nothing of it before. + +## The standard task loop + +Every task below runs this loop; a task lists only what is specific to it. + +1. Read the spec sections the task names, and its rows of the spec's **Tests** table. +2. Write the task's tests (names and assertions as the task gives them, the spec's values in them). Build the test target and run the new cases: they fail, or do not compile, for the reason the task says. +3. Implement the interfaces the task lists, in the files it lists. +4. Run the new cases and the whole target they live in: all pass. +5. Apply each named mutation by hand, run its test, see it fail, revert. Paste the failing line into the commit body. +6. Update the neighbouring specs the task names. Run the docs build when the task adds a public symbol. +7. Commit. + +Commands, with `` and `` from the task: + +```bash +cmake --preset cl-debug # once per tree +cmake --build --preset cl-debug --target +ctest --test-dir build/cl-debug -R "" --output-on-failure +``` + +## Review Focus + +1. **A description that changes revision while state is held.** A view selection, retained rows, a focused cell or a cell draft held when the description takes a new revision (a column removed): the client refetches the description, fits its layout, drops what names a removed column, and never sends a query naming it. Test goes to Task 9 ("revision mid-selection"). +2. **Negative and boundary values in the display.** `-0.125` at two places under `HalfUp` is `-0.13`, under `HalfEven` `-0.12`; `0` with `empty` set shows `0`, not the empty text; a comma locale with grouping shows `-12.500,50`. Test goes to Task 15. +3. **An empty grid.** A grid over an `ObjectArray` with no rows: the walk with no editable cell lets `Tab` leave at once; a paste into it with `appendRows` adds rows, without it drops everything and says so; summaries over no rows show `count` 0 and an empty `mean`. Tests go to Tasks 19, 21 and 22. +4. **A key change onto a key the client already holds.** A `ChangedRow{key: 7, from: 3}` while row 7 is already held and selected: the selection holds 7 once, and the old row 3 is fetched again. Test goes to Task 17c. +5. **Bands under a stored layout.** A stored layout that hides every column of a band and then shows one again: the band reappears over that one column, with its unit; a band whose columns are all pinned draws only in the pinned part. Test goes to Task 22. + +--- + +## Phase A — the table (`table.md`) + +### Task 0: The fixture and the baseline (table.md step 0) + +**Spec:** "Performance" (the fixture, the data, how figures are taken), Delivery order step 0. + +**Files:** +- Create: `tests/table/wide_table_model.hpp`, `tests/table/wide_table_model.cpp` (static library `morph_wide_table_model`, linked by `morph_table_tests`, `morph_tests`, `morph_net_tests`, `morph_qt_quick_tests`) +- Modify: `tests/CMakeLists.txt`, `tests/table/bench_table.cpp`; the WebAssembly CI job that builds `morph_table_tests` and runs it under Node. + +**Interfaces:** +- Produces: `bench::WideTableOptions{std::int64_t rows; int columns; std::uint64_t seed; std::chrono::microseconds loadDelay;}`, `bench::WideTableSource` (a column-store `table::DataSource` over generated typed arrays, handing columns out in runs), with the column mix of "The data" exactly. + +- [ ] Tests: `"wide table: the same seed gives the same cells"` (two sources of seed 7 agree cell for cell over 1,000 × 300), `"wide table: the column mix is the spec's"` (counts per kind for 300 columns: 100 integers, 60 decimals, 30 quantities, 10 measures, 70 texts, 20 dates, 10 booleans; one column in twelve has 10% empty cells). Gate S3. +- [ ] Record the baselines step 0 names (S1, S2, S7 as ten `apply` calls, S8, B1 at 10,000, 20,000 and 100,000 rows, B2, B5, U1 to U5, W7 as a typed page, F2 to F5 on a `rows` table of 2,000 × 300) into table.md's Performance tables. +- [ ] Commit: `test(table): wide-table fixture and baselines`. + +### Task 0a: Remove paging in a database (table.md step 0a) + +**Spec:** "A model's tables" ("Rows in a database are a source too"), "Alternatives considered" ("Paging in the database"), Delivery order step 0a. + +**Files:** +- Delete: `include/morph/table/plan_conformance.hpp`, `tests/table/test_table_database_pages.cpp`, `examples/lims/org/include/lims/org/core/plan_sql.hpp`, `docs/spec/table/database_pages.md` +- Modify: `include/morph/table/query.hpp` (remove the "Query plans" section: `PlanUse`, the planned types, `QueryPlan`, `detail::planned*`, `planQuery`, and `QueryLimits` members only plans use), `include/morph/table/filter.hpp` (its comment naming `planQuery`), `CMakeLists.txt`, `tests/CMakeLists.txt`, `docs/spec/table/engine.md` (header table rows and the database bullet), `docs/spec/README.md` (its `database_pages.md` entry) +- Modify: `examples/lims/lab/{include/lims/lab/models/sample_model.hpp,src/models/sample_model.cpp}`, `examples/lims/ledger/{include/lims/ledger/models/ledger_model.hpp,src/models/ledger_entries.cpp}`, `examples/lims/tests/{test_sample_pages.cpp,test_ledger_entries.cpp,lims_test_support.hpp}`, `examples/lims/README.md` + +**Interfaces:** +- Consumes: `table::apply` over a `DataSource` (existing), `RowsSource` / `VectorSource` (existing). +- Produces: nothing new; the lims sample and ledger list actions read their scoped rows with one plain `SELECT` and answer with `apply`. + +- [ ] Before deleting `test_table_database_pages.cpp`, move every case that tests `apply` or `validate` rather than a plan into `tests/table/test_table_query.cpp`. +- [ ] In the lims tests, delete cases that test SQL translation; keep the cases that test what a user sees (order, filters, totals, scope, paging). Add `"ledger entries: a principal never receives a row outside its scope"` if no kept case pins it. Mutation: the model reading every row unscoped. +- [ ] `grep -rn "planQuery\|QueryPlan\|PlanUse\|plan_conformance\|standardCorpus\|database_pages" include src tests examples docs/spec` prints nothing. +- [ ] Targets `morph_table_tests`, `ladder_lims_tests`; all green. Docs build green. +- [ ] Commit: `refactor(table): remove paging in a database`. + +### Task 1: The description and the generic page (table.md step 1, first half) + +**Spec:** "A table is a description", "Rows on the wire", "A table on the server". + +**Files:** +- Create: `include/morph/table/description.hpp`; `tests/table/test_table_description.cpp` (in `morph_table_tests`) + +**Interfaces:** +- Produces: `TableColumn{ColumnInfo info; std::string label, group; bool sortable, filterable, shown;}` (the display, tone and tooltip members are added by Task 15, `edit` by Task 24), `ColumnGroupInfo`, `TableCapabilities{bool keys, distinct; std::uint32_t maxSortKeys = 8;}`, `TableDescription{id, label, columns, key, groups, levels, units, revision}` with `revision` filled by `table::describe(TableDescription)` from its canonical JSON; `GenericRow = std::vector`; `GenericPage{revision, columns, keys, rows, total, totalKind, offset}`; `writeGenericPage(PageRows const&, DataSource const&, std::span columns) -> GenericPage` and `readGenericPage(std::string_view json, TableDescription const&) -> std::expected`. + +- [ ] Tests: `"description: equal descriptions have equal revisions"`; `"generic page: every cell kind round-trips"` (empty, integer, date, `Rational`, `double` in a `Number` column, text, bool, `QuantityCell` with and without a factor; JSON per the "Rows on the wire" table, e.g. `[3, 10, "kg"]` read only in a `Measure` column once Task 8c lands); `"generic page: a cell that does not read as its column's kind is invalid"`. Gates W5, W7. +- [ ] Commit: `feat(table): table descriptions and the generic page`. + +### Task 2: Tables of a model on the bridge (table.md step 1, second half) + +**Spec:** "A model's tables", "The table's record", "Typed rows are a convenience". + +**Files:** +- Modify: `include/morph/table/description.hpp` (`TableContext`, `ServedTable`, `ChangedRow`, `ChangedRows`, `TableOf`, `table::typed`, `table::scoped`, `table::rowScoped`, `table::derivedFrom`), `include/morph/core/registry.hpp` and `include/morph/core/effects.hpp` (`BRIDGE_REGISTER_TABLE`, `effects::updatesTable`, `effects::changesTable`, the result walk's slot for `ChangedRows`), `include/morph/core/remote.hpp` (`RemoteServerOptions::tableWork`) +- Modify: `tests/table/wide_table_model.*` (`bench::WideTableModel` with tables `wide`, `small`, `scoped`; actions `EditWideRow`, `ReplaceWideRows`) +- Create: `tests/net/test_push_tables.cpp` (in `morph_net_tests`) +- Docs: `docs/spec/core/registry.md`, `docs/spec/core/backend.md` ("Fan-out") per table.md's Neighbouring specs, step 1. + +**Interfaces:** +- Produces: `BRIDGE_REGISTER_TABLE(Model, "id", &Model::fn[, options…])` registering `Table.` (`TableQuery` → `GenericPage`) and `Table..describe` (→ `TableDescription`) and the record `.`; `ServedTable{description, source, levels, mayRead, scope, services, limits}`. + +- [ ] Tests (spec rows "A table from a description", "A model with several tables", "Columns by session"): a model with no row type serves a table; two tables of one model are authorised and changed apart; `updatesTable` without `ChangedRows` is refused when the registries resolve; two principals get different descriptions. Mutations as the spec names. +- [ ] Commit: `feat(core): register tables of a model`. + +### Task 3: Unloaded cells in the engine (table.md step 2a) + +**Spec:** "The unloaded cell", "Semantics of an unloaded cell", "Data levels" (`ColumnInfo::level`, `levelsNeeded`, `LevelLoaded`). + +**Files:** +- Modify: `include/morph/table/data_source.hpp`, `sort.hpp`, `filter.hpp`, `engine.hpp`, `query.hpp`, `edits.hpp` +- Create: `tests/table/test_table_unloaded_cells.cpp`; extend `test_table_filter.cpp`, `test_table_sort.cpp`, `test_table_query.cpp` + +**Interfaces:** +- Produces: `struct Unloaded {}` as a `Cell` alternative (after `QuantityCell`), `CellState::Unloaded`, `enum class Match { No, Yes, Unknown }`, `CompiledFilter::evaluate(...) -> Match`, `ColumnInfo::level`, `LevelsNeeded{order, display}`, `levelsNeeded(query, columns, mayRead)`, `LevelLoaded`, `Engine::undecidedRowCount()`, `undecidedRows()`, `onUndecidedChange`, `PageRows::{totalKind, undecided, awaiting, failed}`, `TableErrorCode::SourceContract`. + +- [ ] Tests: spec rows (a), (b) ×3, (c) and the engine half of (d); invariant 1 and 2 properties over random tables and completions. Gates S1 to S6. +- [ ] Commit: `feat(table): unloaded cells and data levels in the engine`. + +### Task 4: Partial pages and keyed queries on the wire (table.md step 2b) + +**Spec:** "Partial pages", "Keyed queries". + +**Files:** `include/morph/table/query.hpp`, `description.hpp` (generic page members); tests in `test_table_query.cpp`, `test_table_description.cpp`. + +**Interfaces:** +- Produces: `TableQuery::{columns, keys}`, `Page::{unloadedRows, undecided, awaiting, failed, indices}`, `UnloadedRow`, `UnloadedLevels`, the same on `GenericPage`; `validate` bounds per the spec. + +- [ ] Tests: spec row "`TableQuery` on the wire"; a keyed reply carries the view's totals and each row's view row. Gates W3, W4. +- [ ] Commit: `feat(table): partial pages and keyed queries`. + +### Task 5: The level cache (table.md step 2c, first half) + +**Spec:** "The server level cache" (all eleven rules). + +**Files:** Create `include/morph/table/level_cache.hpp`, `tests/table/test_table_level_cache.cpp`; add the slow level `recipe` to the fixture. + +**Interfaces:** +- Produces: `LevelRead`, `LevelUse`, `LevelSnapshot`, `LevelSource`, `LevelCacheOptions`, `LevelCacheStats`, `LevelCache`, `CellLevelCache`. + +- [ ] Tests: spec rows "A value is loaded once" (cache half), "Concurrent queries share a load", "Reloaded once after invalidation", "A load in flight at invalidation", "Visible before told", "Eviction", "A level's layout is the cache's", "A cache's size is deep", "A reader blocks no one", "Order uses are leased", "A failed load". Gates L1 to L8. +- [ ] Commit: `feat(table): the level cache`. + +### Task 6: Announce, attach and the view cache (table.md step 2c, second half) + +**Spec:** "Arrival" steps 1 to 3, "The view cache". + +**Files:** Create `include/morph/table/view_cache.hpp`, `tests/table/test_table_view_cache.cpp`, `tests/net/test_push_unloaded_rows.cpp`; modify `include/morph/core/remote.hpp`, `include/morph/core/change_types.hpp`. + +**Interfaces:** +- Produces: `RemoteServer::announce(record, keys)`, `RemoteServer::attach(std::shared_ptr)`, `effects::IChangeObserver`, `effects::AnnounceSink`, `ViewCacheOptions{maxBytes = 64 MiB}`, `ViewCacheStats`, `ViewCache`; `table::changeKeys`, `table::rowIdOf`. + +- [ ] Tests: spec rows "`announce`", "A cached view is the view", "One sort for many watchers", "No value crosses a scope", "A row-scoped table", and the bridge halves of (a), (c), (d). Gates S7, S9, C1 to C3. +- [ ] Commit: `feat(net): announce arrivals and cache views`. + +### Task 7: The window under changes (table.md step 2d) + +**Spec:** "The window under changes", "The client's caches". + +**Files:** `include/morph/table/query.hpp` (`PageWindow`), `include/morph/ui/interpreter.hpp` (`Session::read`, `Session::watchRecord`, the query cache's open, partial and opaque entries); tests `tests/table/test_table_server.cpp`, `tests/test_ui_interpreter_cache.cpp`. + +**Interfaces:** +- Produces: `PageFetch{request, columns, query, revision}`, `RowsFetch`, `Refresh`, `RefreshPlan`, `PageWindow(pageSize, maxPages, key, maxKeys, maxColumns)` with `visible(first, count, columns)`, `superseded()`, `cancelled`, `refresh(keys)`, `owed()`, `invalidate()`, `accept`, `patch`; `PageWindow`. + +- [ ] Tests: spec rows "An event that precedes its page reply", "An ordinary edit on the same record", "Invalidation faster than a round trip", "A reply holds the named columns only", "A row the window never held", "An open entry hears every change", "A partial entry is not merged into", "An opaque entry", "A read with no entry", the window half of (d). Gates K1 to K4, K6, K7, E1 to E4. +- [ ] Commit: `feat(table): the page window under changes`. + +### Task 8: Filters, groups, units and layout in the engine (table.md step 3) + +Four commits, one per subsystem, each its own reviewer gate. + +**8a — `oneOf` and distinct values.** Spec: "Filters from the header" (the value-set operator, distinct values). Files: `filter.hpp`, `engine.hpp`, `query.hpp`; `tests/table/test_table_distinct.cpp`. Produces: `FilterEntry::oneOf`, `DistinctRequest{column, group, search, bucket, offset, limit = 200}`, `DistinctValue{text, operand, count}`, `DistinctValues{values, distinct, distinctKind, blanks, invalid, partial}`, `DateBucket`, `distinctValues(source, query, request, options)`, `Engine::distinctValues`, `TableQuery::distinct`, `Page::distinct`. Tests: spec rows "`oneOf`", "Distinct values"; invariants 22, 23. Gates D1 to D6. Commit `feat(table): value sets and distinct values`. + +**8b — Column groups.** Spec: "Column groups". Files: `filter.hpp`, `data_source.hpp` (`DataSource::groups()`), `forms/views.hpp` (`x-group`). Produces: `GroupMode{Columns, Together}`, `ColumnCount{Any, All, AtLeast, Exactly; n}`, `GroupFilter{group, columns, mode, match, combine, include, exclude}`, the augmenting-path matching for `Together`. Tests: spec row "Column groups" (the truth table row by row, the three-valued table with unloaded cells); invariant 24. Gates G1 to G7. Commit `feat(table): column groups with quantifiers`. + +**8c — Values with units.** Spec: "Values with units". Files: `data_source.hpp`, `sort.hpp`, `filter.hpp`; `include/morph/ui/measure.hpp` (`parseMeasure`); `tests/table/test_table_measure.cpp`. Produces: `UnitInfo`, `UnitId`, `UnitRegistry` with `lineage()`, `UnitRegistryBuilder{previous}`, `addUnitsOf`, `Measure`, `ColumnKind::Measure`, `ColumnInfo::dimensions`, `Services::units`. Tests: spec rows "Values with units", "Unit numbers across registries"; invariant 25. Gates M1 to M5. Commit `feat(table): values with units`. + +**8d — Column layout.** Spec: "Columns: order, visibility, pinning, width" (the fitting rule). Files: `engine.hpp` (`ColumnLayout`); `tests/table/test_table_columns.cpp`. Produces: `ColumnLayout::setWidth(id, std::optional)`, `widths()`, `setPinned(n)`, `pinned()`, `value()`, `apply(value)`, `rebuild` placing a new column after its declared predecessor. Tests: spec row "Column layout" (the engine half). Commit `feat(table): widths, pinning and fitting in the column layout`. + +### Task 9: `TableRows` and the model-backed table on a screen (table.md step 4) + +**Spec:** "The client table model", "The document `table` node on the engine", "The status line" (counts), "Persisted state" (the interface, the framework's stores, loading before the first rows, saving for sort and filters). + +**Files:** +- Create: `include/morph/ui/table_rows.hpp`, `include/morph/table/state_store.hpp`, `include/morph/ui/table_state.hpp`, `include/morph/ui/table_state_model.hpp`; `tests/test_ui_table_model.cpp`, `tests/test_ui_table_state.cpp`, `tests/table/test_table_state.cpp` +- Modify: `include/morph/ui/interpreter.hpp` (the lowering of a model-backed `table` onto `WindowRows`; `SessionOptions::{tableState, tableStateTimeout, tableStateDebounce, tableSettle, tableAwaitedRefetch}`) +- Docs: `docs/spec/ui/interpreter.md`, `docs/spec/ui/document.md`, `docs/spec/ui/preferences.md`, `docs/spec/table/engine.md` per step 4. + +**Interfaces:** +- Produces: `ui::SortChain`, `TableTotal`, `TableRowRef`, `TableCountsView`, `class TableRows` (pure virtual, members as the spec's block), `EngineRows`, `WindowRows`; `TableStateKey{application, model, table, qualifier, part}`, `StoredTableState`, `SaveOutcome`, `SaveResult`, `ITableStateStore{load, save, remove}`, `MemoryTableStateStore`, `PreferencesTableStateStore`, `TableStateModel` (`morph.tableState`). + +- [ ] Tests: spec rows "Persisted state before the first query", "Stale state is fitted", "Saves are coalesced", "A slow or failed load", "The person is the store's", "Bound values win", "State of any size", "A scroll that passes", "An awaited order is throttled", "The status line" (counts). Review Focus 1: `"table rows: a new description revision drops what names a removed column"` (a held view selection and a cell draft naming column `c` are dropped when `c` leaves the description; the next query does not name `c`). Gates U6, K5, K8, F6, F7, T1, T3 to T5; W6's one request; F1 recorded. +- [ ] Commit: `feat(ui): table rows, model-backed tables and stored state`. + +### Task 10: The column window (table.md step 5) + +**Spec:** "The renderer" (all subsections). + +**Files:** `include/morph/ui/view.hpp`, `include/morph/ui/mount.hpp` (`Virtualization` additions, `ColumnExtents`, cell pools, the frame budget, focus by key and column), `include/morph/ui/backend.hpp` (the range-selection report by view rows); the Qt Quick `MorphTable` and the terminal table; tests `tests/test_ui_mount.cpp`, `tests/qt_quick/`. Docs: `docs/spec/qt_quick/backend.md`. + +- [ ] Tests: spec rows "Windows of the renderer", "Keyboard and focus under windowing", "Range selection in server mode", "Column layout" (gate D8: hiding and moving create no cell widgets). Gates F1 to F5, F8 to F10, U7, D8. **Exit criterion:** U8, the same scrolls in the WebAssembly build within F3's and F4's budgets. +- [ ] Commit: `feat(ui): the column window`. + +### Task 11: Tables over `rows` on the engine (table.md step 6) + +**Spec:** "A document's rows as a source", "What the interpreter does for a table". + +**Files:** `include/morph/ui/table_rows.hpp` (`ValueRowsSource`), `include/morph/ui/interpreter.hpp` (the interpreter's own sort and page go); tests `tests/test_ui_table_model.cpp`, `tests/test_ui_table.cpp`. + +- [ ] Tests: spec rows "A document's rows as a source", "One order on every surface". Gates U1 to U4; B1's 20,000-row step bound. +- [ ] Commit: `feat(ui): tables over rows on the engine`. + +### Task 12: The configuration interface (table.md step 7) + +**Spec:** "Configuring a table" (columns, sort chains, filters from the header, column groups' menu, copy), "Keyboard, focus and assistive technology" (range selection in server mode). + +**Files:** `include/morph/ui/interpreter.hpp` (`layout`, `onLayoutChange`, `configurable`, `locked`, `status`, `persist`, a column's `kind`), the Qt Quick and terminal backends (header and table menus, drag, resize, the sort dialog, the column chooser, the filter menu, the status line), `Backend::copyText`; tests `tests/test_ui_table_model.cpp`, `tests/qt_quick/`. + +- [ ] Tests: spec rows "Sort chains", "Copy", "The status line" (messages and refusals), the menu halves of "Column layout", "Distinct values" and "Column groups". Gates D7, D9, B3, B4. Screenshot baselines of existing screens change here, as intended; regenerate them in this commit. +- [ ] Commit: `feat(ui): configure a table from its header`. + +### Task 13: Collections through the same lowering (table.md step 8) + +**Spec:** "Collections and master-detail". + +**Files:** `include/morph/forms/engine/collection.hpp`, `include/morph/ui/interpreter.hpp`; tests `tests/test_ui_table_model.cpp`. + +- [ ] Tests: a collection and a `rows` table over the same rows show the same order and selection. Gate U5. +- [ ] Commit: `feat(ui): collections on the table pipeline`. + +### Task 14: Saved and shared views (table.md step 9) + +**Spec:** "Saved views", "Persisted state" ("Two windows, two devices", shared views, `defaultOnMount`). + +**Files:** `include/morph/table/state_store.hpp` (`ITableStateStore::shares()`), `include/morph/ui/table_state_model.hpp` (`morph.tableState.publish`); tests `tests/test_ui_table_state.cpp`, `tests/net/test_push_tables.cpp`. + +- [ ] Tests: spec rows "Views are not lost between windows", "Shared views". Gate T2. +- [ ] Commit: `feat(ui): saved and shared table views`. + +### Task 15: How a column shows its cells (table.md step 9a) + +**Spec:** "How a column shows its cells". + +**Files:** `include/morph/table/description.hpp` (`ColumnDisplay`, `Rounding{HalfEven, HalfUp}`, `ValueLabel`; `TableColumn::{display, tone, tooltip}`; `TableDescription::{rowTone, rowLock}`), create `include/morph/ui/cell_format.hpp` (`ui::formatCell(Cell const&, TableColumn const&, Locale const&) -> std::string`), the mount and backends (tone surfaces, tooltips, the "more here" mark, header units); tests `tests/test_ui_cell_format.cpp`, `test_table_description.cpp`, `tests/test_ui_table_model.cpp`. + +**Interfaces:** +- Consumes: `ui::locale_table` (number and date facts per locale). +- Produces: `ui::formatCell`, used by every backend, copy, the status line and the filter menu. + +- [ ] Tests: spec rows "A column's display", "Companion columns travel with their column", "Locked rows and companions" (the companion half). Review Focus 2: `"format cell: negative halves and empty text"` (`-0.125` at 2 places: `HalfUp` → `"-0.13"`, `HalfEven` → `"-0.12"`; `0` with `empty = "–"` → `"0"`; empty cell → `"–"`; locale `de` with grouping → `"-12.500,50"`). +- [ ] If Task 9 has landed first, replace its stand-in formatter with `formatCell` in this commit. +- [ ] Commit: `feat(table): column display, tone and tooltip`. + +### Task 16: A raised sort bound (table.md step 9c) + +**Spec:** "Sort chains" (the bound). + +**Files:** `include/morph/table/description.hpp` (`TableCapabilities::maxSortKeys` filled from `ServedTable::limits`), `include/morph/ui/interpreter.hpp` (`maxSortKeys` on a `rows` table). + +- [ ] Tests: spec row "A raised sort bound". +- [ ] Commit: `feat(table): a sort bound up to 32 keys`. + +### Task 17: Acting on rows (table.md step 9b) + +**Spec:** "Acting on rows" (all subsections). Needs Tasks 9 and 15. Four commits. + +**17a — A selection of the view.** Files: create `include/morph/table/row_selection.hpp` (`RowSelection` with `x-selectionOf`, `SelectionPolicy`, `SelectedRows`, `resolve(model, selection, context, policy)`, `ResolveRequest`, `RowRef` with `x-rowOf`); `view_cache.hpp` (the membership digest); `query.hpp` (`Page::viewRevision`, `TableErrorCode::ViewChanged`, `QueryLimits::maxSelectedRows`); `BRIDGE_REGISTER_TABLE` adds `Table..resolve`; `WindowRows` (select-all as a view, `except`, `selectedCountKind`, the copy of a view selection); tests `tests/table/test_table_selection.cpp`, `tests/net/test_push_tables.cpp`, `tests/test_ui_table_model.cpp`. Tests: spec rows "A selection of the view", "Resolving from elsewhere", "The view revision is membership", "Copy of a selection of the view"; invariants 33, 34. Commit `feat(table): select the whole view and resolve it on the server`. + +**17b — Rows kept in view.** Files: `query.hpp` (`TableQuery::retain`, `Page::unmatched`, `QueryLimits::maxRetained`), `view_cache.hpp` (merging retained rows per request), `engine.hpp` (`Engine::retain`), `WindowRows` (the retained set, `retainHold`). Tests: spec rows "Retained rows", "Retained rows and the event"; invariant 35. Commit `feat(table): keep rows an action changed in view`. + +**17c — Key changes, busy, failed and locked rows.** Files: `description.hpp` (`ChangedRow::from`, `FailedRow`, `ChangedRows::failed`, `rowLock`), `core/effects.hpp` (the walk collecting `from`), `WindowRows`/`EngineRows` (moving state by `from`, busy rows from `RowSelection`/`RowRef`/`CellEdit` inputs, failed marks and "Clear marks", locked rows). Tests: spec rows "A key that changes", "Busy and failed rows", "Locked rows and companions" (the lock half). Review Focus 4: `"key change: onto a key already held"` (`ChangedRow{key: 7, from: 3}` with 7 held and selected: the selection holds 7 once; row 3 is fetched again). Commit `feat(table): key changes, busy, failed and locked rows`. + +**17d — The application's menu entries and a disabled header.** Files: `ui/interpreter.hpp` (`cellMenu`, `headerMenu`, `headerEnabled`, `retainChanged`, element values `selectionSet`, `busyCount`), the backends' menus. Tests: spec rows "The application's menu entries", "A disabled header". Commit `feat(ui): application entries in table menus and a disabled header`. + +### Task 18: Step 10, as measured + +Not a task until a figure justifies it (table.md step 10: repairing a cached view, local work when a window holds the whole view, a persistent tier). Each, when its benchmark shows the need, gets its own plan. + +--- + +## Phase B — editing (`editing.md`) + +### Task 19: The focus chain and a grid of a form (editing.md step 1) + +**Spec:** editing.md "A grid of a form" (binding, columns are fields, submission), "The cell" (editor, entering and leaving, validation, read-only, what changed), "Walking". Needs Tasks 9, 11, 15. + +**Files:** +- Modify: `include/morph/ui/view.hpp`, `include/morph/ui/backend.hpp`, `include/morph/ui/mount.hpp` (a focus chain in document order; `focusNext(node)`, `focusPrevious(node)`) +- Create: `include/morph/ui/form_rows.hpp` (`FormRowsSource`), `include/morph/ui/grid.hpp` (the cell editor host, the walk), `tests/test_ui_grid.cpp` +- Modify: `include/morph/forms/engine/form_session.hpp` (per-cell snapshots of `ObjectArray` rows by row key; `modified(path) -> bool`, `revert(path)`), `include/morph/ui/interpreter.hpp` (a `table` with `form` and `field`; the overlay `{"widget": "grid", "grid": {…}}`; `enterMoves`, `walk`, element values `modifiedCount`, `modifiedRows`, `errorCount`, `editing`) +- Docs: `docs/spec/ui/view_tree.md`, `docs/spec/ui/backend_contract.md`, `docs/spec/forms/engine.md` (the grid widget leaves "Not supported"; per-cell snapshots), `docs/spec/ui/document.md`, `docs/spec/ui/interpreter.md`. + +- [ ] Tests: editing.md rows "A grid is a form", "Read-only is one rule (grid)", "The walk" (grid cases), "A sorted grid keeps its record", "Submission", "What changed". Review Focus 3 (walk half): `"grid: an empty grid lets Tab leave at once"`. +- [ ] Commit: `feat(ui): grids of a form`. + +### Task 20: Derived values (editing.md step 2) + +**Spec:** editing.md "Derived values". + +**Files:** +- Create: `include/morph/forms/derive.hpp` (`DeriveOf`, `Derived{values: [{path, value}]}`, the `derived()` field metadata writing `x-derived`) +- Modify: `include/morph/forms/forms.hpp` (`computedFields` on a row type, recomputed by `recomputeAll` inside `ObjectArray` rows), `include/morph/core/registry.hpp` (`evaluate` recomputing row members), `include/morph/forms/engine/form_session.hpp` (`loadDerived(Derived const&)` and `loadDerived` from an evaluation body; the round's debounce, sequence and row-key mapping), `include/morph/forms/choice.hpp` (options depending on the row), `include/morph/ui/grid.hpp` (kind and overlays per row) +- Create: `tests/forms/test_forms_derive.cpp` +- Docs: `docs/spec/forms/forms.md`, `docs/spec/forms/engine.md`, `docs/spec/core/registry.md`. + +- [ ] Tests: editing.md rows "Derived values", "The server decides", "Kind per row", "Choices per row"; invariants 2, 3, 9. +- [ ] Commit: `feat(forms): derived values for grids`. + +### Task 21: Selecting cells, paste and rows of a grid (editing.md step 3) + +**Spec:** editing.md "Selecting cells, clipboard and fill", "Rows of a grid". + +**Files:** `include/morph/ui/grid.hpp`, `include/morph/ui/interpreter.hpp` (`selectionMode: "cells"` and its value, `appendRows`, `rowsFixed`), the backends; tests `tests/test_ui_grid.cpp`. + +- [ ] Tests: editing.md rows "Cell selection", "Paste, fill, clear", "Rows of a grid"; invariant 7. Review Focus 3 (paste half): `"grid: paste into an empty grid"` (with `appendRows` it adds the block's rows up to `maxItems`; without, it drops every value and the status line counts them). +- [ ] Commit: `feat(ui): cell selection, paste and row operations in grids`. + +### Task 22: Bands, spans and summary rows (editing.md step 4) + +**Spec:** editing.md "Header bands", "Spans", "Summary rows". + +**Files:** `include/morph/table/description.hpp` (`BandInfo{label, unit, columns, bands}`, `TableDescription::bands`), `include/morph/ui/view.hpp`, `include/morph/ui/mount.hpp`, `include/morph/ui/interpreter.hpp` (`bands`, a span part's `columns`, `summaries` on `table` and `group`), the backends; tests `tests/test_ui_mount.cpp`, `tests/test_ui_grid.cpp`, `tests/qt_quick/`. + +- [ ] Tests: editing.md rows "Bands", "Spans", "Summary rows"; invariant 8. Review Focus 5: `"bands: a stored layout hides then shows a band's column"` (the band reappears over the one shown column with its unit; a band whose columns are all pinned draws only in the pinned part). Review Focus 3 (summary half): `"summary: over no rows"` (`count` 0, `mean` empty). +- [ ] Commit: `feat(ui): header bands, spans and summary rows`. + +### Task 23: Fixed grids (editing.md step 5) + +**Spec:** editing.md "Fixed grids". + +**Files:** `include/morph/ui/grid.hpp`, `include/morph/ui/interpreter.hpp` (`layoutRows` with `form`, blocks by `forEach`); tests `tests/test_ui_grid.cpp`. + +- [ ] Tests: editing.md row "Fixed grids". +- [ ] Commit: `feat(ui): fixed grids`. + +### Task 24: Cells of a model's table (editing.md step 6) + +**Spec:** editing.md "Cells of a model's table". Needs Tasks 9 and 17. + +**Files:** `include/morph/table/edits.hpp` (`CellEdit` with `x-cellEditOf`, `RowEdit` with `x-rowEditOf`, `EditInfo{action, member, readonly}`, `CellEdits` committing a `RowEdit` per row and holding drafts by row key and column), `include/morph/table/description.hpp` (`TableColumn::edit`), `include/morph/ui/table_rows.hpp`, `include/morph/ui/interpreter.hpp` (`editMode`); tests `tests/test_ui_table_model.cpp`, `tests/net/test_push_tables.cpp`. + +- [ ] Tests: editing.md rows "Cells of a model's table", "Read-only is one rule (model table)", "The walk" (model-table case); invariant 6. +- [ ] Commit: `feat(table): edit cells of a model's table`. + +--- + +## Order and parallelism + +Tasks run in the order above, except: + +- Task 0a has no dependency and may run first or beside Task 0. +- Tasks 8a to 8d depend only on Task 3 and may run in parallel lanes; they share `filter.hpp`, so they merge one at a time. +- Task 15 may land before Task 9 (then Task 9 uses `formatCell` directly). +- Task 16 depends only on Task 12. +- Phase B starts after Tasks 9, 11 and 15 (Task 19), and Task 24 waits for Task 17. From d98f31013e48527746f92d9f4055924ae2d14386 Mon Sep 17 00:00:00 2001 From: yaraslau Date: Tue, 17 Nov 2026 16:45:19 +0100 Subject: [PATCH 3/3] Plan the one table in seven detailed parts, with a handover The index (2026-11-17-one-table.md) gains the parts table and 29 cross-part decisions that every part follows. Parts A to G plan table.md steps 0 to 9c and editing.md steps 1 to 6 with exact signatures, tests and commands. Every part has had one review round; the handover file lists the findings still open for parts C, D, E and G and the spec questions for the author. Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-11-17-one-table-HANDOVER.md | 137 ++ .../2026-11-17-one-table-a-foundation.md | 812 ++++++++++ .../2026-11-17-one-table-b-slow-columns.md | 855 +++++++++++ .../2026-11-17-one-table-c-engine-config.md | 974 ++++++++++++ .../plans/2026-11-17-one-table-d-screen.md | 925 ++++++++++++ .../plans/2026-11-17-one-table-e-configure.md | 1102 ++++++++++++++ ...26-11-17-one-table-f-display-and-acting.md | 1303 +++++++++++++++++ .../plans/2026-11-17-one-table-g-editing.md | 1150 +++++++++++++++ .../superpowers/plans/2026-11-17-one-table.md | 58 +- 9 files changed, 7311 insertions(+), 5 deletions(-) create mode 100644 docs/superpowers/plans/2026-11-17-one-table-HANDOVER.md create mode 100644 docs/superpowers/plans/2026-11-17-one-table-a-foundation.md create mode 100644 docs/superpowers/plans/2026-11-17-one-table-b-slow-columns.md create mode 100644 docs/superpowers/plans/2026-11-17-one-table-c-engine-config.md create mode 100644 docs/superpowers/plans/2026-11-17-one-table-d-screen.md create mode 100644 docs/superpowers/plans/2026-11-17-one-table-e-configure.md create mode 100644 docs/superpowers/plans/2026-11-17-one-table-f-display-and-acting.md create mode 100644 docs/superpowers/plans/2026-11-17-one-table-g-editing.md diff --git a/docs/superpowers/plans/2026-11-17-one-table-HANDOVER.md b/docs/superpowers/plans/2026-11-17-one-table-HANDOVER.md new file mode 100644 index 000000000..6f231d9b1 --- /dev/null +++ b/docs/superpowers/plans/2026-11-17-one-table-HANDOVER.md @@ -0,0 +1,137 @@ +# The One Table — plan status and handover + +The implementation plan for `docs/spec/table/table.md` and `docs/spec/table/editing.md` is split into an index and seven detailed parts. The index (`2026-11-17-one-table.md`) holds the parts table and the **Cross-part decisions** (1–29), which are binding on every part. No code has been written; only plans exist. + +## Status + +The plans go through review rounds with Fable until a review reports no Critical or Important issue. Round 1 has run on every part. The work stopped mid-round, so some files hold fixes that are only partly applied. + +| Part | File | State | +|---|---|---| +| A — Tasks 0–2 | `…-a-foundation.md` | Round-1 fixes applied (9 findings, plus `detail::serve`). Round-2 review not yet run. | +| B — Tasks 3–7 | `…-b-slow-columns.md` | Round-1 fixes applied (12 findings). Round-2 review not yet run. | +| C — Task 8 | `…-c-engine-config.md` | Round-2 review done. Its 8 findings (below) may be partly applied. | +| D — Tasks 9–11 | `…-d-screen.md` | Round-1 findings (below) may be partly applied. | +| E — Tasks 12–14, 16 | `…-e-configure.md` | Round-1 findings (below) may be partly applied. | +| F — Tasks 15, 17 | `…-f-display-and-acting.md` | Round-1 fixes applied (11 findings). Round-2 review not yet run. | +| G — Tasks 19–24 | `…-g-editing.md` | Round-1 findings (below) may be partly applied. | + +## What to do next + +1. **Finish the open fixes for C, D, E and G.** For each part, check every finding below against the file and apply it where it is missing. Each finding gives the location in the plan, as a line number from when the review ran, and the fix. +2. **Run a fresh Fable review of all seven parts.** Each review checks: + - spec fidelity; + - executability against the real code; + - consistency with sibling parts and with the index's decisions; + - plan quality per `superpowers:writing-plans`. + + Reviewers report only Critical or Important issues, or "NO ISSUES". +3. **Repeat fix and re-review** until every part comes back with "NO ISSUES". When a finding changes a shared name or an owner, add a Cross-part decision to the index and tell the affected parts. +4. **Settle the spec questions below with the spec author.** The plans already take a stated position on each one. +5. **Then execute** with `superpowers:subagent-driven-development`, in index order. + - Implementers follow the user's model rules: Haiku for fully specified tasks, Opus for judgment, Opus for every review. + - Net tests (`morph_net_tests`) run only on `linux-everything`. + +## Open findings + +### C — round 2 + +1. `compileGroup` refuses more than 32 include or 32 exclude entries. `Engine::setFilter` compiles with no limits (`engine.hpp:717`, `:996`). Add a test with 33 entries. +2. A `Together` group takes "0, or 2 to k" include entries, not "2 to k". Add a check for an empty include. +3. Edit `table.md` 2534–2536 (and 2505) for `x-units` in the same commit as 8c. +4. `readGenericPage` with `units == nullptr` resolves units through `registryOf(description)`. Add a test that reads a page with no registry passed. +5. Fix `==` comparisons that do not compile: + - compare `*a == *b` rather than two `expected`s; + - declare `operator== = default` on `TypedMeasure`, `Measure` and `UnitInfo`. +6. Drop the "Unknown ⇒ passes some and fails some" clause from the invariant-1 test and name its generator. +7. Order 8a → 8b → 8d (index decision #17). 8b moves `ColumnGroupInfo` into `data_source.hpp`; no include cycle remains. +8. The `Filtered::by` test helper passes `source->groups()`. + +### D — round 1 + +- **C1 — the cell seam (index #28).** + - Replace `rowView` with `cellView(column, rowSignal)` plus a row-level view. + - `TableSource` names the columns it always fetches. + - `TableColumn::id` lands in 9d. + - Pooled cells own a row signal and are rebound by setting it. + - Define `cellBinds`. +- I1: build the tests' `FormatContext` from the locale table, so the grouping separator is set. +- I2: name how a reply `Value` is decoded into a `GenericPage` and a description. +- I3: segment cleanup uses an intent entry `.pending`, because the preferences store cannot list its keys. +- I4: specify the quota entry's key and arithmetic, and test it. +- I5: persisted state for `rows` tables, with an empty `model` allowed as 0–256 bytes (spec correction). +- I6: `tableMessage` lands in 9d (index #11). +- I7: cell load state reaches both backends (index #29), plus the row-height rule. +- I8: steps for the `model`+`action` and `highlightKey` keys. +- I9: `onMissingRowField` adds only described ids, and is posted rather than run inside a binding. +- I10: the K8 window holds 40 or more columns. +- I11: kind inference for every `ValueKind`, with a timestamp test at milliseconds against seconds. +- I12: declare the group-run index in 11a. +- I13: a save during a held load keeps the stored parts. +- I14: + - `RecordOf` returns an owning row record; + - add the "unknown unit" fit case; + - define the shape of `carried`. + +### E — round 1 + +- C: a user-formed group has no values list; its menu offers conditions and the match line only (index #26). +- I2: a column's values and conditions are exclusive. State how a reopened menu classifies them. +- I3: one rule for when a condition applies: live after `tableFilterDebounce`, and Apply commits. +- I4: a tick model for "(Select all)" under a search. +- I5: a date tree of years, months and days with server-side buckets, one `between` per ticked bucket, and a date picker. +- I6: canonicalise typed operands. Each dimension gets a heading in a measure's values list. +- I7: `kNoDocument` and `publish` go into the spec, a store that ignores the sentinel is tested, and `Sharing` gets a default. +- I8: tests assert the revision `publish` returned, not "1" or "2". +- I9: `TableViews` loads through D's `TableStates` (T5). +- I10: `clipboardText()` (index #27). +- I11: the Review Focus 5 test reaches the reload after `Conflict`. +- I12: the collection test clicks header index 1, and the `rowKeys` helper has an owning task. +- I13: the accessible text is exactly "sorted ascending, key 2 of 3". +- `tableMessage` now comes from D's 9d; E consumes it (index #11). + +### G — round 1 + +- C1: `GridSource` is index-based (`rowCount`, `keyAt(i) -> optional`), and the walk stops at an unfetched row. +- C2: + - `setOnCellClick(viewRow, column, doubleClick)` and `setOnFocusOut`, with recording-backend drivers; + - `GridController::activate`; + - define how the editor Node sits in a cell, and which key chords apply while editing; + - a TUI case. +- I1: `EngineRows` over a `shared_ptr` (index #21), with the grid's own `RecordOf`. +- I2: "Undo" and "Recalculate" are `StatusEntry`s (index #19). +- I3: the grid's own `CellEdits` runs are not registered in `RowStates`. Test both cases. +- I4: `modified()` is false only for `x-computed` and `x-derived` members. +- I5: the schema-root `x-cellEditOf` / `x-rowEditOf` write and read path. Check that glaze reflects a base class. +- I6: interpreter tests on Windows for the Task 24 lowering, and the `editMode` refusal. +- I7: menu entries: Revert cell and Revert row, Insert row, Delete rows, Add row. +- I8: a view and backend contract for cell selection. +- I9: + - break the `form_view.hpp` ↔ `grid.hpp` include cycle; + - `formGrid` takes a `FormatContext`; + - `FixedGridSource` takes a `FormSession*`. +- I10: when a row is absent from a refetch, clear its draft. +- I11: name a mutation for each test at ~313, 629, 878, 883, 888, 940, 1114. +- I12: the "row is gone" test runs capture → remove → load. +- I13: one `committed()` per row operation. + +## Spec questions for the author + +The plans deviate from or settle the spec on each of these. Confirm each one, then update the spec to match. + +1. The spec says `RemoteServerOptions`; the code's type is `ServerConfig`. +2. `Backend::copyText` is really `IViewBackend::copyText`. +3. `formatCell` takes `ui::FormatContext`; the spec's `Locale` type does not exist. +4. `resolve` takes `Model&` rather than `Model const&`, because table functions are non-const (index #23). +5. `x-units` in the row schema contradicts "units come from the description alone" (2505 against 2534). +6. Two models cannot both offer a table with the same id, because action ids are `Table.`. +7. Step 1 against step 2b: who adds `TableQuery::columns`. The plans put it in step 1. +8. An empty `model` in a table-state key (2216 says 1 to 256 bytes). +9. `ITableStateStore` needs `publish` and an "expect nothing stored" sentinel. +10. Range selection is listed at both step 5 and step 7. The plans put it at step 5. +11. Smaller gaps are noted in each part's Decisions section: + - the busy-row exemption for a row's own queued edits (editing.md); + - the timing of a row-mode refusal (editing.md); + - whether a deletion asks before it deletes, or deletes and offers Undo (editing.md); + - the `AtLeast` total; + - the typo "apunloaded" at table.md:800. diff --git a/docs/superpowers/plans/2026-11-17-one-table-a-foundation.md b/docs/superpowers/plans/2026-11-17-one-table-a-foundation.md new file mode 100644 index 000000000..e651bd0f7 --- /dev/null +++ b/docs/superpowers/plans/2026-11-17-one-table-a-foundation.md @@ -0,0 +1,812 @@ +# The One Table, Part A: the Fixture, the Description and the Generic Page — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Land table.md Delivery order steps 0, 0a and 1: the wide-table fixture and its baselines, the removal of paging in a database, and the generic table contract (a table as a run-time description, served by any model through `BRIDGE_REGISTER_TABLE`, answered as a generic page on `tableWork`). + +**Architecture:** The fixture is a column-store `DataSource` generated from a seed, in a static test library every test executable links. Paging in a database leaves the library and the lims examples, which read their rows and answer with `apply`. A new header `include/morph/table/description.hpp` holds the description, the generic page and its codec, and the bridge-facing types; `BRIDGE_REGISTER_TABLE` registers two actions and one keyed record per table, and the result walk learns to find a table's keys inside `ChangedRows`. + +**Tech Stack:** C++23, glaze, Catch2, CMake presets `cl-debug` / `cl-release` (MSVC), Emscripten 4.0.7 + Node for the WebAssembly table tests, Qt 6 Quick offscreen for the frontend baseline, POSIX `morph::net` for the socket tests. + +**Spec:** `docs/spec/table/table.md`, sections "Performance" (the fixture, the data, how figures are taken, and the baseline rows of step 0), "A table is a description", "A model's tables" (with "The table's record"), "Typed rows are a convenience", "Rows on the wire", "A table on the server", "Alternatives considered" (the "Paging in the database" item), the rows of "Tests" and "Invariants" named in each task, the step-0a and step-1 rows of "Neighbouring specs", and "Delivery order" steps 0, 0a, 1. Roadmap: `docs/superpowers/plans/2026-11-17-one-table.md` (its Global Constraints, standard task loop and task numbers apply unchanged). + +Part A of 7; index: 2026-11-17-one-table.md + +## Global Constraints + +Everything in the roadmap's **Global Constraints** applies. In addition, for this part: + +- The fixture has no row type anywhere: its description is built from `WideTableOptions`, and a reply is the generic page written from the arrays (table.md, "The fixture"). +- "The same seed gives the same table on every machine": no `std::uniform_*_distribution`, no `std::random_device`, no clock in the generator; cells come only from the integer hash this plan pins. +- "A benchmark is recorded from the step that lands what it measures. No baseline is taken of a thing that does not exist." +- Timings are never CI gates; counts are, and run as ordinary tests. Benchmarks are tagged `"[.benchmark]"` and first assert that the path they time was reached. +- Timings are taken on Release, median and 10th–90th percentile over at least seven runs at 100,000 rows, with the machine's load recorded beside them (`engine.md`, "Measurements"). +- `QueryLimits::maxColumns = 512` (roadmap Global Constraints). +- A `TableDescription`'s `revision` is filled by the framework from its canonical JSON; a model leaves it empty. +- The action ids a table registers are `Table.` and `Table..describe`; its record is `.
`. + +## Implementer models + +Per the user's standing instructions: Task 1 is fully specified (Haiku). Tasks 0, 0a and 2 need investigation across layers (the Emscripten build, the examples' data access, the dispatcher's handler call) and go to Opus. Every review goes to Opus. + +## Review Focus + +1. **A `columns` list that is not clean.** Duplicates, ids out of the description's order, an id the description lacks next to valid ones, an empty list: the page names each valid column once, in the description's order; an unknown id is `UnknownColumn`; an empty list is a page of keys. Test in Task 1: `"table query: a columns list is answered in the description's order, once each"`. +2. **Numbers JSON cannot hold.** `NaN`, `+inf` and `-inf` in a `Number` column; `INT64_MIN`/`INT64_MAX` integers; a `Rational` whose scaled numerator would overflow: each crosses the wire and reads back as the same cell, so the client keys it as the server did. Test in Task 1: `"generic page: numbers at the edges cross the wire unchanged"`. +3. **A reply that is malformed.** A row whose length differs from `columns`, `keys` and `rows` of different lengths, a `columns` entry the description lacks, a cell `[1, 0]` or `{"x": 1}`: the structural faults refuse the page with a `TableError`; a cell fault keeps the page and makes that one cell invalid. Test in Task 1: `"generic page: a malformed reply is refused and a malformed cell is invalid"`. +4. **The same seed on two machines.** A fixture that agrees with itself on one machine and differs between MSVC and Clang passes every same-process test. Test in Task 0: `"wide table: seed 7 has the pinned digest"`, run in the Windows and Linux CI legs and under Node. +5. **A model whose table function misbehaves.** It throws, or returns a source without a column its description declares: the action replies with an error, the next request to the same model succeeds, and no reply holds a partial page. Test in Task 2: `"table action: a failing table function is an error reply and the model stays usable"`. + +None of the roadmap's Review Focus items falls in this part. + +--- + +### Task 0: The fixture and the baseline (table.md step 0) + +**Files:** +- Create: `tests/table/wide_table_model.hpp`, `tests/table/wide_table_model.cpp` (static library `morph_wide_table_model`), `tests/table/test_wide_table.cpp` (in `morph_table_tests`), `tests/qt_quick/bench_qt_quick_table.cpp` (in `morph_qt_quick_tests`), `tests/qt_quick/wasm/wide_rows_baseline.cpp` and `tests/qt_quick/wasm/CMakeLists.txt` (B5's browser harness) +- Modify: `CMakeLists.txt:1252` (an `if(EMSCRIPTEN AND MORPH_BUILD_QT_QUICK AND MORPH_BUILD_TESTS)` branch adding `tests/qt_quick/wasm`), `tests/CMakeLists.txt:882-909` (the library; `morph_table_tests` links it and `test_wide_table.cpp`; `morph_tests` links it; an `if(EMSCRIPTEN)` branch that builds only `morph_wide_table_model` and `morph_table_tests`), `tests/net/CMakeLists.txt:13-17`, `tests/qt_quick/CMakeLists.txt` (link and the bench file), `tests/table/bench_table.cpp`, `tests/bench_interpreter.cpp`, `tests/table/test_table_engine.cpp:210` (tag `[threads]`), `.github/workflows/ci.yml` (new job `table-wasm`), `docs/spec/table/table.md` (a "Recorded baselines" table at the end of "Performance", before "## Invariants") + +**Interfaces:** +- Consumes: `table::DataSource`, `RowSnapshot::readColumn`, `ColumnSink`, `table::apply`, `Engine` (all existing, `include/morph/table/`). +- Produces (namespace `bench`, header `tests/table/wide_table_model.hpp`): + +```cpp +enum class WideRole : std::uint8_t { Key, Integer, Decimal, Quantity, Measure, Text, LongText, Date, DateTime, Bool }; + +struct WideColumn { + morph::table::ColumnInfo info; // id and kind; Measure-role columns are ColumnKind::Decimal until Task 8c + WideRole role; + bool sparse = false; // 10% of its cells empty + std::string level; // "recipe" or empty; used from Task 2 on (ColumnInfo::level lands in Task 3) + std::string group; // "g3", "g16" or empty +}; + +struct WideTableOptions { + std::int64_t rows = 100'000; + int columns = 300; // 300 or 30; anything else throws std::invalid_argument + std::uint64_t seed = 1; + std::chrono::microseconds loadDelay{0}; // read by the slow level from Task 5 on +}; + +[[nodiscard]] std::uint64_t cellBits(std::uint64_t seed, std::string_view column, std::int64_t row) noexcept; +[[nodiscard]] std::vector wideColumns(int columns); +[[nodiscard]] std::size_t approximateCellBytes(morph::table::Cell const& cell) noexcept; + +class WideTableSource final : public morph::table::DataSource { +public: + explicit WideTableSource(WideTableOptions options); + std::span columns() const override; + std::shared_ptr snapshot() const override; + void subscribe(morph::table::ChangeListener& listener) override; + void unsubscribe(morph::table::ChangeListener& listener) override; + [[nodiscard]] std::vector const& layout() const noexcept; + [[nodiscard]] WideTableOptions const& options() const noexcept; + [[nodiscard]] std::size_t residentBytes() const noexcept; // bytes of the column arrays, capacity included + void setCell(std::int64_t row, std::string_view column, morph::table::Cell value); // copy-on-write of that one column; notifies Updated + void regenerate(std::uint64_t seed); // notifies Reset + // snapshot(), setCell() and regenerate() hold one mutex around reading or swapping the snapshot pointer; listeners are + // notified after the lock is released. The source is shared by every model instance's strand (Part B's fifty clients). +}; +``` + +**The layout** (`wideColumns(300)`, in declared order; position `p` counts from 0): + +| Ids | Role, kind | Count | +|---|---|---| +| `id` | `Key` role, `ColumnKind::Integer`, the row key, value `row + 1`, never empty | 1 | +| `i001`…`i099` | Integer | 99 | +| `d000`…`d059` | Decimal; `d000`…`d009` have `level = "recipe"` | 60 | +| `q000`…`q029` | Quantity, canonical unit gram | 30 | +| `m000`…`m009` | Measure role, `ColumnKind::Decimal` | 10 | +| `t000` | LongText, `ColumnKind::Text` | 1 | +| `t001`…`t069` | Text; `t001`…`t010` have `level = "recipe"`; `t004`…`t006` have `group = "g3"`; `t011`…`t026` have `group = "g16"` | 69 | +| `a000`…`a009` | Date | 10 | +| `s000`…`s009` | DateTime | 10 | +| `b000`…`b009` | Bool | 10 | + +`sparse` is true exactly when `p % 12 == 11` (25 columns). `wideColumns(30)` keeps, in the same order, `id`, `i001`…`i009`, `d000`…`d005`, `q000`…`q002`, `m000`, `t000`…`t006`, `a000`, `s000`, `b000`, each with every attribute it has in the 300 layout, `sparse` included, so a column shared by `wide` and `small` has the same cells in both (table.md, "The fixture": the two tables share the rows they have in common). + +**The generator.** A cell depends only on `(seed, column id, row)`, so `wide` and `small` agree on the rows and columns they share: + +```cpp +// splitmix64: z += 0x9E3779B97F4A7C15; z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9; +// z = (z ^ (z >> 27)) * 0x94D049BB133111EB; return z ^ (z >> 31); +cellBits(seed, column, row) = splitmix64(splitmix64(seed ^ fnv1a64(column)) + static_cast(row)); +``` + +`fnv1a64` is `morph::model::detail::fnv1a64` (`include/morph/core/payload_schema.hpp:213`). With `b = cellBits(...)`: a sparse column's cell is empty when `b % 10 == 0`; otherwise by role — +Integer `static_cast(b % 2'000'001) - 1'000'000`; +Decimal `Rational{Numerator{static_cast(b % 20'000'001) - 10'000'000}, Denominator{100}, DecimalPlaces{2}}`; +Quantity with `n = b % 10'000'001`: when `(b >> 32) & 1` a `QuantityCell{Rational{n, 1'000'000, 6} /* kg */, Rational{1000, 1, 0}}`, otherwise `Rational{n, 1000, 3}` (grams); +Measure role `Rational{static_cast(b % 1'000'001), 100, 2}`; +Text: length `4 + b % 57` (LongText: `200 + b % 1801`) characters, each the entry `syllables[splitmix64(b + i) % 32]` of the fixed table `{"a","e","i","o","u","b","c","d","f","g","k","l","m","n","p","r","s","t","v","z","É","é","Ä","ä","Ö","ø","Ü","ç","ß","Z","K","M"}`, truncated to the length in UTF-8 code points; +Date `static_cast(b % 20'000)`; DateTime `static_cast(b % 1'700'000'000)`; Bool `(b & 1) != 0`. + +**The column store.** One typed array per column (`std::vector` for integer, date, date-time, decimal numerators at a fixed denominator and quantity numerators; one `std::uint8_t` unit flag for quantities and one for booleans; texts as one `std::string` pool with `std::uint32_t` offsets; a `std::vector` empty mask for sparse columns), generated once in the constructor. The snapshot holds `std::shared_ptr` to each column's arrays; `readColumn` converts into a local buffer of 256 cells and hands each run to the sink in one `cells` call. + +- [ ] **Step 1: Write the failing tests** in `tests/table/test_wide_table.cpp` + +```cpp +TEST_CASE("wide table: the same seed gives the same cells", "[table][wide]") { + bench::WideTableSource const a{{.rows = 1'000, .columns = 300, .seed = 7}}; + bench::WideTableSource const b{{.rows = 1'000, .columns = 300, .seed = 7}}; + auto const sa = a.snapshot(); auto const sb = b.snapshot(); + REQUIRE(sa->rowCount() == 1'000); + for (std::size_t r = 0; r < 1'000; ++r) { + REQUIRE(sa->rowId(r) == sb->rowId(r)); + for (std::size_t c = 0; c < 300; ++c) { REQUIRE(sa->cell(r, c) == sb->cell(r, c)); } // Mutation: seed from std::random_device + } + bench::WideTableSource const other{{.rows = 1'000, .columns = 300, .seed = 8}}; + CHECK_FALSE(other.snapshot()->cell(0, 1) == sa->cell(0, 1)); // Mutation: generator ignoring the seed +} + +TEST_CASE("wide table: the column mix is the spec's", "[table][wide]") { + auto const roles = [](int n) { std::map count; for (auto const& c : bench::wideColumns(n)) ++count[c.role]; return count; }; + auto const wide = roles(300); + CHECK(wide.at(bench::WideRole::Key) + wide.at(bench::WideRole::Integer) == 100); + CHECK(wide.at(bench::WideRole::Decimal) == 60); + CHECK(wide.at(bench::WideRole::Quantity) == 30); + CHECK(wide.at(bench::WideRole::Measure) == 10); + CHECK(wide.at(bench::WideRole::Text) + wide.at(bench::WideRole::LongText) == 70); + CHECK(wide.at(bench::WideRole::LongText) == 1); + CHECK(wide.at(bench::WideRole::Date) + wide.at(bench::WideRole::DateTime) == 20); + CHECK(wide.at(bench::WideRole::Bool) == 10); + auto const columns = bench::wideColumns(300); + CHECK(std::ranges::count_if(columns, &bench::WideColumn::sparse) == 25); + CHECK(std::ranges::count(columns, std::string{"recipe"}, &bench::WideColumn::level) == 20); + CHECK(std::ranges::count(columns, std::string{"g3"}, &bench::WideColumn::group) == 3); + CHECK(std::ranges::count(columns, std::string{"g16"}, &bench::WideColumn::group) == 16); + CHECK(bench::wideColumns(30).size() == 30); + // One column in twelve has 10% empty cells: 1,000 rows of a sparse column hold between 50 and 150 empties. + bench::WideTableSource const source{{.rows = 1'000, .columns = 300, .seed = 7}}; + auto const snap = source.snapshot(); + auto const empties = [&](std::size_t c) { int n = 0; for (std::size_t r = 0; r < 1'000; ++r) n += std::holds_alternative(snap->cell(r, c)); return n; }; + CHECK(empties(11) > 50); CHECK(empties(11) < 150); // position 11 is sparse + CHECK(empties(0) == 0); // `id` is never empty + // Mutation: sparse every 11th column instead of every 12th. +} + +TEST_CASE("wide table: shared rows agree between 300 and 30 columns", "[table][wide]") { + bench::WideTableSource const wide{{.rows = 2'000, .columns = 300, .seed = 7}}; + bench::WideTableSource const small{{.rows = 1'000, .columns = 30, .seed = 7}}; + // For every column of `small`, its cells equal `wide`'s cells of the same id on rows 0..999, empty cells included. + // Mutation: wideColumns(30) recomputing `sparse` from the 30 layout's positions. + // Mutation: hashing the column's position instead of its id. +} + +TEST_CASE("wide table: seed 7 has the pinned digest", "[table][wide]") { + // FNV-1a 64 over displayText(cell) of rows 0..999, columns 0..299, each followed by '\x1f'. + CHECK(bench::digestForTesting(bench::WideTableSource{{.rows = 1'000, .columns = 300, .seed = 7}}) == kPinnedDigest); + // kPinnedDigest is computed once by the implementer on MSVC and checked unchanged by the Linux leg and Node. + // Mutation: a std::uniform_int_distribution in the generator (passes on one standard library, fails on another). +} + +TEST_CASE("wide table: apply reads only the sorted and filtered columns", "[table][wide]") { + // Gate S3. A RecordingSource wraps a WideTableSource{1'000, 300, 7}; its snapshot counts cells handed out per column + // through both cell() and readColumn(). + // Query: sort {i001 asc}, filter t001 contains "a", page {0, 200}. + // CHECK: counts[i001] == 1'000, counts[t001] == 1'000, and the other 298 counts are 0. + // Mutation: apply listing every column in JobInput::needed. +} + +TEST_CASE("wide table: an edit changes one cell and copies one column", "[table][wide]") { + // setCell(5, "i001", Cell{std::int64_t{42}}): the new snapshot's cell(5, i001) is 42; the snapshot taken before + // still reads the old value; a subscribed listener got one RowChange{Updated, {5}}. + // Mutation: setCell writing into the shared array (the old snapshot changes too). +} +``` + +`bench::digestForTesting(WideTableSource const&) -> std::uint64_t` is declared in the fixture header for this test. + +- [ ] **Step 2: Run them to see them fail** + +Run: `cmake --preset cl-debug && cmake --build --preset cl-debug --target morph_table_tests` +Expected: compile failure, `wide_table_model.hpp` not found. + +- [ ] **Step 3: Implement the fixture** in `tests/table/wide_table_model.{hpp,cpp}` with the interface, layout and generator above, and add to `tests/CMakeLists.txt`: + +```cmake +add_library(morph_wide_table_model STATIC table/wide_table_model.cpp) +target_link_libraries(morph_wide_table_model PUBLIC morph::morph) +target_include_directories(morph_wide_table_model PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}) +apply_warnings(morph_wide_table_model) +``` + +`morph_table_tests`, `morph_tests`, `morph_net_tests` and `morph_qt_quick_tests` link it as `"$"` (the ladder's precedent, `cmake/morph_add_rung.cmake:209`), so the registrars Task 2 adds are never dropped. Tag the case at `tests/table/test_table_engine.cpp:210` `[threads]`. + +- [ ] **Step 4: Run the tests to see them pass** + +Run: `cmake --build --preset cl-debug --target morph_table_tests && ctest --test-dir build/cl-debug -R "wide table" --output-on-failure` +Expected: 6 tests pass; then the whole target: `ctest --test-dir build/cl-debug -R "^table" --output-on-failure` all pass. + +- [ ] **Step 5: Build the table tests for WebAssembly and run them under Node** + +In `tests/CMakeLists.txt`, at the top after the pinned-facts block: under `if(EMSCRIPTEN)`, define `morph_wide_table_model` and `morph_table_tests` exactly as above, register no `catch_discover_tests`, and `return()`. Add to `.github/workflows/ci.yml` a job: + +```yaml + table-wasm: + name: Table tests (WebAssembly, Node) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: mymindstorm/setup-emsdk@v14 + with: { version: 4.0.7 } + - name: Configure + run: emcmake cmake -S . -B build-wasm -G Ninja -DCMAKE_BUILD_TYPE=Release -DMORPH_BUILD_TESTS=ON -DMORPH_BUILD_EXAMPLES=OFF + - name: Build + run: cmake --build build-wasm --target morph_table_tests + - name: Run + run: node build-wasm/tests/morph_table_tests.js "~[threads]" +``` + +Run locally (with emsdk active): the same three commands. +Expected: `All tests passed` with the digest case among them. If a header of `morph::morph` does not build under Emscripten, the fix belongs to this commit (AGENTS.md: build configuration is fixed in the change that needs it); a defect in `include/morph` found this way is filed. + +- [ ] **Step 6: Write the baseline benchmarks** + +Each case is `"[.benchmark][baseline]"`, prints Markdown rows as `bench_table.cpp`'s `printRow` does, and first `REQUIRE`s the guard in the last column. + +| Id | File, case name | What it times or counts | Guard | +|---|---|---|---| +| S1 | `bench_table.cpp`, `"table baseline: S1 S2 S7 over the wide table"` | `apply`, sort `d000` asc, page {0, 200}, 100,000 × 300 | the page has 200 rows and the view 100,000 | +| S2 | same case | `apply`, filter `t001` contains `"ka"` | the view is neither empty nor all rows | +| S7 | same case | ten consecutive `apply` calls of S1's query, offsets 0, 200, … 1,800: total and per call | each page's first key differs | +| S8 | `bench_table.cpp`, `"table baseline: S8 resident bytes"` | `WideTableSource::residentBytes()` at 100,000 × 300, and the sum of `approximateCellBytes` over the same rows held as a `VectorSource` | the `VectorSource` has 100,000 rows | +| B1 | `bench_table.cpp`, `"table baseline: B1 client mode with no worker"` | an `Engine` over `WideTableSource{rows, 30}` with an owner and no worker, at 10,000, 20,000, 100,000 rows: sort on `a000` asc, `d000` desc, `i001` asc, then filter `t001` contains `"ka"`; request to result, and the longest owner step | the view's first row changed after the sort | +| B2 | the S1, S2 and B1 cases and W2's decode (Task 1) | run under Node from the `table-wasm` build: `node build-wasm/tests/morph_table_tests.js "[.benchmark][baseline]"`; recorded as a factor over native | as the native case | +| W7 | `bench_table.cpp`, `"table baseline: W7 as a typed page"` | bytes and encode time of a page of 200 rows × 300 columns written as row objects keyed by column id, each cell as glaze writes the typed member (`Rational` as its `{"num","den","dp"}` object) | the page holds 60,000 cells | +| U1 | `tests/bench_interpreter.cpp`, `"interpreter baseline: U1 U2 a rows table of 100,000"` | a `ui/1` document whose `table` reads state `items` (100,000 rows of `id`, `name` from `t001`, `qty` from `i001`) and binds `sort` to state `order`; a button sets `order` to `[{"column":"name","direction":"asc"}]`; click to settled | the first rendered row's `name` changed | +| U2 | same case | the sorted table, a button replacing one row's `name` in `items`; click to settled | that row moved | +| U3 | `bench_interpreter.cpp`, `"interpreter baseline: U3 folds per sort"` | the U1 sort with a counting collator installed where the interpreter takes its collator; folds performed | the count is above 100,000 | +| U4 | `bench_interpreter.cpp`, `"interpreter baseline: U4 memory of a rows table"` | 10,000 × 300 rows as document values, sorted on one column: `approximateCellBytes` sum against the document's value bytes | the table mounted with 10,000 rows | +| U5 | `bench_interpreter.cpp`, `"interpreter baseline: U5 a collection of 20,000"` | a forms-engine collection over a `ListRows` reply of 20,000 rows, as `tests/test_forms_engine_collection.cpp` builds one: mount, sort, one row edited | the collection holds 20,000 rows | +| F2–F5 | `tests/qt_quick/bench_qt_quick_table.cpp`, `"qt quick baseline: F2 to F5 on a rows table of 2,000 x 300"` | a `rows` table of `WideTableSource{2'000, 300}` cells in a 1,280 × 800 offscreen window: widgets alive (F2), frame time of vertical (20 rows/s, 2,000 rows/s, thumb drag) and horizontal (10 and 100 columns/s) scrolls timed as table.md "How a frame is timed" says for CI (F3, F4), memory per realized row (F5) | the table realized at least 30 rows | +| B5 | `tests/qt_quick/wasm/wide_rows_baseline.cpp` | a Qt WebAssembly executable (`tests/qt_quick/wasm/CMakeLists.txt`, reached from the root branch below) that mounts F2's table, scrolls one screen and prints mount time and the scroll's frame times to the console; run locally in a headless browser with `emrun --browser chrome --browser_args=--headless=new` | its console shows 2,000 rows mounted | + +- [ ] **Step 7: Record the baselines** + +Run on a Release build, machine load recorded: +`cmake --preset cl-release && cmake --build --preset cl-release --target morph_table_tests morph_tests` +`build/cl-release/tests/morph_table_tests.exe "[.benchmark][baseline]"` and `build/cl-release/tests/morph_tests.exe "[.benchmark][baseline]"`; +`cmake --preset cl-qt-release -DMORPH_BUILD_QT_QUICK=ON && cmake --build --preset cl-qt-release --target morph_qt_quick_tests`, then with `QT_QPA_PLATFORM=offscreen`: `build/cl-qt-release/tests/qt_quick/morph_qt_quick_tests.exe "[.benchmark][baseline]"`; +B2 under Node as in the table above. B5: in the root `CMakeLists.txt`, beside `:1252` (which adds `tests/qt_quick` only when `NOT EMSCRIPTEN`), add `if(EMSCRIPTEN AND MORPH_BUILD_QT_QUICK AND MORPH_BUILD_TESTS) add_subdirectory(tests/qt_quick/wasm) endif()`; configure with the form of `.github/workflows/wasm-demo.yml:114-119`: `"$WASM/bin/qt-cmake" -S . -B build-wasm-qt -G Ninja -DCMAKE_BUILD_TYPE=Release -DQT_HOST_PATH="$HOST" -DMORPH_BUILD_TESTS=ON -DMORPH_BUILD_EXAMPLES=OFF -DMORPH_BUILD_QT=ON -DMORPH_BUILD_QT_QUICK=ON`, build `cmake --build build-wasm-qt --target wide_rows_baseline`, and run it with `emrun` as the table says. (The `if(EMSCRIPTEN)` branch of `tests/CMakeLists.txt` returns before anything else, so this build compiles only `morph_table_tests`, its fixture and this harness.) +Expected: every case prints its rows and none fails its guard. Add to table.md, at the end of "Performance": + +```markdown +### Recorded baselines + +| Id | Step | Figure | Machine, load | Date | +|---|---|---|---|---| +``` + +one row per figure (S1, S2, S7, S8, B1 ×3 native, B2 for S1, S2, B1 ×3 and W2 under Node, B5, U1–U5, W7, F2–F5), the figure as median and p10–p90. + +- [ ] **Step 8: Mutations, then commit** + +Apply each mutation named in Step 1, see its test fail, revert; paste the failing lines into the commit body. + +```bash +git add tests/table/wide_table_model.hpp tests/table/wide_table_model.cpp tests/table/test_wide_table.cpp \ + tests/table/bench_table.cpp tests/table/test_table_engine.cpp tests/bench_interpreter.cpp \ + tests/qt_quick/bench_qt_quick_table.cpp tests/qt_quick/wasm tests/qt_quick/CMakeLists.txt \ + CMakeLists.txt tests/CMakeLists.txt tests/net/CMakeLists.txt .github/workflows/ci.yml docs/spec/table/table.md +git commit -m "test(table): wide-table fixture and baselines" +``` + +--- + +### Task 0a: Remove paging in a database (table.md step 0a) + +**Files:** +- Delete: `include/morph/table/plan_conformance.hpp`, `tests/table/test_table_database_pages.cpp`, `examples/lims/org/include/lims/org/core/plan_sql.hpp`, `docs/spec/table/database_pages.md` +- Modify: `include/morph/table/query.hpp:1-16` (file comment: "or with a database query that honours the same semantics" goes), `:252-268` (`escapeLikePattern` goes), `:354-592` (the "Query plans" section: `PlanUse`, `FilterOp`, `PlannedOperand`, `PlannedSortKey`, `PlannedCondition`, `PlannedColumnFilter`, `PlannedGroup`, `QueryPlan`, `detail::plannedOperand`, `detail::plannedCondition`, `detail::plannedColumnFilter`, `detail::untranslatable`, `planQuery`); no `QueryLimits` member is used only by plans, so `QueryLimits` is unchanged +- Modify: `include/morph/table/filter.hpp:645` (the comment becomes "The one compile step `apply` and `Engine` share, so a throwing service is an error value in both."), `CMakeLists.txt:722`, `tests/CMakeLists.txt:891`, `tests/table/test_table_query.cpp` (receives the moved cases; loses `"table: LIKE patterns keep % and _ ordinary"`) +- Modify: `docs/spec/table/engine.md:52-53` (the `query.hpp` row loses `escapeLikePattern`, `QueryPlan`, `planQuery`; the `plan_conformance.hpp` row goes), `:300-306` and `:311-312` (see Step 5), `docs/spec/README.md:138-139` +- Modify: `examples/lims/lab/include/lims/lab/models/sample_model.hpp:96-110`, `examples/lims/lab/src/models/sample_model.cpp:280-390`, `examples/lims/ledger/include/lims/ledger/models/ledger_model.hpp:166-191`, `examples/lims/ledger/src/models/ledger_entries.cpp:535-580`, `examples/lims/tests/test_sample_pages.cpp`, `examples/lims/tests/test_ledger_entries.cpp:1440-1785`, `examples/lims/README.md:1795-1846` + +**Interfaces:** +- Consumes: `table::apply(std::vector const&, RowKey, TableQuery const&, ApplyOptions const&)` (`query.hpp:891`), `validate` (`query.hpp:228`). +- Produces: `lims::SampleModel::pageSamples(std::vector const& sites, ::morph::table::TableQuery const& query) -> ::morph::table::Page` and `lims::LedgerModel::pageEntries(CompanyId company, ::morph::table::TableQuery const& query) -> ::morph::table::Page`, each throwing `ValidationError{": " + error.message}` on a `TableError`. `canTranslateSampleColumn` and `canTranslateEntryColumn` go. + +**Test inventory.** `tests/table/test_table_database_pages.cpp` holds 14 cases: + +| Case | Fate | +|---|---| +| `"table: a total is exact or a lower bound, and a lower bound does not end the window"` | Moved to `test_table_query.cpp`, tag `[table][query]`, with its mutation line ("`PageWindow` stopping at an `AtLeast` total") | +| `"table: a total kind goes over the wire by name"` | Moved, tag `[table][query]` | +| `"table: a plan is refused as apply refuses"` | Replaced by `"table: apply refuses what validate and compile refuse, by code"` (below); its other queries are already pinned by `"table: an untrusted query is bounded"` and `"table: a column the principal may not read is refused"` | +| `"table: a throwing collator is a ServiceFailed error from planQuery"` | Deleted: `"table: a collator that throws while the filter compiles is a ServiceFailed error from apply"` (`test_table_query.cpp:457`) pins the shared step | +| `"table: a model that follows the plan answers as apply does"`, `"…the conformance helper reports a model that disagrees with apply"`, `"…a model refuses what it cannot translate with Unavailable"`, `"…a plan carries the clamped page, the cap and operands typed by their column"`, `"…the standard corpus filters every column kind with operands that parse"`, `"…an exact total above the cap must equal the true count"`, `"…the corpus holds a translation to non-integral operands on integer columns"`, `"…the corpus writes date operands the date parser reads"`, `"…the corpus holds a translation to the floor of a negative instant"`, `"…the helper reports a column whose queries every side refused"` | Deleted with the file (they test plans, the corpus or the helper) | + +In the lims tests: + +| File, case | Fate | +|---|---| +| `test_sample_pages.cpp`, `"lab: a page of samples from the database equals the engine's page over the same rows, for every sort and filter the list offers"` | Deleted (SQL translation against `apply`) | +| `test_sample_pages.cpp`, `"lab: a plan the statement cannot translate is refused"` | Deleted | +| `test_sample_pages.cpp`, `"lab: the sample list answers through the action with an exact total"` | Kept unchanged: it pins scope (bob's total is his sites' rows) and the unknown-column refusal | +| `test_ledger_entries.cpp`, the four `"ledger: a page of entries from the database equals the engine's page … (part N of 4)"` | Deleted, with `checkEntryPagesAgainstEngine`, `kCorpusParts`, `dateQueries`, `groupQueries` | +| `test_ledger_entries.cpp`, `"ledger: the entry page refuses the columns the database cannot compare, and nothing else"` | Deleted | +| `test_ledger_entries.cpp`, `"ledger: text operands that look like SQL or like patterns are only data"` | Deleted | +| `test_ledger_entries.cpp`, `"ledger: a page of entries is the caller's, with an exact total from the database"` | Kept; renamed `"ledger: a page of entries is the caller's, with an exact total"`; it pins scope (dave gets no row) | + +A helper left with no caller (`journalRowsFromLedger`, `visibleViews`, the sample `dateQueries`, anything in `lims_test_support.hpp` only these cases used) is deleted; the compiler's unused-function warning finds them. + +- [ ] **Step 1: Move the cases and write the replacement** in `tests/table/test_table_query.cpp` + +```cpp +TEST_CASE("table: apply refuses what validate and compile refuse, by code", "[table][query]") { + auto const source = tabletest::makeSource({tabletest::col("id", ColumnKind::Key), tabletest::col("n", ColumnKind::Integer)}, + {{Cell{std::int64_t{1}}, Cell{std::int64_t{5}}}}); + auto const code = [&](TableQuery const& q) { auto r = apply(*source, q); REQUIRE_FALSE(r.has_value()); return std::pair{r.error().code, r.error().column}; }; + TableQuery far; far.page.offset = 10'000'001; + CHECK(code(far).first == TableErrorCode::LimitExceeded); + CHECK(code(withFilter("n", FilterEntry{.contains = "1"})) == std::pair{TableErrorCode::UnsupportedOperator, std::string{"n"}}); + CHECK(code(withFilter("n", FilterEntry{.eq = "1", .ne = "2"})) == std::pair{TableErrorCode::InvalidSpec, std::string{"n"}}); + TableQuery group; group.filters.groups.push_back(GroupFilter{.columns = {"n"}, .combine = Combine::Any, .include = {FilterEntry{.contains = "x"}}}); + CHECK(code(group) == std::pair{TableErrorCode::UnsupportedOperator, std::string{"n"}}); + // Mutation: apply skipping validate (the offset case answers an empty page). +} +``` + +(`withFilter` is the one-column helper the moved cases bring with them; keep a single definition.) + +- [ ] **Step 2: Delete** the four files listed, remove `include/morph/table/plan_conformance.hpp` from the `FILE_SET` at `CMakeLists.txt:722` and `table/test_table_database_pages.cpp` from `tests/CMakeLists.txt:891`, remove the query-plans section and `escapeLikePattern` from `query.hpp`, and delete `"table: LIKE patterns keep % and _ ordinary"`. + +- [ ] **Step 3: Rewrite the lims list actions** to read their rows and answer with `apply` + +- `SampleModel::pageSamples(sites, query)`: one `mapper.Query().WhereIn(::Lightweight::FieldNameOf<&db::SampleRecord::siteId>, siteIds).All()` (the `WhereIn` form `examples/bank/src/models/statement_model.cpp:62` uses), each record through `toView`, then `::morph::table::apply(rows, [](SampleView const& s) { return RowId{*s.id}; }, query)`. `execute(ListSamples)` calls it with `OrgModel::visibleSites(currentPrincipal())`; `samplesWhere` builds its `TableQuery` as now and calls it with `callerSitesAt(company)`. +- `LedgerModel::pageEntries(company, query)`: the entries `execute(ListEntries)` already reads (`ledger_entries.cpp:520-532`, one query on `companyId` and `toView(mapper, loadWritten(…))` per entry), then `apply`. `execute(ListEntriesPage)` keeps its visibility check and empty page. +- Doc comments on both say what they now do: "reads the caller's rows and runs the query over them with `table::apply`; the total is exact". +- `plan_sql.hpp`'s includes leave `sample_model.cpp` and `ledger_entries.cpp`. + +- [ ] **Step 4: Prove scope is still pinned** + +Mutation: make `pageSamples` ignore `sites` (read every sample) and `execute(ListEntriesPage)` skip its visibility check, one at a time. +Expected: `"lab: the sample list answers through the action with an exact total"` fails on the first (`page.total` is all samples), `"ledger: a page of entries is the caller's, with an exact total"` on the second (dave gets rows). If either passes, add `"ledger entries: a principal never receives a row outside its scope"` asserting the failing condition, and run the mutation again. + +- [ ] **Step 5: Update the documentation** + +- `engine.md:300-306` becomes: "A model whose rows live in a database reads them as a `DataSource`, whole or lazily by column, and the server sorts, filters, pages and counts them with `apply`; `Page::totalKind` says whether a total is the count or a lower bound (`AtLeast`). `compileFilterGuarded` is the one compile step `apply` and `Engine` share, so a collator or date parser that throws is `ServiceFailed` in both." The `escapeLikePattern` bullet (`:311-312`) goes. +- `docs/spec/README.md:138-139`: the `database_pages.md` entry goes. +- `examples/lims/README.md:1795-1846`: replace the database-page section with one paragraph: the sample and ledger lists read the caller's rows with one query and answer with `table::apply`, so their order is the engine's on every surface. + +- [ ] **Step 6: Verify nothing names the removed code** + +Run: `git grep -n -E "planQuery|QueryPlan|PlanUse|plan_conformance|standardCorpus|conformanceFailures|database_pages|escapeLikePattern|plan_sql|canTranslate" -- include src tests examples docs/spec ':!docs/spec/table/table.md'` +Expected: no output. (`table.md` names the removals in its API, Neighbouring-specs and Delivery-order rows, which stay.) + +- [ ] **Step 7: Run the suites** + +Run: `cmake --build --preset cl-debug --target morph_table_tests && ctest --test-dir build/cl-debug -R "^table" --output-on-failure` +Run: `cmake --preset cl-debug -DMORPH_BUILD_LADDER=ON -DMORPH_LADDER_RUNGS=lims && cmake --build --preset cl-debug --target ladder_lims_tests && ctest --test-dir build/cl-debug -L ladder-lims --output-on-failure` +Run the docs build of the roadmap's Global Constraints. +Expected: all green; the docs build prints no warning. + +- [ ] **Step 8: Commit** + +```bash +git add -A include/morph/table tests/table tests/CMakeLists.txt CMakeLists.txt examples/lims docs/spec +git commit -m "refactor(table): remove paging in a database" +``` + +--- + +### Task 1: The description and the generic page (table.md step 1, first half) + +**Files:** +- Create: `include/morph/table/description.hpp` (add to the `FILE_SET` beside `query.hpp`, `CMakeLists.txt:723`), `tests/table/test_table_description.cpp` (in `morph_table_tests`) +- Modify: `include/morph/table/query.hpp:67-78` (`TableQuery::columns`), `:102-121` (`QueryLimits::maxColumns`), `:228-250` (`validate` checks `columns`), `tests/table/bench_table.cpp` (W1, W2, W5, W6, W7) + +**Interfaces:** +- Consumes: `PageRows` (`query.hpp:282`), `TotalKind`, `DataSource`, `ColumnInfo`, `Cell`, `RowId`, `TableError` (existing); `morph::model::detail::fnv1a64`, `hex64` (`core/payload_schema.hpp`); `bench::WideTableSource` (Task 0). +- Produces, in `namespace morph::table`: + +```cpp +// query.hpp +struct TableQuery { std::vector sort; FilterSpec filters; PageRequest page; + std::optional> columns; /* absent: every column */ }; +struct QueryLimits { /* … */ std::size_t maxColumns = 512; }; + +// description.hpp +struct TableColumn { ColumnInfo info; std::string label; std::string group; + bool sortable = true; bool filterable = true; bool shown = true; + bool operator==(TableColumn const&) const = default; }; +struct ColumnGroupInfo { std::string id; std::string label; std::vector columns; + bool operator==(ColumnGroupInfo const&) const = default; }; +struct TableCapabilities { bool keys = true; bool distinct = true; std::uint32_t maxSortKeys = 8; + bool operator==(TableCapabilities const&) const = default; }; +struct TableDescription { std::string id; std::string label; std::vector columns; std::string key; + std::vector groups; std::vector levels; + TableCapabilities capabilities; std::string revision; + bool operator==(TableDescription const&) const = default; }; +[[nodiscard]] TableDescription describe(TableDescription description); +[[nodiscard]] std::vector columnInfos(TableDescription const& description); + +using GenericRow = std::vector; +struct GenericPage { std::string revision; std::vector columns; std::vector keys; + std::vector rows; std::optional total; + TotalKind totalKind = TotalKind::Exact; std::int64_t offset = 0; }; +[[nodiscard]] GenericPage writeGenericPage(PageRows const& page, DataSource const& source, + std::span columns); +[[nodiscard]] std::expected readGenericPage(std::string_view json, + TableDescription const& description); +namespace detail { +void appendCellJson(std::string& out, Cell const& cell); +[[nodiscard]] Cell cellFromJson(std::string_view json, std::optional kind); +} +``` + +Members later tasks insert, at the place table.md's block puts them: `TableColumn::{display, tone, tooltip}` (Task 15), `TableColumn::edit` (Task 24), `TableDescription::bands` after `groups` (Task 22), `units` after `levels` (Task 8c), `rowTone`, `rowLock` after `units` (Task 15); the generic page's sidecar members (Task 4). `TableQuery::keys` is Task 4's. + +**Pinned rules.** +- `describe` returns the description with `revision = hex64(fnv1a64(glz::write_json(d with revision cleared)))`: 16 lowercase hex digits. The canonical JSON is glaze's default write of the struct, members in declaration order. +- The page's JSON members, in this order: `revision`, `columns`, `keys`, `rows`, `total` (absent when not counted), `totalKind` (`"exact"`/`"atLeast"`), `offset`. `glz::meta` maps `rows` through a `glz::custom` pair that writes each cell with `appendCellJson` and reads each with `cellFromJson(json, std::nullopt)`; `readGenericPage` re-reads the cells with their column's kind. +- A cell is written by its alternative (table.md, "Rows on the wire"): `null`; an integer; a finite `double` as glaze writes it, a non-finite one as the string `"nan"`, `"inf"` or `"-inf"`; a string; `true`/`false`; a `Rational` with `p = decimalPlaces` as `[numerator × (10^p / denominator), 10^p]` when `10^p` is a multiple of the denominator and the product fits `int64`, otherwise `[numerator, denominator]`; a `QuantityCell` as `[n, d, [fn, fd]]` with amount and factor written by the `Rational` rule, and as the plain `Rational` `[n, d]` when its factor is 1. +- `cellFromJson` reads by shape: `null` → empty; an integral number → `int64` (a `double` when `kind` is `Number`); any other number → `double`; a string → `std::string` (in a `Number` column, `"nan"`, `"inf"`, `"-inf"` → that `double`); a boolean → `bool`; `[n, d]` with `d > 0` → `Rational{Numerator{n}, Denominator{d}, DecimalPlaces{k}}` where `k` is such that `d == 10^k` (`k ≤ 18`), otherwise `kMaxDecimalPlaces`; `[n, d, [fn, fd]]` → `QuantityCell`; anything else (an object, `[n, 0]`, a negative denominator, a third element of another shape, another length) → the `std::string` of its JSON text, which no kind but `Text` reads as valid. Read this way, the client's cell keys exactly as the server's did, so a cell the server keyed as invalid is invalid on the client. +- `readGenericPage` refuses with `TableError{.code = InvalidValue}` (naming the column where there is one) when the JSON is not a page, a `columns` entry is not in the description (`UnknownColumn`), a row's length differs from `columns`, or `keys` and `rows` differ in length. +- `writeGenericPage` writes `columns` as given (the caller orders and deduplicates them), `keys[i] = snapshot->rowId(page.rows[i])`, the cells of those columns only, `total` and `offset` from `page`; `revision` is left for the caller. +- `validate` checks `query.columns` when present: more than `maxColumns` entries is `LimitExceeded`; each entry through `detail::checkColumn` (`UnknownColumn`, `ColumnNotReadable`). + +- [ ] **Step 1: Write the failing tests** in `tests/table/test_table_description.cpp` + +```cpp +TEST_CASE("description: equal descriptions have equal revisions", "[table][description]") { + TableDescription base{.id = "designs", .label = "Designs", + .columns = {TableColumn{.info = {.id = "id", .kind = ColumnKind::Integer}, .label = "Id"}, + TableColumn{.info = {.id = "name", .kind = ColumnKind::Text}, .label = "Name"}}, + .key = "id"}; + auto const a = describe(base); auto const b = describe(base); + CHECK(a.revision.size() == 16); + CHECK(a.revision == b.revision); + CHECK(describe(a).revision == a.revision); // Mutation: hashing with the old revision in + auto relabelled = base; relabelled.columns[1].label = "Title"; + CHECK(describe(relabelled).revision != a.revision); // Mutation: hashing the columns' ids only + auto hidden = base; hidden.columns[1].shown = false; + CHECK(describe(hidden).revision != a.revision); +} + +TEST_CASE("generic page: every cell kind round-trips", "[table][description]") { + // Description: id Integer, name Text, net Decimal, share Decimal, dose Quantity, ratio Number, flag Bool, on Date. + // Rows (keys 101, 102): + // 101: 101, "Alpha", 12.50 (places 2), 0.003 (places 3), QuantityCell{5 (places 0), 1/1000}, 0.1, true, 19'000 + // 102: 102, "Beta", empty, empty, Rational 250 (places 0), 2.5, false, empty + // writeGenericPage over a VectorSource of these rows, PageRows{.rows = {0, 1}, .total = 9093, .offset = 0}, + // columns {"id","name","net","share","dose"}, revision set to "9f2c"; glz::write_json gives exactly: + // {"revision":"9f2c","columns":["id","name","net","share","dose"],"keys":[101,102], + // "rows":[[101,"Alpha",[1250,100],[3,1000],[5,1,[1,1000]]],[102,"Beta",null,null,[250,1]]], + // "total":9093,"totalKind":"exact","offset":0} + // readGenericPage(that JSON, description) gives back: keys, columns, total, offset equal; each cell equal to the + // written one (Rational by value and decimalPlaces; QuantityCell by amount and toCanonical). + // The same page with all eight columns: `ratio` reads as double 0.1 and 2.5, `flag` as bool, `on` as int64 19'000. + // Mutation: writing a Rational as glaze's {"num","den","dp"} object. +} + +TEST_CASE("generic page: a cell that does not read as its column's kind is invalid", "[table][description]") { + // readGenericPage of a one-column Integer page whose rows are ["abc"], [2.5], [[1,0]], [{"x":1}], [7]: + // buildKeys over a VectorSource of the read rows gives states {Invalid, Invalid, Invalid, Invalid, Valid}. + // The same values in a Text column: the last four keys are Valid (a Text column has no invalid cell). + // Mutation: cellFromJson parsing "abc"-like strings for the column's kind on read (2.5 read as Integer 2). +} + +TEST_CASE("generic page: numbers at the edges cross the wire unchanged", "[table][description]") { // Review Focus 2 + // Number column: NaN, +inf, -inf, -0.0, 1e308 → written as "nan", "inf", "-inf", -0.0, 1e308 → read back equal + // (NaN by std::isnan). Integer column: INT64_MIN, INT64_MAX. Decimal column: Rational{INT64_MAX, 3, places 2} + // (10^2 is not a multiple of 3, so written [INT64_MAX, 3]) and Rational{INT64_MAX / 10, 1, places 18} + // (the scaled numerator overflows, so written [n, 1]) → read back equal by value. + // Mutation: writing a non-finite double with glaze's default (the page fails to write, or reads back empty). +} + +TEST_CASE("generic page: a malformed reply is refused and a malformed cell is invalid", "[table][description]") { // Review Focus 3 + // Refused, code and column: + // a row of 2 cells under 3 columns → InvalidValue; keys [1,2] with one row → InvalidValue; + // "columns":["nope"] → UnknownColumn, column "nope"; "not json" → InvalidValue. + // Kept, one cell invalid: an Integer cell [1,0] in an otherwise valid page reads, and that cell keys Invalid. + // Mutation: readGenericPage padding a short row with empty cells. +} + +TEST_CASE("table query: columns round-trip and are bounded", "[table][query]") { + // glz::write_json(TableQuery{.columns = std::vector{"a","b"}}) contains "columns":["a","b"] and reads back + // equal; R"({"sort":[],"filters":{},"page":{"offset":0,"limit":10}})" reads with columns == std::nullopt. + // validate against columns {id, a, b} with mayRead refusing "b": + // columns {"zz"} → UnknownColumn "zz"; {"b"} → ColumnNotReadable "b"; 513 entries → LimitExceeded; {} → accepted. + // Mutation: validate ignoring `columns`. +} + +TEST_CASE("table query: a columns list is answered in the description's order, once each", "[table][query]") { // Review Focus 1 + // Over the description of wideColumns(30): + // detail::orderedColumns(description, std::vector{"t000","id","t000","i001"}) == {"id","i001","t000"}; + // detail::orderedColumns(description, std::nullopt) == the 30 ids in layout order. + // (`detail::orderedColumns(TableDescription const&, std::optional> const&) + // -> std::vector` drops ids the description lacks; validate has refused them before Task 2 calls it.) + // writeGenericPage of rows 0..199 of WideTableSource{1'000, 30, 7} with an empty column list gives 200 keys and + // 200 rows of zero cells. + // Mutation: writing the columns in the query's order. +} + +TEST_CASE("generic page: a page names only the asked columns", "[table][description]") { // gate W1 + // WideTableSource{1'000, 300, 7}; PageRows of rows 0..199; 40 columns: positions 100..139. + // The JSON's "columns" has 40 entries; every row array has 40 cells; no other column id appears in "columns". + // Mutation: writeGenericPage ignoring `columns` (writes all 300). +} + +TEST_CASE("generic page: a whole page fits an envelope", "[table][description]") { // gate W7 + // The same 200 rows with all 300 columns: glz::write_json(page)->size() + 4'096 < morph::wire::kMaxEnvelopeBytes. +} + +TEST_CASE("description: a 300-column description round-trips", "[table][description]") { // W6's codec + // A description of wideColumns(300) (label = id, group from the layout, groups g3 and g16, levels {"recipe"}), + // through describe, glz::write_json and glz::read_json: equal to the original. +} +``` + +`detail::orderedColumns` is declared in `description.hpp` beside the page writer. + +- [ ] **Step 2: Run them to see them fail** + +Run: `cmake --build --preset cl-debug --target morph_table_tests` +Expected: compile failure, `morph/table/description.hpp` not found. + +- [ ] **Step 3: Implement** `TableQuery::columns`, `QueryLimits::maxColumns`, `validate`'s check in `query.hpp`, and every symbol listed in Produces in `description.hpp`, with the pinned rules above. Every public symbol gets complete Doxygen. + +- [ ] **Step 4: Run them to see them pass** + +Run: `ctest --test-dir build/cl-debug -R "description|generic page|table query" --output-on-failure` +Expected: 10 tests pass; then `ctest --test-dir build/cl-debug -R "^table" --output-on-failure`, all pass. + +- [ ] **Step 5: Write the wire benchmarks** in `bench_table.cpp`, `"table benchmark: the generic page"`, `"[.benchmark][wire]"`: W1 (bytes of the 200 × 40 page in an `ok` envelope), W2 (encode from the column store, and `readGenericPage`), W5 (the same 200 × 30 page as generic rows and as a typed `Page`, where `WideRow30` is a 30-member struct in the bench file mirroring `wideColumns(30)`), W6 (bytes, encode and decode of the 300-column description), W7 (the 200 × 300 page: bytes, encode, decode). Each first `REQUIRE`s its page's cell count. Run on `cl-release` and add rows W1, W2, W5, W6, W7 to "Recorded baselines". + +- [ ] **Step 6: Mutations, docs build, commit** + +Apply each mutation named in Step 1; paste the failing lines into the commit body. Run the docs build (Global Constraints). + +```bash +git add include/morph/table/description.hpp include/morph/table/query.hpp CMakeLists.txt \ + tests/table/test_table_description.cpp tests/table/bench_table.cpp tests/CMakeLists.txt docs/spec/table/table.md +git commit -m "feat(table): table descriptions and the generic page" +``` + +--- + +### Task 2: Tables of a model on the bridge (table.md step 1, second half) + +**Files:** +- Modify: `include/morph/table/description.hpp` (the bridge-facing types below) +- Modify: `include/morph/core/detail/result_walker.hpp:139-165` (a holder reports its record at a member), `include/morph/core/records.hpp:43-58` (`RecordInfo::table`) and a `registerTableRecordOnce`, `include/morph/core/effects.hpp:113-127` (`ActionRecord::opaque`), `:410-418` (`sameRegistration` compares `table`), `:520-527` (a table record may be keyed and derived), `:543-560` (an opaque slot), `:73-89` (`updatesTable`, `changesTable`) +- Modify: `include/morph/core/registry.hpp:841,967,1030,1039` and `include/morph/core/bridge.hpp:612,625,2245,2434` (call `invokeExecute`), and `BRIDGE_REGISTER_TABLE` after `BRIDGE_REGISTER_ACTION_SOURCE` (`registry.hpp:2206`) +- Create: `include/morph/table/runtime.hpp` (`TableRuntime`, `currentTableRuntime`, `TableRuntimeScope`; it includes only `core/executor.hpp`, so `core/remote.hpp` includes it without reaching the table headers; add it to the `FILE_SET` beside `query.hpp`); define the thread-local behind `currentTableRuntime()` in `src/ui/process_state.cpp` beside `tlsCurrent()` (`:144`), as a `MORPH_API` function so a shared build has one copy +- Modify: `include/morph/core/remote.hpp:310-367` (`ServerConfig::tableWork`), `:2861` and `:2875-2885` (`startRemote` and `startTaskRemote` install a `table::TableRuntimeScope` beside `ScopedContext`, with `work` = `tableWork` or, when null, the server's worker pool) +- Modify: `tests/table/wide_table_model.{hpp,cpp}` (`bench::WideTableModel`), `tests/table/test_table_description.cpp` +- Create: `tests/net/test_push_tables.cpp` (in `morph_net_tests`, `tests/net/CMakeLists.txt:1-12`) +- Docs: `docs/spec/core/registry.md` (a section `### BRIDGE_REGISTER_TABLE(M, ID, FN, ...)` after `### BRIDGE_REGISTER_ACTION(M, A, NAME, ...)`, `:999`, and a paragraph on `invoke` under `### ActionTraits`, `:217`), `docs/spec/core/backend.md` (`tableWork` in the `ServerConfig` table, `:2491`, and one bullet in "Fan-out", `:1221`, saying a table action's reply is built on `tableWork` and its fan-out is the record machinery's) + +**Interfaces:** +- Consumes: Task 1's `TableDescription`, `describe`, `GenericPage`, `writeGenericPage`, `detail::orderedColumns`, `columnInfos`; `validate`, `apply`, `RowsSource`, `detail::reflectedColumns`; `effects::updates`, `changes`; `session::current()`; `forms::detail::findFieldMeta`, `forms::detail::inferTitle`; `morph::async::DelayAwaiter` as the model for an awaiter that resumes on the strand. +- Produces, `namespace morph::table`: + +```cpp +class LevelSource; // defined by Task 5 as a class +struct TableContext { session::Context const& session; }; +struct ServedTable { + TableDescription description; + std::shared_ptr source; + std::vector> levels; // empty until Task 5 gives it meaning + std::function mayRead; + std::string scope; + Services services; + QueryLimits limits; +}; +struct ChangedRow { RowId key; }; // Task 17c adds `from` +template struct ChangedRows { // Task 17c adds `failed` + std::vector rows; + using heldRecord = TableOf; + static constexpr std::string_view heldMember() noexcept { return "rows"; } +}; +template struct TableTraits { static constexpr bool registered = false; }; // specialised by the macro +template struct TableOf { static_assert(TableTraits::registered, "not a registered table"); }; +template struct Typed { Fn function; }; +template [[nodiscard]] constexpr Typed typed(Fn function) noexcept; +struct rowScoped {}; +template struct derivedFrom {}; +[[nodiscard]] std::shared_ptr scoped(std::shared_ptr source, std::vector keys); +template struct QueryTable { TableQuery query; /* JSON is the TableQuery's: glz::meta value = &QueryTable::query */ + template static core::async::Task invoke(Model& model, QueryTable& action); }; +template struct DescribeTable { + template static TableDescription invoke(Model& model, DescribeTable& action); }; +namespace detail { +struct AnswerContext {}; // Task 6 adds `ViewCache* views` +[[nodiscard]] std::expected answerQuery(ServedTable const& served, TableQuery const& query, + AnswerContext const& context = {}); // served.source is pinned +[[nodiscard]] ServedTable pinned(ServedTable served); // on the owner's thread: source → PinnedSource of one snapshot +template +[[nodiscard]] ServedTable serve(Model& model, TableContext const& context); // the one way to a table's ServedTable +} +// include/morph/table/runtime.hpp +struct TableRuntime { exec::IExecutor* work = nullptr; }; // Task 6 adds `ViewCache* views` +[[nodiscard]] MORPH_API TableRuntime const* currentTableRuntime() noexcept; // null outside a RemoteServer dispatch +class TableRuntimeScope { // installs `runtime` for the scope, restores the previous one +public: explicit TableRuntimeScope(TableRuntime const& runtime) noexcept; ~TableRuntimeScope(); + TableRuntimeScope(TableRuntimeScope const&) = delete; TableRuntimeScope& operator=(TableRuntimeScope const&) = delete; }; +} +namespace bench { inline constexpr std::string_view kWideModelType = "WideTableModel"; } // tests/table/wide_table_model.hpp +namespace morph::effects { +template using updatesTable = updates<::morph::table::TableOf>; +template using changesTable = changes<::morph::table::TableOf>; +} +namespace morph::model::detail { +template decltype(auto) invokeExecute(Model& model, Action& action); +// `Action::invoke(model, action)` when the action declares one, `model.execute(action)` otherwise. +} +// ServerConfig: std::shared_ptr<::morph::exec::IExecutor> tableWork; // null: the server's worker pool +// effects::ActionRecord: bool opaque = false; // held at no path; `paths` is empty +// record::RecordInfo: bool table = false; // registered by BRIDGE_REGISTER_TABLE +``` + +`BRIDGE_REGISTER_TABLE(M, ID, FN, ...)`, `ID` a string literal, `FN` a member function pointer or a `table::typed(…)` expression, options `table::rowScoped` and `table::derivedFrom<…>`. It requires `BRIDGE_REGISTER_MODEL(M, …)` before it and `` in the translation unit, and emits: +1. `template <> struct morph::table::TableTraits { using model = M; using row = R; static constexpr bool registered = true; static constexpr std::string_view id = ID; static constexpr auto serve = FN; }`, where `KEY` is the member function pointer: `FN` itself, or `FN.function` for `typed(&M::fn)`, and `R` is `Row` for a typed table and `void` otherwise (`table::detail::tableKey(FN)` and `table::detail::TableRow` compute them). Every type below is keyed on `KEY`, so `TableOf<&M::fn>`, `ChangedRows<&M::fn>` and `updatesTable<&M::fn>` name a typed table as they name a generic one; `QueryTable::invoke` reads `TableTraits::serve` to call the function; +2. `ActionTraits` and registrars for `QueryTable` under `"Table." ID` (result `GenericPage`) and `DescribeTable` under `"Table." ID ".describe"` (result `TableDescription`), as `BRIDGE_DETAIL_ACTION_TRAITS_BODY` does for an action with an explicit result; +3. `registerTableRecordOnce, Options…>(ModelTraits::typeId() + "." ID)`: a `RecordInfo` with `keyField = "key"`, `table = true`, `privateKeys` from `rowScoped`, `derivedFrom` from `derivedFrom<…>`; +4. one more `ResultSlot{typeid(TableOf), "", .opaque = true}` on `"Table." ID`'s result slots (`ResultSlot` gains `bool opaque = false`), which `resolve` turns into `ActionRecord{.record, .keyField, .paths = {}, .opaque = true}` so the table's action holds its record for watch admission (`remote.hpp:1794`) and names no path. + +**Pinned behaviour.** +- The walk (`result_walker::visit`): a type `T` with `T::heldRecord` and `T::heldMember()` adds `ResultSlot{typeid(typename T::heldRecord), path.empty() ? heldMember() : path + "." + heldMember()}` beside its own slot. +- `resolve` does not report "derived from other records but has a key" for a record with `table == true`: a source's change reaches it as a whole-record change, as `addDerivedChanges` already sends. +- `QueryTable::invoke`: on the strand, build `TableContext` from `session::current()` (a static empty `Context` when null) and call `detail::serve(model, context)`; read `currentTableRuntime()` before the first suspension and build `AnswerContext{}` from it; then, still on the strand, replace `served.source` with a `detail::PinnedSource` (the one `snapshot()` taken now and a copy of `columns()`), because a `DataSource` is owner-affine and only its snapshot may cross threads (`data_source.hpp:728`; table.md "the framework takes the source's snapshot, and then on the server's table executor…"); then run `answerQuery(pinnedServed, action.query, context)` as one work item posted to `runtime->work` and resume with its result through an awaiter built like `DelayAwaiter`, so the coroutine's own steps stay on the strand and the sort does not. With no runtime installed or a null `work` (`LocalBackend`), `answerQuery` runs inline. A `TableError` becomes `model::ActionRefusal{"table." + , {{"column", error.column}}, error.message}`. +- `answerQuery`: `validate(query, columnInfos(served.description), served.limits, served.mayRead)`; the query's columns as `orderedColumns(description, query.columns)`; `served.source` is a `detail::PinnedSource` (a `DataSource` holding one snapshot and a copy of the columns, whose `snapshot()` returns that snapshot and whose `subscribe` does nothing); `answerQuery` never takes a snapshot of a live source, and both `apply` (with `ApplyOptions{limits, services, mayRead}`) and `writeGenericPage` read the pinned one; a caller that passes a live source (the tests) gets it pinned by `detail::pinned(ServedTable)`, called on the caller's thread; with `query.columns` absent, the page's columns are the description's minus those `served.mayRead` refuses (invariant 14); `page.revision = served.description.revision`. A description column the source lacks is `TableError{UnknownColumn, column, "the table's source has no column "}`. +- `DescribeTable::invoke` returns `describe(served.description)` (so a model never fills `revision`); it runs on the strand and touches neither the source nor a snapshot. The function result is described once per call; a model whose description does not vary returns the same object each time. +- `detail::serve(model, context)` is the only path from a model to its `ServedTable`; `QueryTable::invoke` and `DescribeTable::invoke` call it on the strand, and Task 17a's `resolve` will too. For a plain table it calls `(model.*Function)(context)`; for a typed table (`TableTraits::row` not `void`) it calls the typed function and builds the description and the `RowsSource` as the next bullet says. +- `typed(fn)`: `fn` is `std::shared_ptr const> (Model::*)(TableContext const&)`. The served description's columns are `detail::reflectedColumns()` in member order, labels `findFieldMeta(id)->label` or `inferTitle(id)`, key the `keyField` of the record registered for `Row`; the source is `RowsSource` keyed by that member. +- `scoped(source, keys)`: a `DataSource` whose snapshot maps its rows, in the shared source's order, onto the shared source's rows whose key is in `keys` (keys the source lacks are skipped); it copies no cell; when the shared source's snapshot pointer changes, the next `snapshot()` rebuilds the row map under a mutex. +- `bench::WideTableModel` (registered `"WideTableModel"`), process-wide state set by `static void configure(WideTableOptions wide, WideTableOptions small)` (defaults `{100'000, 300, 1}` and `{1'000, 30, 1}`). `configure` may be called at any time, from any test case: it replaces both options under the state's mutex and drops both sources; each source is built lazily on the first table call that needs it after that. A request already holding a source keeps it. + - `wide(TableContext const&)`, `small(…)`: the description from the layout (label = id, `group` and `groups` from the layout, `levels` empty), key `id`, one shared `WideTableSource` each, `Services{}`. + - `scoped(…)`: `small`'s source; the description is `small`'s without `d000`…`d005` for every principal except `"extended"` (all 30) and `"restricted"` (also without `t000`…`t006`); `mayRead` is membership in the caller's description. + - Actions: `EditWideRow{std::int64_t row; std::string column; std::int64_t value;}` sets the cell in `wide`, and in `small` when the row and column are there, and returns `EditWideRowResult{ChangedRows<&WideTableModel::wide> wide; ChangedRows<&WideTableModel::small> small;}` with the changed row's key in each it changed; effects `updatesTable` on `wide` and `small`. `ReplaceWideRows{std::uint64_t seed;}` regenerates both; effects `changesTable` on both. + +- [ ] **Step 1: Write the failing local tests** in `tests/table/test_table_description.cpp` + +```cpp +TEST_CASE("table registration: a model with no row type serves a table", "[table][registration]") { + // Spec row "A table from a description", server half. WideTableModel::configure({2'000, 300, 7}, {1'000, 30, 7}). + // served = WideTableModel{}.small(ctx); answerQuery(served, {sort {i001 asc}, filter t001 contains "a", page {0, 10}}): + // 10 keys, equal to the keys of apply over a fresh WideTableSource{1'000, 30, 7} with the same query; + // columns == the 30 ids in layout order; revision == describe(served.description).revision. + // Mutation: answerQuery ignoring query.sort. +} + +TEST_CASE("table registration: two tables of one model are two records and two actions", "[table][registration]") { + // effects::recordNamed("WideTableModel.wide")->keyField == "key" and ->table; same for ".small" and ".scoped". + // effects::recordsOf("Table.wide") == {ActionRecord{.record = "WideTableModel.wide", .keyField = "key", .paths = {}, .opaque = true}}. + // effects::actionsHolding("WideTableModel.wide") == {"EditWideRow", "Table.wide"}; ("…small") == {"EditWideRow", "Table.small"}. + // effects::effectsOf("EditWideRow") == {{Updates, "WideTableModel.wide"}, {Updates, "WideTableModel.small"}}. + // effects::effectsOf("ReplaceWideRows") == {{Changes, "WideTableModel.wide"}, {Changes, "WideTableModel.small"}}. + // Mutation: one record for the model (the record keyed on M instead of FN): the second registration conflicts. +} + +TEST_CASE("changed rows: the walk reports the table at its rows member", "[table][registration]") { + // walkResult() contains {typeid(TableOf<&WideTableModel::wide>), "wide.rows"} and + // {typeid(TableOf<&WideTableModel::small>), "small.rows"}. + // effects::deriveChanges("EditWideRow", R"({"wide":{"rows":[{"key":5}]},"small":{"rows":[{"key":5}]}})", ranOn) + // names record "WideTableModel.wide" with keys [5] and "WideTableModel.small" with keys [5]. + // Mutation: the walker without the heldRecord rule (the registry then refuses EditWideRow's updates). +} + +TEST_CASE("changed rows: updatesTable without its ChangedRows is refused when the registries resolve", "[table][registration]") { + // resolveForTesting(records = {the "WideTableModel.wide" RecordInfo}, slots = {"Bad": walkResult()}, + // declared = {"Bad": {updates>}}) throws EffectsRegistrationError whose problems + // contain "action `Bad` declares updates on record `WideTableModel.wide`, but its result holds no `WideTableModel.wide`". + // NoTableResult is a test struct {std::vector ids;}. +} + +TEST_CASE("table registration: rowScoped and derivedFrom are the record's", "[table][registration]") { + // Test model RtModel registers table "mine" with table::rowScoped and "theirs" with table::derivedFrom + // (RtSource a registered record; RtTouch an action declaring changes). + // recordNamed("RtModel.mine")->privateKeys; recordNamed("RtModel.theirs")->derivedFrom == {typeid(RtSource)} and + // keyField == "key"; deriveChanges("RtTouch", "{}", ranOn) names "RtModel.theirs" with no keys. + // Mutation: keeping "a derived record is keyless" for tables (the process registry fails to resolve). +} + +TEST_CASE("columns by session: two principals get two descriptions", "[table][registration]") { + // Spec row "Columns by session", server half. describe(scoped(ctx "alice")) has 24 columns, "extended" 30, + // "restricted" 17; their three revisions differ. answerQuery for "restricted" with columns absent names its 17; + // with columns {"d000"} it is UnknownColumn "d000"; for "alice" a page's revision is alice's description's. + // A ServedTable whose description lists i001 but whose mayRead refuses it: with columns absent, the page's columns + // lack i001 and no row holds its cell. Mutation: answerQuery not filtering the absent-columns list by mayRead. + // Mutation: answerQuery validating against the source's columns instead of the description's. +} + +TEST_CASE("typed: a typed table writes the page the generic source writes", "[table][registration]") { + // Spec row "The generic page", typed half. TypedRow{std::int64_t id; std::string name; Rational price;}, registered + // as a record keyed `id`; TypedModel::rows returns 3 rows; table "typedRows" registered with typed(&TypedModel::rows). + // The served description: columns id (Integer), name (Text), price (Decimal), labels "Id", "Name", "Price", key "id". + // glz::write_json of answerQuery(typed served, q) == of answerQuery(a ServedTable over a VectorSource of the same + // rows and the same description, q), for q = {sort {price desc}, page {0, 2}}. + // detail::serve<&TypedModel::rows>(model, ctx).description, through describe(), equals what DescribeTable<&TypedModel::rows> + // returns through the bridge. Mutation: DescribeTable building the typed description its own way and dropping the labels. + // Mutation: the typed layer reading cells in glaze's key order while the description lists reflected order differently. +} + +TEST_CASE("scoped: a scope's source reads the shared rows of its keys", "[table][registration]") { + // base VectorSource keys 1..5; scoped(base, {4, 2, 9}): rowCount 2, rowId(0) == 2, rowId(1) == 4, cell(1, c) == base's + // row of key 4; after base->updateRow(index of 4, new cells), scoped->snapshot()->cell(1, c) is the new value. + // Mutation: scoped copying the rows when built (the update is not seen). +} + +TEST_CASE("table action: a failing table function is an error reply and the model stays usable", "[table][registration]") { // Review Focus 5 + // FlakyModel's table "flaky" throws std::runtime_error on its first call and on the second returns a description + // naming column "ghost" its source lacks; the third call is well formed. + // Through Bridge + LocalBackend (as tests/test_coroutine_model.cpp:513 drives a Task handler): the first + // QueryTable<&FlakyModel::flaky> fails with the thrown message; the second fails with ActionRefusal code + // "table.unknownColumn" and arg column "ghost"; the third succeeds with its rows. + // Mutation: answerQuery writing the page column by column without checking the source first (a partial page). +} +``` + +- [ ] **Step 2: Write the failing socket tests** in `tests/net/test_push_tables.cpp`, built on the server-and-two-clients harness of `tests/net/test_push_two_clients.cpp`, with a test model `PtModel` serving tables `first` and `second` over small `VectorSource`s, actions `EditFirst` (→ `ChangedRows<&PtModel::first>`, `updatesTable` first) and `EditBoth` (→ `{ChangedRows<&PtModel::first> first; ChangedRows<&PtModel::second> second;}`, `updatesTable` on both), and an authoriser that refuses principal `bob` both `Table.second` and `EditBoth`. A watch of a record is admitted when the principal may run any action holding it (`remote.hpp:1794-1800`), and `EditBoth` holds `PtModel.second`, so refusing `Table.second` alone would still admit bob: + +```cpp +TEST_CASE("tables: a principal allowed one table's action is refused the other", "[net][tables]") { + // bob (refused Table.second and EditBoth, the only actions holding PtModel.second): watch {PtModel.first} admitted, + // watch {PtModel.second} in refusedRecords; executing Table.second is refused. + // alice: both admitted. Mutation: one record for the model (bob is admitted to both or neither). +} +TEST_CASE("tables: updatesTable on one table marks only its watchers", "[net][tables]") { + // c1 watches PtModel.first, c2 PtModel.second. EditFirst changing key 3: c1 receives a change of PtModel.first + // with keys [3]; c2 receives nothing within 3 × coalesceWindow. +} +TEST_CASE("tables: an action changing two tables sends each table its keys", "[net][tables]") { + // EditBoth changing first key 1, second key 2: c1 gets PtModel.first [1]; c2 gets PtModel.second [2]. +} +TEST_CASE("tables: a page's columns are the caller's", "[net][tables]") { + // Spec row "Columns by session" over the socket: WideTableModel's Table.scoped.describe for alice and restricted + // differ; each one's Table.scoped page lists its own columns and carries its own description's revision. +} +TEST_CASE("tables: a table's reply is built on tableWork", "[net][tables]") { + // ServerConfig::tableWork = a ThreadPoolExecutor{2} wrapped to record its threads' ids. PtModel.first's source is a + // recording source whose snapshot records the thread of every readColumn call. A Table.first query with a sort: + // every recorded read is on a tableWork thread, none on the thread that ran PtModel::first. + // Mutation: RemoteServer not installing the TableRuntimeScope (answerQuery runs inline on the strand). +} +``` + +- [ ] **Step 3: Run them to see them fail** + +Run: `cmake --build --preset cl-debug --target morph_table_tests` +Expected: compile failure, `ServedTable` undeclared. +Socket tests (POSIX only; on Windows in WSL or the Linux CI leg): `cmake --preset linux-everything && cmake --build --preset linux-everything --target morph_net_tests` +Expected: compile failure, `BRIDGE_REGISTER_TABLE` undefined. + +- [ ] **Step 4: Implement** every symbol of Produces and the pinned behaviour, in the files listed; register `bench::WideTableModel` with its three tables and two actions in `wide_table_model.cpp`. Every public symbol gets complete Doxygen. + +- [ ] **Step 5: Run them to see them pass** + +Run: `ctest --test-dir build/cl-debug -R "table registration|changed rows|columns by session|typed:|scoped:|table action" --output-on-failure` +Expected: 9 tests pass. Then the suites the dispatcher change touches: `cmake --build --preset cl-debug --target morph_tests morph_table_tests && ctest --test-dir build/cl-debug --output-on-failure -j 8`, all pass. +Run: `ctest --test-dir build/linux-everything -L net -R "tables:" --output-on-failure` +Expected: 5 tests pass; `ctest --test-dir build/linux-everything -L net` all pass. + +- [ ] **Step 6: Record W1 and W2 over the bridge** + +Extend `"table benchmark: the generic page"` with the `Table.wide` reply of 200 rows × 40 named columns through `LocalBackend` (W1's `ok` envelope bytes, W2's encode on the server side); first `REQUIRE` the reply's 200 keys. Add the rows to "Recorded baselines". + +- [ ] **Step 7: Neighbouring specs, mutations, docs build, commit** + +Write `registry.md`'s `BRIDGE_REGISTER_TABLE` section (what the macro emits, its options, the record's name and key, `TableOf`, the walk's rule for a holder's `heldRecord`, the opaque holding, and the admission rule: a principal may watch a table's record when it may run any action holding it, the table's own `Table.` or an action whose result holds its `ChangedRows`, so refusing a table means refusing all of them) and the `invoke` paragraph; `backend.md`'s `tableWork` row and Fan-out bullet. Nothing about `openMembers`, `partialMember` or `announce` (steps 2c, 2d). Apply each mutation; paste the failing lines into the commit body; run the docs build. + +```bash +git add include/morph/table/description.hpp include/morph/table/runtime.hpp include/morph/core src/ui/process_state.cpp CMakeLists.txt \ + tests/table tests/net docs/spec/core/registry.md docs/spec/core/backend.md docs/spec/table/table.md +git commit -m "feat(core): register tables of a model" +``` + +--- + +## Self-review + +**Spec coverage.** "The fixture", "The data", "How figures are taken" → Task 0 (layout, generator, guards, Release medians). Step 0's records S1, S2, S7, S8, B1, B2, B5, U1–U5, W7, F2–F5 and gate S3 → Task 0 Steps 6–7 and the S3 test. "The table tests are built for WebAssembly and run under Node" → Task 0 Step 5. Step 0a and "Paging in the database", "Rows in a database are a source too", the step-0a Neighbouring-specs row (`engine.md`, `database_pages.md`) → Task 0a. "A table is a description" (the four structs, `revision`), "Rows on the wire" (every encoding, `columns`, `keys`, `revision`), `TableQuery::columns`, W1, W5, W6's codec, W7 → Task 1. "A model's tables" (`TableContext`, `ServedTable`, strand then `tableWork`, services, scope, `scoped`, the action table, the table's record, `TableOf`, options, `ChangedRows`, `updatesTable`/`changesTable`, two members for two tables, the opaque holding), "Typed rows are a convenience", "A table on the server" (one `DataSource` per table and scope, `mayRead` narrowing), invariant 14's `mayRead` half and invariant 15, the fixture as a registered model with three tables and no slow level, W2, the step-1 Neighbouring-specs rows (`registry.md`, `backend.md`) → Task 2. Left to other parts by design: `levels` meaning and level-cache sharing ("A table on the server", second bullet) → Task 5; `announce` → Task 6; the client's refetch on a new `revision` → Task 9; `ChangedRow::from`, `FailedRow` → Task 17c; `Table..resolve` → Task 17a; unloaded and measure cells in the generic page → Tasks 4 and 8c. + +**Step scan.** Each test step names its cases and their assertions with the spec's values; the bodies written as comments state exact inputs and expected values. Code steps give signatures and the rules the tests do not determine (the cell codec, the generator, the hop to `tableWork`). The pinned digest is the one value computed during implementation, and the step says how. + +**Type consistency.** `GenericPage`, `writeGenericPage`, `readGenericPage`, `TableColumn`, `TableDescription`, `ServedTable`, `ChangedRows`, `TableOf` match the roadmap's Interfaces; `TableQuery::columns` and `QueryLimits::maxColumns` land in Task 1, and `ServerConfig::tableWork`, `answerQuery(…, AnswerContext const& = {})` and `TableRuntime`/`currentTableRuntime`/`TableRuntimeScope` are as the index's "Cross-part decisions" 1, 2, 5 and 6 name them. + +**Proportion.** About as long as the spec sections it implements; the code blocks are signatures, layouts and test assertions. diff --git a/docs/superpowers/plans/2026-11-17-one-table-b-slow-columns.md b/docs/superpowers/plans/2026-11-17-one-table-b-slow-columns.md new file mode 100644 index 000000000..c7a977c24 --- /dev/null +++ b/docs/superpowers/plans/2026-11-17-one-table-b-slow-columns.md @@ -0,0 +1,855 @@ +# Slow Columns: Unloaded Cells, Partial Pages, the Level and View Caches, the Window Under Changes — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A table served over `apply` and read through a `PageWindow` shows a first page before a slow level loads, shows its cells as unloaded, loads a level only when a named, sorted or filtered column needs it, and updates exactly the rows that arrived, with no polling. + +**Architecture:** The engine learns a fourth cell state (`Unloaded`), a three-valued filter and an undecided set (Task 3). Pages and the generic page carry which cells are unloaded and answer keyed queries (Task 4). The server gains a shared, immutable-snapshot level cache that loads each value once (Task 5), an `announce`/`attach` path and a view cache so a sort runs once for every watcher (Task 6). The client's `PageWindow` holds columns per page and turns change events into keyed fetches with probes or page refetches, and the session reads and watches without cache entries (Task 7). + +**Tech Stack:** C++23, header-only morph, glaze, Catch2, CMake/Ninja (MSVC preset `cl-debug`; timings on `cl-release`). + +**Spec:** `docs/spec/table/table.md` — "The unloaded cell", "Semantics of an unloaded cell" (all), "Data levels", "Partial pages", "Keyed queries", "Arrival", "The window under changes" (all), "Caching layers" through "Which client cache a screen uses", and their rows of "Tests", "Invariants", "Failure modes", "Performance" (S1–S9, L1–L8, C1–C3, W3, W4, K1–K7, E1–E4), "Neighbouring specs" (steps 2a–2d) and "Delivery order" (steps 2a–2d). Roadmap: `docs/superpowers/plans/2026-11-17-one-table.md`, Tasks 3–7. +Part B of 7; index: 2026-11-17-one-table.md + +## Global Constraints + +Every line of the roadmap's Global Constraints applies; the ones this part touches most: + +- Comments state what the code does now and why; no history, no issue numbers (AGENTS.md). +- Every public symbol has complete Doxygen `@param`/`@tparam`/`@return`; docs build: `cmake -S . -B build/doc -G Ninja -DMORPH_BUILD_DOCUMENTATION=ON -DMORPH_BUILD_TESTS=OFF -DMORPH_BUILD_EXAMPLES=OFF && cmake --build build/doc --target doc` exits 0. +- No company, product or customer names; examples use `lims`. +- Commits authored `yaraslau `, ending with the session's attribution lines. +- No GUI application is launched. +- Every test carries a `// Mutation: …` comment naming the change it was seen to fail on; the task applies that mutation once and pastes the failing line into the commit body. +- Limits verbatim: `QueryLimits::maxKeys = 1000` (`maxColumns = 512` is Task 1's); `PageWindow(pageSize = 200, maxPages = 8, maxKeys = 1000, maxColumns = 96)`; `LevelCacheOptions{maxBytes = 256U << 20U, expectedBytes = 0 (64 bytes a column), batch = 64, orderLease = 60 s, retryAfter = 30 s, retryMax = 600 s}`; `ViewCacheOptions::maxBytes = 64U << 20U`; `effects::kMaxChangeKeys = 4096` (existing); level cache shards: 256. +- Timings are never CI gates. Counts and bytes are, and run as ordinary test cases. Timings run by tag `"[.benchmark]"` on `cl-release` and are written into table.md's Performance tables; each benchmark first `REQUIRE`s that its feature was active. +- A spec that describes code takes its part in the task that builds it: `docs/spec/forms/views.md` (Task 3), `docs/spec/core/backend.md` "Fan-out" (Task 6), `docs/spec/core/registry.md` and `docs/spec/ui/interpreter.md` "Queries and the cache" (Task 7). + +**Commands** (``, `` per step): + +```bash +cmake --preset cl-debug +cmake --build --preset cl-debug --target +ctest --test-dir build/cl-debug -R "" --output-on-failure +# timings +cmake --preset cl-release && cmake --build --preset cl-release --target +build/cl-release/bin/.exe "" +``` + +Targets: `morph_table_tests` (tests/table/*), `morph_tests` (tests/test_*.cpp at the top level), `morph_net_tests` (tests/net/*). Catch2 test names are the ctest names, so `-R` takes a regex over them. `morph_net_tests` is POSIX-only (index decision 18): its cases build and run on Linux with `cmake --preset linux-everything && cmake --build --preset linux-everything --target morph_net_tests && ctest --test-dir build/linux-everything -R "" --output-on-failure`; on Windows only the `cl-debug` targets above run. + +## Interfaces from other parts + +These follow the index's "Cross-part decisions" and Part A (`2026-11-17-one-table-a-foundation.md`). + +- **A1. The answer path** (decision 5) is `table::detail::answerQuery(ServedTable const& served, TableQuery const& query, AnswerContext const& context = {}) -> std::expected` in `description.hpp`. Task 2 defines it with an empty `struct AnswerContext {}`. `QueryTable::invoke` gets the `ServedTable` through `table::detail::serve(model, context)` (decision 23), replaces `served.source` on the strand with a `detail::PinnedSource` of one snapshot, and runs `answerQuery` on the table executor, so `answerQuery` never snapshots a live source (tests that pass a live source get it pinned by `detail::pinned(served)`). Tasks 4 and 5 change its body; Task 6 adds `AnswerContext::views`. +- **A2. Where table work runs** (decisions 1, 6). `ServerConfig::tableWork` (Task 2); `table::TableRuntime{exec::IExecutor* work = nullptr;}`, `table::currentTableRuntime()` and `table::TableRuntimeScope`, installed by `RemoteServer` around a dispatch (Task 2). Task 6 adds `ViewCache* views` to `TableRuntime` and `ServerConfig::views`; `QueryTable::invoke` builds `AnswerContext{.views = runtime->views}` from it. +- **A3. The fixture** (Tasks 0, 2). `WideTableSource` is internally synchronised and `WideTableModel::configure(wide, small)` may be called at any time (it drops both sources; the next table call rebuilds them). The key column is `id`, Integer, value `row + 1`, so row `r`'s key is `r + 1` and a table of n rows has keys 1…n. The model type id is `"WideTableModel"`, so the records are `"WideTableModel.wide"`, `"WideTableModel.small"`, `"WideTableModel.scoped"`. The `recipe` level's columns are already declared by Part A's layout (`bench::WideColumn::level`): `d000`…`d009` (Decimal) and `t001`…`t010` (Text) in `wide`; `small` (`wideColumns(30)`) has only `d000`…`d005` and `t001`…`t006` of them. `scoped` serves `small`'s rows to every principal and varies its columns: every principal lacks `d000`…`d005` except `"extended"` (all 30), and `"restricted"` also lacks `t000`…`t006`. No fixture table scopes rows. +- **A4. The record walk** (Task 2). `ResultSlot` gains `bool opaque = false` and `effects::ActionRecord` gains `bool opaque = false`; `Table.`'s registration adds an opaque slot, so `recordsOf("Table.small") == {ActionRecord{.record = "WideTableModel.small", .keyField = "key", .paths = {}, .opaque = true}}`. A holder type with `heldRecord`/`heldMember()` reports its record at that member. Task 7b adds `open` and `partial` beside `opaque`, and copies all three to `ActionRecordInfo`. +- **A5.** Task 1's `GenericPage`, `writeGenericPage`, `readGenericPage`, `TableQuery::columns` and `QueryLimits::maxColumns` (decision 2), with `validate`'s `columns` rules; the writer writes `Unloaded` as `null` once Task 3 adds it. + +--- + +### Task 3: Unloaded cells in the engine (table.md step 2a) + +Two commits, each its own reviewer gate: **3a** the cell, its order, the filter and the undecided set; **3b** data levels on columns, rows and pages. + +**Spec:** "The unloaded cell", "Semantics of an unloaded cell", "Data levels"; Tests rows (a) engine half, (b) ×3, (c) `levelsNeeded` table, (d) engine half; invariants 1, 2, 3 (without retain), 4, 5, 6 (first and third clause), 7; Failure modes "A level never arrives", "A `Unloaded` cell in a base-level column", "A `columnLevels()` entry naming no column", "An edit of an unloaded cell". + +**Files (3a):** +- Modify: `include/morph/table/data_source.hpp` (`Unloaded`, `Cell`, `ColumnInfo::level`, `TableErrorCode::SourceContract` and its glaze name, `displayText`) +- Modify: `include/morph/table/sort.hpp` (`CellState`, `KeyColumn::read`, `compareKeys`) +- Modify: `include/morph/table/filter.hpp` (`Match`, `evaluate`, `evaluateGroup`, `evaluateRow`, `FilterRun`) +- Modify: `include/morph/table/engine.hpp` (`detail::JobInput`, `detail::JobResult`, `detail::ViewJob`, `Engine`) +- Modify: `include/morph/table/query.hpp` (`apply`: undecided, `totalKind`; `PageRows::{totalKind, undecided, awaiting, failed}`) +- Modify: `include/morph/table/edits.hpp` (`CellEdits::commit`) +- Modify: `tests/table/table_fixtures.hpp` (`levelCol`, `slowColumn`) +- Create: `tests/table/test_table_unloaded_cells.cpp` (add to `morph_table_tests` in `tests/CMakeLists.txt`) +- Test: `tests/table/test_table_sort.cpp`, `tests/table/test_table_filter.cpp`, `tests/table/test_table_changes.cpp` + +**Files (3b):** +- Modify: `include/morph/table/query.hpp` (`ColumnLevel`, `LevelLoaded`, `RowsSnapshot`, `RowsSource`, typed `apply`, `LevelsNeeded`, `levelsNeeded`, `UnloadedRow`, `Page::{unloadedRows, undecided, awaiting, failed}`) +- Modify: `include/morph/forms/forms.hpp` (`detail::mergeSchemaExtras` writes `x-level`), `include/morph/forms/views.hpp` (`buildColumnEntry` copies `x-level`), `include/morph/forms/engine/collection.hpp` (`CollectionColumn::level`) +- Modify: `tests/table/wide_table_model.{hpp,cpp}` (`bench::withUnloaded`), `tests/table/bench_table.cpp` (S1, S2, S4–S6) +- Test: `tests/table/test_table_unloaded_cells.cpp`, `tests/table/test_table_query.cpp`, `tests/test_views.cpp` +- Docs: `docs/spec/forms/views.md` (`x-level` on a schema property; `deriveColumns` copies it) + +**Interfaces:** +- Consumes: `bench::WideTableOptions`, `bench::WideTableSource` (Task 0). +- Produces (3a): + - `EngineStats::filterRuns` (`std::size_t`): incremented by the `Engine` when it applies a job result whose `detail::JobResult::filtered` is set; `ViewJob::plan()` sets `filtered` when it constructs a `FilterRun` (a repair never does) + - `struct Unloaded { friend constexpr bool operator==(Unloaded, Unloaded) noexcept = default; };` + - `using Cell = std::variant;` (Task 8c appends `Measure` after it) + - `ColumnInfo::level` (`std::string`, after `format`; part of `operator==`). It lands in 3a because the base-level rule needs it: a cell may be `Unloaded` only in a column with a level. + - `TableErrorCode::SourceContract` (JSON `"sourceContract"`, appended last) + - `CellState::Unloaded` (appended after `Invalid`) + - `enum class Match : std::uint8_t { No, Yes, Unknown };` in `filter.hpp` + - `static Match CompiledFilter::evaluate(Column const& filter, KeyColumn const& keys, std::uint32_t row);` `static bool passes(...)` returns `evaluate(...) == Match::Yes` + - `static Match CompiledFilter::evaluateGroup(Group const& group, std::span keys, std::uint32_t row);` (keys by column index) + - `Match CompiledFilter::evaluateRow(std::span keys, std::uint32_t row) const;` + - `std::vector FilterRun::take() &&` (the `Yes` rows), `std::vector FilterRun::takeUndecided() &&` (the `Unknown` rows); both keep candidate order; each may be called once, in either order + - `detail::JobInput::previousUndecided`, `detail::JobResult::undecided` (`std::vector`, source rows ascending) + - `std::optional const& detail::ViewJob::failure() const noexcept` (replaces the `std::optional`) + - `std::size_t Engine::undecidedRowCount() const noexcept;` `std::span Engine::undecidedRows() const noexcept;` `void Engine::onUndecidedChange(std::function handler);` + - `PageRows::{TotalKind totalKind; std::int64_t undecided; std::vector awaiting; std::vector failed;}` +- Produces (3b): + - `struct ColumnLevel { std::string_view column; std::string_view level; };` + - `template using LevelLoaded = std::function;` + - `RowsSource(rows, key, kinds = {}, comparators = {}, LevelLoaded loaded = {})`; typed `apply(rows, key, query, options = {}, kinds = {}, comparators = {}, LevelLoaded loaded = {})` + - `struct LevelsNeeded { std::vector order; std::vector display; };` + - `[[nodiscard]] LevelsNeeded levelsNeeded(TableQuery const& query, std::span columns, std::function const& mayRead = {});` + - `struct UnloadedRow { std::int64_t index = 0; std::vector levels; std::vector failed; };` + - `Page::{std::vector unloadedRows; std::int64_t undecided = 0; std::vector awaiting; std::vector failed;}` + - `CollectionColumn::level` (`std::string`) + - Test helpers (3a): `tabletest::levelCol(std::string id, ColumnKind kind, std::string level) -> ColumnInfo`; `tabletest::slowColumn(ColumnKind kind, std::vector cells) -> std::shared_ptr` (one column `c` at level `"slow"`, keys 0…n−1). (3b): `bench::withUnloaded(std::shared_ptr source, std::vector columns, std::string level, double loadedFraction, std::uint64_t seed) -> std::shared_ptr` (a wrapping source: the named columns get `level`, and a deterministic `1 - loadedFraction` of rows read `Unloaded` there) + +#### 3a — the cell, its order, the filter and the undecided set + +- [ ] **Step 1: Write the failing order tests** in `tests/table/test_table_sort.cpp` + +```cpp +// Mutation: compareKeys tying unloaded with empty (rank Empty, Invalid and Unloaded alike). +TEST_CASE("table: unloaded cells are the tail after the empties in both directions", "[table][sort][unloaded]") { + using morph::table::Unloaded; + auto source = tabletest::slowColumn(ColumnKind::Integer, + {Cell{std::int64_t{3}}, Cell{Unloaded{}}, Cell{}, Cell{std::int64_t{1}}, + Cell{Unloaded{}}, Cell{std::string{"x"}}}); + for (auto const dir : {SortDirection::Ascending, SortDirection::Descending}) { + Engine engine{source}; + REQUIRE(engine.setSort({SortKey{.column = "c", .dir = dir}})); + auto const keys = tabletest::viewKeys(engine); + auto const valid = dir == SortDirection::Ascending ? std::vector{3, 0} : std::vector{0, 3}; + CHECK(std::vector(keys.begin(), keys.begin() + 2) == valid); + CHECK(std::vector(keys.begin() + 2, keys.begin() + 4) == std::vector{2, 5}); // empty, invalid + CHECK(std::vector(keys.begin() + 4, keys.end()) == std::vector{1, 4}); // unloaded + } +} +``` + +Every column that holds an `Unloaded` cell in this plan's tests is declared with a level (`levelCol`, `slowColumn`); a base-level one is the `SourceContract` case. + +Add `"table: rows with no unloaded sort cell keep their order under any completion"` (invariant 2): 200 random tables of 40 rows × 2 `Integer` columns at level `"slow"` from `std::mt19937{seed}` with 20% `Unloaded`; for each, 20 random completions replacing every `Unloaded` with an integer; for every pair of rows with no unloaded cell in the sort columns, their relative order under `setSort({a asc, b desc})` is the same before and after. Mutation: `compareKeys` ranking `Unloaded` as `Valid` with key 0. + +- [ ] **Step 2: Write the failing filter tests** in `tests/table/test_table_filter.cpp` + +```cpp +// Mutation: evaluate returning passes (two-valued), which rejects the unloaded row. +TEST_CASE("table: an unloaded cell is unknown to every operator", "[table][filter][unloaded]") { + using morph::table::Unloaded; + struct Case { ColumnKind kind; FilterEntry entry; }; + std::vector const cases{ + {ColumnKind::Integer, {.eq = "1"}}, {ColumnKind::Integer, {.ne = "1"}}, {ColumnKind::Integer, {.lt = "1"}}, + {ColumnKind::Integer, {.le = "1"}}, {ColumnKind::Integer, {.gt = "1"}}, {ColumnKind::Integer, {.ge = "1"}}, + {ColumnKind::Decimal, {.between = std::array{"1", "2"}}}, + {ColumnKind::Text, {.contains = "a"}}, {ColumnKind::Text, {.startsWith = "a"}}, + {ColumnKind::Text, {.endsWith = "a"}}, {ColumnKind::Date, {.eq = "2026-01-01"}}, + {ColumnKind::Bool, {.eq = "true"}}, {ColumnKind::Text, {.isEmpty = true}}, + {ColumnKind::Text, {.notEmpty = true}}}; + for (auto const& [kind, entry] : cases) { + for (bool const exclude : {false, true}) { + for (auto const combine : {Combine::Any, Combine::All}) { + auto source = tabletest::slowColumn(kind, {Cell{Unloaded{}}}); + ColumnFilter filter{.combine = combine}; + (exclude ? filter.exclude : filter.include).push_back(entry); + auto page = apply(*source, TableQuery{.filters = FilterSpec{.columns = {{"c", filter}}}}); + REQUIRE(page.has_value()); + CHECK(page->rows.empty()); + CHECK(page->undecided == 1); + CHECK(page->total == 0); + CHECK(page->totalKind == TotalKind::AtLeast); + } + } + } +} +``` + +Add: +- `"table: a group admits on one loaded match and is unknown otherwise"`: columns `a`, `b` Text at level `"slow"`; group `{columns {a,b}, include {contains "x"}}`; rows `{x, Unloaded}` → view; `{Unloaded, y}` → undecided; `{y, z}` → rejected; `{Unloaded, Unloaded}` → undecided. Mutation: a group evaluated as `all` of its columns. +- `"table: a row unknown at one stage and rejected at a later one is rejected"`: filters `a eq "1"` (row's `a` unloaded) and `b eq "2"` (row's `b` = `"3"`): `rows` empty, `undecided == 0`, `totalKind == Exact`. Mutation: `FilterRun` dropping the undecided mark when a later stage answers `No` (keeping the row undecided). +- `"table: the filter is sound under every completion"` (invariant 1): 200 random tables of 30 rows × 3 columns (`Integer`, `Text`, `Bool`, level `"slow"`) with 25% `Unloaded`, random filters of 1–3 column entries and 0–1 group from `{eq, ne, lt, gt, contains, isEmpty, notEmpty}` in include and exclude; 20 completions each: every row of the view is in the completed view, every rejected row is rejected there. Mutation: `evaluate` answering `No` for `ne` on an unloaded cell. + +- [ ] **Step 3: Write the failing engine tests** — `tests/table/test_table_unloaded_cells.cpp` (header: what the file covers; each case its `// Mutation:`) + +- `"table: a base-level unloaded cell is a source contract error"`: `apply` over a source whose column `c` (`tabletest::col`, level `""`) holds `Unloaded` in row 1 returns `TableErrorCode::SourceContract` with `.column == "c"` when sorting by `c`; an `Engine` sorting by `c` calls `onError` with the same and keeps `viewKeys` as before; the same table with `c` at level `"slow"` sorts with row 1 last. Mutation: key building without the base-level check. +- `"table: cellAt returns the unloaded cell"`: `engine.cellAt(0, 0)` holds `Unloaded` and `displayText(engine.cellAt(0, 0)) == ""`. Mutation: `displayText` returning `"…"`. +- `"table: an edit of an unloaded cell is refused on the cell"`: a `VectorSource` with columns `c` (level `"slow"`, row 0 `Unloaded`) and `d` (base, row 0 `"x"`); `CellEdits edits{*source, commit, &engine}`; `edits.commit({.row = 0, .column = "c", .value = Cell{std::int64_t{5}}})`: the commit function is not called and `edits.error(0, "c") == "not loaded yet"`; an edit of loaded column `d` of the same row is committed. Mutation: `commit` not looking at the cell. +- `"table: selectAll does not select undecided rows"` and `"table: a selected row that becomes undecided is pruned"` (`SelectionOptions{.pruneOnFilter = true}`; row 1 selected, its filtered cell updated to `Unloaded` via `updateRow`: `selection.contains(1)` is false). Mutation: `selectAll` over `view ∪ undecided`. + +Engine half of (d) in `tests/table/test_table_changes.cpp`: + +```cpp +// Mutation: a repair that ignores previousUndecided (an arrived row stays undecided). +TEST_CASE("table: an undecided row that arrives is one insert and one count change", "[table][changes][unloaded]") { + using morph::table::Unloaded; + auto source = tabletest::makeSource({tabletest::levelCol("a", ColumnKind::Integer, "slow")}, + {{Cell{std::int64_t{1}}}, {Cell{Unloaded{}}}, {Cell{std::int64_t{3}}}}); + Engine engine{source}; + REQUIRE(engine.setSort({SortKey{.column = "a"}})); + REQUIRE(engine.setFilter(FilterSpec{.columns = {{"a", ColumnFilter{.include = {FilterEntry{.gt = "0"}}}}}})); + CHECK(engine.undecidedRowCount() == 1); + std::vector counts; + std::vector order; + ViewChange last; // the engine has one view handler: record here rather than through FollowingModel + engine.onViewChange([&](ViewChange const& change) { last = change; order.emplace_back("view"); }); + engine.onUndecidedChange([&](std::size_t count) { counts.push_back(count); order.emplace_back("undecided"); }); + source->updateRow(1, {Cell{std::int64_t{2}}}); + CHECK(tabletest::viewKeys(engine) == std::vector{0, 1, 2}); + REQUIRE(last.ops.size() == 1); + CHECK(last.ops[0].kind == ViewOp::Kind::Inserted); + CHECK(last.ops[0].first == 1); + CHECK(last.ops[0].count == 1); + CHECK(counts == std::vector{0}); + CHECK(order == std::vector{"view", "undecided"}); +} +``` + +Add: +- `"table: an undecided row rejected on arrival is no view operation"`: same table, `updateRow(1, {Cell{std::int64_t{-5}}})`: `onViewChange` not called, `counts == {0}`. Mutation: `onUndecidedChange` called only with a view change. +- `"table: a displayed row whose level arrives is changed in place"` for `ReorderPolicy::Immediate` and `Deferred`: sorted by `a`, filtered on nothing, row 1's non-sort column `b` goes from `Unloaded` to `"x"`: one `Changed` op at view row 1, no `Moved`. Mutation: a repair reporting every dirty row as `Moved`. +- `"table: a cell dropped back to unloaded moves to the tail or leaves the view"` (Review Focus 2): sorted ascending by `a`, unfiltered, row 0's `a` updated from `1` to `Unloaded`: row 0 is last; with `a gt 0` filtered, the same update makes it undecided (`viewKeys` lacks 0, `undecidedRowCount() == 1`). Mutation: `KeyColumn::set` keeping the old key for a cell that became `Unloaded`. +- `"table: under deferred reorder an undecided row takes its true place at once"` (Review Focus 5): `EngineOptions{.reorder = ReorderPolicy::Deferred}`, view `{0: 1, 2: 3}`, undecided row 1 arrives with `2`: `viewKeys == {0, 1, 2}` before `settleNow()`, and `heldRows()` does not contain 1. Mutation: the repair holding entering rows like updated ones. + +- [ ] **Step 4: Run the new cases to verify they fail** + +Run: `cmake --build --preset cl-debug --target morph_table_tests` +Expected: compile errors naming `Unloaded`, `undecided`, `onUndecidedChange`, `totalKind` on `PageRows`. + +- [ ] **Step 5: Implement the cell, `ColumnInfo::level`, `displayText` and `SourceContract`** in `data_source.hpp` + +`ColumnInfo::level` carries the spec's Doxygen ("The data level the column's cells need. Empty is the base level, which every row has when it exists."); `tabletest::col` keeps its signature (level empty). `displayText` gets an `Unloaded` branch returning `{}`. The glaze enumeration of `TableErrorCode` adds `"sourceContract", SourceContract`. + +- [ ] **Step 6: Implement the order** in `sort.hpp` + +`KeyColumn::read` first maps `Unloaded` to `CellState::Unloaded` (before the empty check and the kind switch) and stores no key. `compareKeys` compares a rank before keys, ignoring `dir`: + +```cpp +// Valid 0; Empty and Invalid 1 (they tie); Unloaded 2 (they tie). Ranks never flip with the direction. +int const rankA = rank(keys.states[left]); +int const rankB = rank(keys.states[right]); +if (rankA != rankB) { return rankA < rankB ? -1 : 1; } +if (rankA != 0) { return 0; } +``` + +`RowOrder` already breaks ties by source row. + +- [ ] **Step 7: Implement the three-valued filter** in `filter.hpp` + +- `evaluate`: a cell whose state is `Unloaded` is `Unknown` whatever the entries (`isEmpty`, `notEmpty` and `exclude` included); otherwise `Yes`/`No` exactly as the current `passes` body. +- `evaluateGroup` (plain group; Task 8b adds quantifiers): `Yes` if any column is `Yes`; else `Unknown` if any is `Unknown`; else `No`. +- `evaluateRow`: `No` if any column filter or group is `No`; else `Unknown` if any is `Unknown`; else `Yes`. +- `FilterRun` keeps a mark per candidate (`std::vector`, compacted with `_rows`): a stage answering `No` drops the row, `Unknown` keeps it marked, `Yes` keeps it as it was. `take()` returns the unmarked survivors, `takeUndecided()` the marked ones. + +- [ ] **Step 8: Implement the undecided set** in `engine.hpp` + +- `ViewJob::step` maps an exception to `TableError{ServiceFailed, {}, "a service failed: " + what}`; the three call sites (`Engine` ×2, `apply`) use `failure()` as the error directly. +- Key building: after a needed column's keys are built, and after each dirty row is patched, a column with `level.empty()` holding a `CellState::Unloaded` fails the job with `TableError{SourceContract, , "an unloaded cell in a base-level column"}`. +- `order()`: after the filter run, `_result.undecided = std::move(run).takeUndecided()` sorted ascending. +- `repair()`: each dirty row is evaluated with `evaluateRow`: `Yes` joins `passing`, `Unknown` joins the new undecided rows, `No` neither; `_result.undecided` = (`previousUndecided` minus `dirty`, both ascending: a set difference, no filter run) ∪ the new undecided, ascending. +- `Engine` keeps `_undecided` with `_view` and passes it as `previousUndecided`; on applying a result it sets `_undecided`, then calls `onViewChange` if the view changed, then `onUndecidedChange(count)` if the count differs from the one before the job. +- A row entering the view from the undecided set is not in `previousView`, so under `Deferred` the diff inserts it at its true place; displayed rows keep their held places. + +- [ ] **Step 9: Implement totals and the edit refusal** + +- `apply` fills `PageRows::undecided` from the job's `undecided.size()` and `totalKind = undecided > 0 ? AtLeast : Exact`; `total` stays absent above `totalCap`. +- `CellEdits::commit`: when `_engine` is set and `_engine->viewRowOfKey(request.row)` finds a row whose `cellAt(viewRow, column)` holds `Unloaded` (column found by id in `_engine->columns()`), it records the error `"not loaded yet"` for the cell and returns without queueing; the cell's other edits are unaffected. + +- [ ] **Step 10: Run the target** + +Run: `cmake --build --preset cl-debug --target morph_table_tests && ctest --test-dir build/cl-debug -R "^table" --output-on-failure` +Expected: all pass, the new cases included. + +- [ ] **Step 11: Apply each named mutation, see its test fail, revert. Commit** + +```bash +git add include/morph/table tests/table tests/CMakeLists.txt +git commit -m "feat(table): unloaded cells, their order and the undecided set" +``` + +#### 3b — data levels on columns, typed rows and pages + +- [ ] **Step 12: Write the failing level tests** in `tests/table/test_table_query.cpp` + +```cpp +// Mutation: levelsNeeded returning every level of the columns. +TEST_CASE("table: levelsNeeded names the levels a query sorts, filters and shows", "[table][query][levels]") { + using tabletest::levelCol; + std::vector const columns{tabletest::col("id", ColumnKind::Integer), + levelCol("r1", ColumnKind::Decimal, "recipe"), + levelCol("r2", ColumnKind::Text, "recipe"), + levelCol("s1", ColumnKind::Text, "stock"), + levelCol("p1", ColumnKind::Integer, "price")}; + auto const needed = [&](TableQuery const& query) { return levelsNeeded(query, columns); }; + using V = std::vector; + TableQuery named{.columns = V{"id", "r2"}}; + CHECK(needed(named).order.empty()); + CHECK(needed(named).display == V{"recipe"}); + TableQuery sorted{.sort = {SortKey{.column = "s1"}}, .columns = V{"id"}}; + CHECK(needed(sorted).order == V{"stock"}); + CHECK(needed(sorted).display.empty()); + TableQuery filtered{.filters = FilterSpec{.columns = {{"p1", ColumnFilter{.include = {{.gt = "1"}}}}}}, + .columns = V{"p1"}}; + CHECK(needed(filtered).order == V{"price"}); + CHECK(needed(filtered).display == V{"price"}); + TableQuery grouped{.filters = FilterSpec{.groups = {GroupFilter{.columns = {"r1", "s1"}}}}, .columns = V{}}; + CHECK(needed(grouped).order == V{"recipe", "stock"}); + CHECK(needed(TableQuery{.columns = V{"id"}}).display.empty()); + CHECK(needed(TableQuery{}).display == V{"price", "recipe", "stock"}); // columns absent: every column + CHECK(levelsNeeded(TableQuery{}, columns, [](std::string_view c) { return c != "p1"; }).display == + V{"recipe", "stock"}); // only what the caller may read + CHECK(needed(TableQuery{.columns = V{}}).display.empty()); // columns empty: keys alone +} +``` + +(`TableQuery::columns` is Task 1's, decision 2.) + +In `tests/table/test_table_unloaded_cells.cpp`: + +- `"table: apply reads only the sorted and filtered columns"` (row (a), gate S3 with levels): a recording `DataSource` wrapping `bench::withUnloaded(WideTableSource{{.rows = 1000, .columns = 300, .seed = 7}}, `d000`…`d009`, `t001`…`t010`, "recipe", 0.7, 7)` logs the column of every `cell` and `readColumn` call; `apply` sorted on one recipe decimal and filtered `contains "a"` on one recipe text returns a page; the logged columns are exactly those two. Mutation: `apply` building keys for every column. +- `"table: a typed page names exactly its unloaded rows and levels"` (row (b)): rows `struct LevelRow { std::int64_t id; std::string name; std::int64_t qty; static constexpr std::array columnLevels() noexcept { return {{{"qty", "stock"}}}; } };` ids 0–9, `LevelLoaded` refusing `stock` for odd ids; typed `apply` with `columns` absent: `unloadedRows` is `{{1,{"stock"},{}}, {3,…}, {5,…}, {7,…}, {9,…}}` in index order; with `columns = {"id","name"}` and no sort/filter on `qty`, `unloadedRows` is empty (the level is not needed). Mutation: the typed `apply` dropping `unloadedRows`. +- `"typed: a row's column levels reach the description"` (in `tests/table/test_table_description.cpp`): `detail::serve<&LevelModel::rows>(model, ctx).description` for a typed table of `LevelRow` has column `qty` with `info.level == "stock"` and `levels == {"stock"}`. Mutation: `columnLevels()` applied in `RowsSource` only, not in `reflectedColumns`. +- `"table: a column level naming no column is refused where columns are reflected"`: a row whose `columnLevels()` names `"nope"`: `RowsSource` construction throws `std::invalid_argument` whose message contains `nope`. Mutation: the entry ignored. +- `"table: a typed table without levels is whole"` (invariant 7): typed `apply` with no `LevelLoaded` and no `columnLevels()` yields `unloadedRows`, `awaiting`, `failed` empty, `undecided == 0`, `totalKind == Exact`, and the same `rows`, `total` as before this change. Mutation: `awaiting` filled with every level when `loaded` is empty. + +In `tests/test_views.cpp`: `"views: deriveColumns copies x-level"` — for `LevelRow`, `forms::schemaJson()` has `"x-level":"stock"` on `qty`, and `deriveColumns` gives that column `level == "stock"` (and `CollectionColumn::level` reads it). Mutation: `buildColumnEntry` not copying `x-level`. + +- [ ] **Step 13: Run the new cases to verify they fail** — `cmake --build --preset cl-debug --target morph_table_tests morph_tests`; expected: compile errors naming `level`, `levelsNeeded`, `unloadedRows`, `ColumnLevel`. + +- [ ] **Step 14: Implement levels on typed rows, pages and schemas** + +- `detail::reflectedColumns()` (in `query.hpp`, which Task 2's `typed` also reads) applies `Row::columnLevels()` when the member function exists: each entry sets that column's `level`; an entry naming no column throws `std::invalid_argument{"columnLevels() names no column: "}`. So `RowsSource` and Task 2's `typed` description both carry the levels, and 3b changes `typed`'s description builder (`detail::serve`'s typed branch) to set `description.levels` to the sorted, unique non-empty levels of its columns. `RowsSnapshot` holds the `LevelLoaded` and a per-column level; `cell` and `readColumn` return `Unloaded{}` for a column whose level the predicate refuses, without calling the member's reader. +- `levelsNeeded`: `order` = levels of every sort key's column, every column with a filter entry, every column of every group (Task 8a adds the distinct request's column or group); `display` = levels of `query.columns`, or of every column `mayRead` allows when absent; both sorted, unique, without `""`. Pure; no rows read. +- Typed `apply`: copies `undecided`, `totalKind`, `awaiting`, `failed` from `PageRows`; builds `unloadedRows` for the page's rows from `LevelLoaded` over the needed levels (`order ∪ display`), each row's `levels` sorted; `failed` per row stays empty (a predicate cannot say a load failed). `awaiting` = the `order` levels the predicate refuses for at least one row of the view or the undecided set. Without a predicate all three stay empty. +- `forms::detail::mergeSchemaExtras` writes `"x-level": ` on each property `columnLevels()` names; `views::buildColumnEntry` adds `"x-level"` to the keys it copies; `CollectionColumn::level` reads it. +- `bench::withUnloaded` as the Interfaces block states; its snapshot hands runs through `readColumn`. + +- [ ] **Step 15: Add the benchmarks** in `tests/table/bench_table.cpp`, tag `"[.benchmark][table][unloaded]"`, each first `REQUIRE`-ing unloaded cells present: +- S1: `apply`, sort one recipe decimal, page 200, 100,000 × 300 via `withUnloaded(…, 0.7, 7)`, against the same with `loadedFraction = 1.0`. S2: text `contains` on a recipe text column. +- S4: `levelsNeeded` over 300 columns, 8 sort keys, 32 filtered columns. +- S5: an `Engine` sorted on a recipe decimal, 100,000 rows; `updateRow` of 200 and of 1,000 rows from `Unloaded` to values. Gates as ordinary cases (`"table: an arrival is one repair"`): `EngineStats` shows one repair job and no full sort, and the arrived rows sit at the view rows a fresh `apply` over the same snapshot gives (the oracle). +- S6: 50,000 undecided rows (filter on a recipe column with 50% loaded), 1,000 decided: gate case `"table: deciding rows runs no filter over all rows"`: one repair, `EngineStats::filterRuns` unchanged by the arrival, and each decided row is in the view or rejected as the oracle has it. + +- [ ] **Step 16: Run the targets** — `ctest --test-dir build/cl-debug -R "^(table|views)" --output-on-failure`: all pass. Record S1, S2, S4–S6 on `cl-release` into table.md's "Benchmarks: server" table (median, p10–p90, load average). + +- [ ] **Step 17: Update `docs/spec/forms/views.md`** (`x-level` on a schema property; `deriveColumns` copies it to the column as it copies `x-comparator`). Docs build green. + +- [ ] **Step 18: Mutations, then commit** + +```bash +git add include/morph/table include/morph/forms tests docs/spec +git commit -m "feat(table): data levels on columns, typed rows and pages" +``` + +--- + +### Task 4: Partial pages and keyed queries on the wire (table.md step 2b) + +**Spec:** "Partial pages", "Keyed queries", "Data levels" (the `columns` rules of `validate`); Tests rows "`TableQuery` on the wire", "The generic page" (unloaded); Failure modes "A keyed query to a model that does not translate keys", "A query's `columns` names a column the table lacks…". + +**Files:** +- Modify: `include/morph/table/query.hpp` (`TableQuery::keys`, `TableQuery::retry`; `QueryLimits::maxKeys`; `validate`'s keyed rules; `PageRows::indices`; `Page::indices`; `UnloadedLevels`; `detail::ComputedView`, `detail::computeView`, `detail::pageRows`; keyed `apply`) +- Modify: `include/morph/table/description.hpp` (`GenericPage::{unloadedRows, undecided, awaiting, failed, indices}`; the writer and reader; `detail::answerQuery`, A1) +- Test: `tests/table/test_table_query.cpp`, `tests/table/test_table_description.cpp`; benchmarks in `tests/table/bench_table.cpp` + +**Interfaces:** +- Consumes: Task 3's `PageRows`, `UnloadedRow`, `Page`; Task 1's `GenericPage`, `writeGenericPage(PageRows const&, DataSource const&, std::span columns)`, `readGenericPage(std::string_view, TableDescription const&)`; `TableQuery::columns`, `QueryLimits::maxColumns` and their `validate` rules (Task 1, A5). +- Produces: + - `TableQuery::keys` (`std::optional>`, after `columns`) and `TableQuery::retry` (`bool retry = false`, after `keys`; Task 8a puts `distinct` before it, as the spec's block orders them) — decision 25 + - `QueryLimits::maxKeys = 1000` + - `PageRows::indices`, `Page::indices` (`std::vector`, after `failed`) + - `struct UnloadedLevels { std::span levels; std::span failed; };` + - `struct detail::ComputedView { std::vector rows; std::vector undecided; };` + - `[[nodiscard]] std::expected detail::computeView(DataSource const& source, TableQuery const& validated, ApplyOptions const& options);` + - `[[nodiscard]] PageRows detail::pageRows(detail::ComputedView const& view, RowSnapshot const& snapshot, TableQuery const& validated, QueryLimits const& limits, std::span viewRowOf = {});` (`viewRowOf`: view row by source row, `UINT32_MAX` for absent; built on demand when empty) + - `GenericPage::{std::vector unloadedRows; std::int64_t undecided = 0; std::vector awaiting; std::vector failed; std::vector indices;}` + +- [ ] **Step 1: Write the failing wire tests** in `tests/table/test_table_query.cpp` + +```cpp +// Mutation: TableQuery::keys declared as std::vector (text keys fail to read). +TEST_CASE("table: columns and keys round-trip through glaze", "[table][query][keyed]") { + TableQuery query; + query.columns = std::vector{"id", "name"}; + query.keys = std::vector{RowId{std::int64_t{7}}, RowId{std::string{"A-1"}}}; + query.retry = true; + std::string json; + REQUIRE_FALSE(glz::write_json(query, json)); + TableQuery back; + REQUIRE_FALSE(glz::read_json(back, json)); + CHECK(back.columns == query.columns); + CHECK(back.keys == query.keys); + CHECK(back.retry); + TableQuery bare; + REQUIRE_FALSE(glz::read_json(bare, R"({"sort":[],"filters":{},"page":{"offset":0,"limit":200}})")); + CHECK_FALSE(bare.columns.has_value()); + CHECK_FALSE(bare.keys.has_value()); + CHECK_FALSE(bare.retry); +} +``` + +Add: +- `"table: validate refuses a keyed query out of bounds"`: `keys` with `page.offset = 10` → `InvalidSpec`; 1,001 keys → `LimitExceeded`; a text key of 257 bytes → `LimitExceeded`. Mutation: the keyed checks skipped. +- `"table: a keyed query to a table without keyed queries is unavailable"` (`test_table_description.cpp`): `detail::answerQuery` over a description with `capabilities.keys = false` answers a query with `keys` set `TableErrorCode::Unavailable`; the same table answers a page. Every other table answers keyed queries through `answerQuery`, so this is the one server-side form of the spec's "a model that does not translate keys"; the client's fallback to `invalidate` is `WindowRows`'s (Task 9). Mutation: `answerQuery` ignoring `capabilities.keys`. +- `"table: an empty columns list gives a page of keys"` (generic, `test_table_description.cpp`; skip if Task 1 already pins it): `detail::answerQuery` over a 3-row table with `columns = {}` returns `columns.empty()`, `keys == {0,1,2}`, every row an empty `GenericRow`. Mutation: an empty list read as absent. +- `"table: a keyed reply carries the view's totals and each row's view row"`: + +```cpp +// Mutation: indices filled with positions in the reply instead of view rows. +TEST_CASE("table: a keyed reply carries the view's totals and each row's view row", "[table][query][keyed]") { + using morph::table::Unloaded; + auto source = tabletest::makeSource({tabletest::levelCol("a", ColumnKind::Integer, "slow")}, + {{Cell{std::int64_t{40}}}, {Cell{std::int64_t{10}}}, {Cell{Unloaded{}}}, + {Cell{std::int64_t{30}}}, {Cell{std::int64_t{-1}}}, {Cell{std::int64_t{20}}}}); + TableQuery query{.sort = {SortKey{.column = "a"}}, + .filters = FilterSpec{.columns = {{"a", ColumnFilter{.include = {{.gt = "0"}}}}}}}; + query.keys = std::vector{RowId{std::int64_t{3}}, RowId{std::int64_t{1}}, RowId{std::int64_t{2}}, + RowId{std::int64_t{4}}, RowId{std::int64_t{99}}}; + auto page = apply(*source, query); + REQUIRE(page.has_value()); + CHECK(page->rows == std::vector{1, 3}); // view order; 2 undecided, 4 rejected, 99 absent + CHECK(page->indices == std::vector{0, 2}); + CHECK(page->total == 4); + CHECK(page->undecided == 1); + CHECK(page->totalKind == TotalKind::AtLeast); +} +``` + +- `"generic page: unloaded rows, undecided and indices round-trip"` (`test_table_description.cpp`): a `GenericPage` with `unloadedRows {{1, {"recipe"}, {"recipe"}}}`, `undecided 40`, `awaiting {"stock"}`, `failed {"recipe"}`, `indices {5, 9}` writes and reads back equal; JSON lacking all five reads as a fully loaded page (members empty, `undecided == 0`). Mutation: the reader requiring `unloadedRows`. +- `"generic page: an unloaded cell is null and named per level"`: `writeGenericPage` over a 2-row source whose row 1 has `Unloaded` in recipe column `r` and named `columns {"id","r"}` writes `rows[1][1]` as `null` and `unloadedRows == {{1, {"recipe"}, {}}}`; a level whose columns are not named is not listed. Mutation: the writer emitting a sentinel string for `Unloaded`. + +- [ ] **Step 2: Run to verify failure** — `cmake --build --preset cl-debug --target morph_table_tests`: compile errors naming `keys`, `indices`, `maxKeys`. + +- [ ] **Step 3: Implement `validate`'s keyed rules** in `query.hpp` + +(`columns` is validated by Task 1.) `keys`: `page.offset != 0` → `InvalidSpec` ("a keyed query has no page"); more than `maxKeys` → `LimitExceeded`; a text key longer than `maxTextLength` → `LimitExceeded`. (Task 8a adds "keys with `distinct`" → `InvalidSpec`.) + +- [ ] **Step 4: Split `apply` and answer keyed queries** + +`apply` = `validate` → `computeView` → `pageRows`. `pageRows` fills `total` (absent above `totalCap`), `totalKind`, `undecided`; for a non-keyed query `offset` and the page's rows as before; for a keyed one, `rows` = the source rows whose key is in `keys` and whose view row exists, ascending by view row, `indices` = those view rows, `offset = 0`. Keys are matched by scanning the snapshot's `rowId`s once against an `std::unordered_set` of the requested keys. The typed `apply` copies `indices`. + +- [ ] **Step 5: Implement the generic page members and the keys capability** + +`detail::answerQuery` refuses a query with `keys` set as `TableError{Unavailable, {}, "this table answers no keyed query"}` when `served.description.capabilities.keys` is false. + +`writeGenericPage` copies `totalKind`, `undecided`, `awaiting`, `failed`, `indices` from `PageRows`, and lists row `i` in `unloadedRows` with the sorted levels of the named columns whose cell it wrote as `Unloaded` (Task 5's answer path replaces this with the level snapshots' states). The reader accepts every new member as optional. + +- [ ] **Step 6: Run the target** — `ctest --test-dir build/cl-debug -R "^(table|generic page|description)" --output-on-failure`: all pass. + +- [ ] **Step 7: Benchmarks** in `bench_table.cpp` (`"[.benchmark][table][wire]"`): W3 — bytes of the W1 page (200 rows, 40 named columns from the 100,000 × 300 fixture) with all 200 rows naming one unloaded level, minus the same page with none, as a share of W1; W4 — a keyed reply of 20 keys and 40 columns: bytes, encode, decode. Record into table.md's "Benchmarks: wire". + +- [ ] **Step 8: Mutations, then commit** + +```bash +git add include/morph/table tests/table +git commit -m "feat(table): partial pages and keyed queries" +``` + +--- + +### Task 5: The level cache (table.md step 2c, first half) + +**Spec:** "The server level cache" (all thirteen rules and the layout, size and use paragraphs), "A table on the server" (level caches by record); Tests rows "A value is loaded once" (cache half), "Concurrent queries share a load", "Reloaded once after invalidation" (cache half), "A load in flight at invalidation", "Visible before told", "Eviction", "A level's layout is the cache's" (cache half), "A cache's size is deep", "A reader blocks no one", "Order uses are leased", "A failed load" (cache half); invariants 11, 12, 13, 14 (layout clause); gates L1–L8. + +**Files:** +- Create: `include/morph/table/level_cache.hpp`; `tests/table/test_table_level_cache.cpp` (in `morph_table_tests`) +- Modify: `include/morph/core/change_types.hpp` (`effects::AnnounceSink`, `effects::IChangeObserver`: the cache derives from it, so it lands here rather than in Task 6) +- Modify: `include/morph/table/description.hpp` (`detail::answerQuery`, A1: level snapshots, `LeveledSource`, `want`, `unloadedRows`/`awaiting`/`failed` from snapshots) +- Modify: `tests/table/wide_table_model.{hpp,cpp}` (the slow level `recipe`, `EditWideRecipe`), `tests/table/bench_table.cpp` (L1–L3, L6) + +**Interfaces:** +- Consumes: Task 3's `Unloaded`, `levelsNeeded`; Task 4's `computeView`, `pageRows`, `GenericPage` members; Task 2's `ServedTable::levels` (`std::vector>`). +- Produces (in `morph::table`, `level_cache.hpp`): the spec's block verbatim — `LevelRead`, `LevelUse`, `LevelSnapshot`, `LevelSource`, `LevelCacheOptions`, `LevelCacheStats`, `LevelCache` with `Loader`, `SizeOf`, `Cells`, both constructors, `invalidate`, `invalidateAll`, `onLoadFailed`, `stats`, and `using CellLevelCache = LevelCache>;` — plus: + - `[[nodiscard]] std::size_t cellBytes(Cell const& cell) noexcept;` (in `data_source.hpp`: `sizeof(Cell)` plus a string's heap capacity when it is not in the small buffer) + - `struct LevelInput { std::shared_ptr source; std::shared_ptr snapshot; };` + - `class LeveledSource final : public DataSource` — `LeveledSource(std::shared_ptr base, std::vector columns, std::vector levels);` `columns()` the given columns; `snapshot()` reads a base-level column from `base` by id and a level column from its level's snapshot at the column's place in `LevelSource::columns()` (by id), `Unloaded` unless `read` answers `Loaded`; `subscribe`/`unsubscribe` do nothing (it is one request's immutable view); `[[nodiscard]] LevelRead state(std::size_t row, std::string_view level) const;` +- Produces (in `morph::effects`, `change_types.hpp`): + - `using AnnounceSink = std::function> keys)>;` + - `class IChangeObserver { public: virtual ~IChangeObserver() = default; virtual void changed(Changes const& changes) = 0; virtual void bind(AnnounceSink sink) = 0; };` (special members as the file's other interfaces declare them) +- Fixture: `WideTableOptions::recipePreloaded` (`double`, default `1.0`), `WideTableOptions::recipeBatch` (`std::size_t`, default `64`), `bench::WideTableModel::recipeGate()` returning a `bench::LoadGate&` with `hold()`, `release()` (every call from now on passes) and `release(std::size_t calls)` (that many loader calls pass, then it holds again), `loaderCalls() -> std::vector>`; action `EditWideRecipe{}` declaring `changes` on the recipe record the level's `invalidatedBy` names. The recipe level's columns are Part A's `d000`…`d009` and `t001`…`t010` (A3); the cache's `columns` list them in that order. + +- [ ] **Step 1: Write the failing cache tests** in `tests/table/test_table_level_cache.cpp` + +Shared helpers in the file's anonymous namespace: `tabletest::ManualExecutor loads;` a `CellLevelCache` over options `{.records = {"T.rows"}, .level = "recipe", .columns = {levelCol("r", Text, "recipe")}, .loads = &loads, .clock = [&] { return now; }}` and a loader that logs its keys and returns `{key, {Cell{"v" + text(key)}}}` per key; `keys(n)` gives `RowId{0}`…`RowId{n-1}`; `readCell(snapshot, key)`; an `AnnounceSink` that records `(record, keys)` and, when `readOnAnnounce` is set, reads the cache's snapshot for each key at call time. + +```cpp +// Mutation: want not checking the cache (loads held keys again): loadedKeys 400. +TEST_CASE("level cache: a value is loaded once", "[table][levels]") { + Rig rig; + REQUIRE(rig.cache.want(keys(200), LevelUse::Display)); + rig.loads.runAll(); // until idle + REQUIRE(rig.cache.want(keys(200), LevelUse::Display)); + rig.drain(); + CHECK(rig.cache.stats().loadedKeys == 200); + CHECK(rig.loggedKeys() == keys(200)); // each once, in any order: compared sorted + CHECK(readCell(*rig.cache.snapshot(), RowId{std::int64_t{5}}) == Cell{std::string{"v5"}}); +} +``` + +Add, each with the spec's mutation: +- `"level cache: concurrent wants share one load"`: loader blocks on a latch; 8 `std::thread`s call `want(keys(60), Display)`; after release and drain, `loaderCalls == 1`. Mutation: no in-flight set. +- `"level cache: invalidated keys are absent and load once more"`: 200 loaded; `invalidate(keys 0..9)`: those read `Unloaded`, the rest `Loaded`; two `want(keys(200))` load exactly keys 0–9, once; `invalidatedKeys == 10`. Mutation: `invalidate` leaving the values. +- `"level cache: the observer invalidates by record"`: `changed(Changes{.records = {{"T.rows", keys {"3"}}}})` makes key 3 absent and announces nothing; `{{"T.rows", nullopt}}` empties the level; `{{"Recipe", nullopt}}` with `invalidatedBy {"T.rows","Recipe"}` empties it and announces `("T.rows", nullopt)`; a record in neither list changes nothing. Mutation: `changed` ignoring the record. +- `"level cache: a load in flight at invalidation does not land"`: hold the loader on key 1; `invalidate({1})`; release returning the old value: key 1 reads `Unloaded`, `droppedLoads == 1`; the next `want({1})` calls the loader. Mutation: storing without the generation check. +- `"level cache: a want after an invalidation does not wait for the stale load"` (Review Focus 1): hold the loader on key 1; `invalidate({1})`; `want({1})` again before releasing: after release and drain the loader was called twice for key 1, the second call's value is stored, `droppedLoads == 1`. Mutation: "in flight" ignoring the generation, which leaves key 1 unloaded after the stale load is dropped. +- `"level cache: values are stored before they are announced"`: `readOnAnnounce`: every announced key reads `Loaded` at announce time; after `invalidateAll()` through `changed` with another record, the announce sink reads every key `Unloaded`. Mutation: announcing before storing. +- `"level cache: eviction drops the least recently used"`: `maxBytes` fitting 100 values; load 100, read keys 50–99 by `want`, load 50 more: keys 0–49 read `Unloaded`, 50–149 `Loaded`, `evictedKeys == 50`, the announce sink got only the loads' keys (no event for an eviction); `want(keys 0..9)` loads them again. Mutation: evicting the most recently used. +- `"level cache: a value's layout is the cache's"`: cache columns `{r1, r2, r3}`; `LeveledSource` A with description columns `{id, r3, r1}`, B with `{id, r2}`: A's row 0 reads `r3`'s value under `r3` and `r1`'s under `r1`; B reads `r2` under `r2`; one loader call served both. Mutation: reading the value by the caller's column order. +- `"level cache: size is deep"`: values of twenty text cells of 40 characters; `maxBytes = 100 * `; after loading 150 rows, `stats().bytes <= maxBytes` and `evictedKeys == 50`. Mutation: `SizeOf` as `sizeof(Value)`. +- `"level cache: a held snapshot blocks no store"`: hold `snapshot()`; on another thread `want` + drain, `invalidate`, `changed(...)` each complete within 1 s (`std::future::wait_for`); the held snapshot still reads its old values and its `generation()` differs from a new snapshot's. Mutation: a snapshot holding a shared lock. +- `"level cache: order uses are leased and the second is refused"`: `maxBytes` fitting 150 values, `expectedBytes` = one value's size; `want(keys 0..99, Order)` loads 100; `want(keys 100..199, Order)` returns `TableErrorCode::Unavailable`, `refusedOrders == 1`, loader calls unchanged; a `Display` load of 100 other keys evicts none of 0–99; advance the clock past `orderLease`: a further `Display` load evicts from 0–99. Mutation: no lease (the loader's call count grows without end over ten alternating rounds: assert `loaderCalls` after ten rounds equals that after one). +- `"level cache: leased values outgrowing the bound stop the order load"`: `expectedBytes` one tenth of the true size; `want(keys(100), Order)` is accepted; while loading, leased bytes exceed `maxBytes`: the remaining order batches are not loaded, the lease is dropped, `("T.rows", nullopt)` is announced, and the next `want(keys(100), Order)` is `Unavailable`. Mutation: the over-bound check skipped. +- `"level cache: a failed load is remembered"`: the loader returns no value for keys 0–9 of 20: those read `Failed`; `failedLoads == 10`; `onLoadFailed` got keys 0–9 once; the sink announced keys 0–9 once; `want(keys 0..9)` loads nothing until `retryAfter` (30 s) on the clock, then once; a second failure waits 60 s; `want(…, Display, true)` loads at once; `invalidate({3})` clears key 3's failure. A key the loader returns with an empty value reads `Loaded` with empty cells. Mutation: a failed key wanted again by every request. A loader that throws fails every key of the call. +- `"level cache: a level no cache serves is unloaded and awaited"` (in `tests/table/test_table_description.cpp`; spec failure mode "A filter on a level no cache serves"): a `ServedTable` whose description has column `r` at level `"recipe"` and whose `levels` is empty, filtered `r contains "a"`: `answerQuery` returns no rows, `undecided` = every row, `awaiting == {"recipe"}`, and a recording `LevelSource` for another level registered beside it saw no `want`. Mutation: a level without a source read from the base source. +- `"table: change keys round-trip awkward keys"` (in `tests/table/test_table_query.cpp`; Review Focus 3): `changeKeys` then `rowIdOf` gives back `RowId{std::string{"a\"b\\c é"}}` and `RowId{std::int64_t{9'007'199'254'740'993}}` exactly; `rowIdOf` of `true`, `1.5` and `{}` is `nullopt`; `changeKeys` of `RowId{7}` and `RowId{"x"}` writes `7` and `"x"`. Mutation: writing an integer key through `double`. + +- [ ] **Step 2: Run to verify failure** — `cmake --build --preset cl-debug --target morph_table_tests`: `level_cache.hpp` not found. + +- [ ] **Step 3: Implement `level_cache.hpp`** (the classes as the Interfaces block; the spec's Doxygen on each member) + +The decisions the rules and tests leave open: + +- **Storage.** 256 shards by `std::hash`; each `std::shared_ptr`, a `Shard` being an immutable `std::unordered_map>`. A `Slot` holds the `Value` (or the failure: count and time), its deep bytes, and `mutable std::atomic lastUse` (a cache-wide use counter) and `mutable std::atomic leaseUntil` (clock ticks; 0 none). `snapshot()` copies the 256 pointers and a generation under the mutex; the mutex is held only to copy pointers or swap in changed shards. Every store, removal and observer call copies only the shards it touches and bumps the generation. +- **Generations.** A key's generation lives in `std::unordered_map` beside the shards (under the same mutex); `invalidate` bumps each key's, `invalidateAll` bumps a level epoch. A load records `(epoch, key generation)` per key when it starts; the store keeps a key only when both are unchanged; others are dropped and counted in `droppedLoads`. A key is "in flight" only while a load holding its current generation runs, so a `want` after an invalidation starts a new load. +- **Queue and batches.** Two LIFO stacks of wanted keys, `Display` and `Order`: each `want` pushes its keys as one block, in the order given, on top of its stack, and a batch takes keys from the top block in that order (the newest want first, each want in its own order). One loader call runs at a time per cache: when none runs, the next batch (up to `batch` keys, `Display` stack first) is posted to `loads`; its completion stores, announces, and posts the next. Keys held, in flight, or failed inside their delay are not queued. +- **Order fit.** `want(keys, Order)` computes `keys.size() × mean` (mean = stored bytes / stored values; `expectedBytes` while none; 64 × columns when that is 0) plus the bytes of slots leased by other keys; over `maxBytes` it returns `Unavailable` and changes nothing. Otherwise every key's lease becomes `now + orderLease` (a key still loading receives it when stored). +- **Eviction.** After a store, while `bytes > maxBytes`, evict the slots with the smallest `lastUse` whose lease is past; a `Display` value stored with nothing evictable is stored with `lastUse = 0` so it goes first. While leased bytes exceed `maxBytes`, clear the `Order` stack, set every lease to 0, and announce each of `records` without keys. +- **Failure delay.** `retryAfter × 2^(failures − 1)`, capped at `retryMax`, read from `clock` when `want` sees the key. +- **Announce.** After a store, the stored keys (written with `changeKeys` — this task adds `[[nodiscard]] std::vector changeKeys(std::span keys);` and `[[nodiscard]] std::optional rowIdOf(glz::raw_json const& key);` to `query.hpp`: an integer key is a JSON number, a text key a JSON string, anything else `nullopt`) go to the sink once per record of `records`; failed keys likewise, once. +- **Observer.** `changed` runs on the server strand and only removes: for each change whose record is in `invalidatedBy` — a record of `records` with keys invalidates those keys (unreadable keys invalidate the level), without keys the level; any other record invalidates the level and announces each of `records` without keys. +- `stats()` reads atomics; `bytes` is the deep total. + +- [ ] **Step 4: Implement `detail::answerQuery` with levels** (A1) in `description.hpp` + +For a validated query: `levelsNeeded`; one snapshot per needed level (`ServedTable::levels` by `level()`); a `LeveledSource` over Task 2's `detail::PinnedSource` (the one source snapshot the reply reads), the description's columns and those snapshots; `computeView` and `pageRows` over it; the writer over the reply's rows and named columns; `unloadedRows[i].levels` = the needed levels whose `state(row, level)` is not `Loaded`, `failed` = those `Failed`; `PageRows::awaiting` = the `order` levels with an `Unloaded` state for some row of the view or the undecided set, `failed` = those with a `Failed` one. Then `want`, every call with `retry = query.retry` (decision 25): each `display` level with the reply's keys (`Display`), each `order` level with the keys of the view and the undecided rows (`Order`); a needed level no entry of `served.levels` serves reads `Unloaded` for every row, counts as `awaiting` when it is an `order` level, and is wanted from no one; an `Unavailable` from an `Order` want refuses the query with `TableError{Unavailable, , …}` — the want happens before the reply is returned, so a refused order use returns no page. + +- [ ] **Step 5: Add the fixture's slow level** in `wide_table_model.{hpp,cpp}`: one `CellLevelCache` shared by `wide` and `small` (`records` both; `columns` = `d000`…`d009`, `t001`…`t010` from `wideColumns(300)`; `batch` = `WideTableOptions::recipeBatch`), loader over the generator's recipe cells, sleeping `loadDelay` per key or waiting on `recipeGate()`; `recipePreloaded` of the rows stored when the cache is built (rows chosen by `(seed, row)`, so the same seed preloads the same rows); `invalidatedBy` = both table records and the recipe record `EditWideRecipe` changes. Each table function's `ServedTable::levels` is `{the cache}`; its columns keep Part A's `info.level` (`"recipe"` on the recipe columns) and its description keeps `levels = {"recipe"}`. `configure` rebuilds the cache with the sources. The default `recipePreloaded = 1.0` keeps every Part A page test (which reads recipe values) unchanged; the level tests and benchmarks set `0` or `0.7` explicitly. + +- [ ] **Step 6: Run the target** — `ctest --test-dir build/cl-debug -R "^(level cache|table|generic page)" --output-on-failure`: all pass. + +- [ ] **Step 7: Benchmarks** (`"[.benchmark][levels]"`, loader sleeping a fixed time per key): L1 (snapshot + 200 reads; 100,000 reads), L2 (bytes per row at twenty columns, and for 100,000 rows), L3 (`invalidate` of 1,000 keys; the observer call for a 4,096-key change; both while another thread holds a snapshot 40 ms — gate case `"level cache: invalidation does not wait for a reader"` asserts each returns before the reader releases), L6 (loader return to announce per batch of 64 at 100,000 held rows; gate case asserts shards copied ≤ keys stored, through a test-only counter `LevelCacheStats` does not expose — read it with a `detail::` accessor). L4, L5, L7, L8 are the gate cases of Step 1 ("concurrent wants", "a value is loaded once", "order uses", "a failed load"); L8's ten page requests inside `retryAfter` are asserted in Task 6's bridge case. Record into table.md's "Benchmarks: server level cache". + +- [ ] **Step 8: Mutations, then commit** + +```bash +git add include/morph/table include/morph/core/change_types.hpp tests/table tests/CMakeLists.txt +git commit -m "feat(table): the level cache" +``` + +--- + +### Task 6: Announce, attach and the view cache (table.md step 2c, second half) + +**Spec:** "Arrival" steps 1–3, "The view cache", "A model's tables" (row scope), "The server level cache" (how the server learns of a cache); Tests rows "`announce`", "A cached view is the view", "One sort for many watchers", "No value crosses a scope", "A row-scoped table", "A value is loaded once" (two clients), "Reloaded once after invalidation" (through the server), "A level's layout is the cache's" (bridge), "A failed load" (bridge), (a) bridge, (c) bridge; invariants 13, 14, 29; gates S7, S9, C1–C3. + +**Files:** +- Create: `include/morph/table/view_cache.hpp`; `tests/table/test_table_view_cache.cpp` (in `morph_table_tests`); `tests/net/test_push_unloaded_rows.cpp` (in `morph_net_tests`, which links `morph_wide_table_model`) +- Modify: `include/morph/core/remote.hpp` (`RemoteServer::announce`, `attach`, the observer call in `fanOut`, `ServerConfig::views`, the server's `ViewCache` installed in its `TableRuntime`) +- Modify: `include/morph/table/description.hpp` (`AnswerContext::views`; `detail::answerQuery` through the view cache; `rowScoped` with an empty `scope`; `TableRuntime::views`; `QueryTable::invoke` passing it) +- Test: `tests/net/test_push_tables.cpp` +- Docs: `docs/spec/core/backend.md` "Fan-out" (`tableWork`, `announce`, `attach`, `IChangeObserver`, the view cache) + +**Interfaces:** +- Consumes: Task 5's `LevelCache`, `LeveledSource`, `IChangeObserver`, `AnnounceSink`, `changeKeys`, `rowIdOf`; Task 4's `computeView`, `pageRows`; A1–A4. +- Produces: + - `std::expected RemoteServer::announce(std::string_view record, std::optional> keys);` + - `void RemoteServer::attach(std::shared_ptr observer);` + - `ServerConfig::views` (`table::ViewCacheOptions`) + - `struct ViewCacheOptions { std::size_t maxBytes = 64U << 20U; };` + - `struct ViewCacheStats { std::size_t hits, misses, computations, waits, evictions, invalidations, bytes; };` + - `struct ViewKey { std::string record; std::string scope; std::string query; };` (`query`: the canonical JSON of the validated `sort` and `filters` only) + - `struct ViewComputation { detail::ComputedView view; std::vector awaiting; std::vector failed; };` (what `compute` returns; the cache builds the entry from it) + - `struct ViewEntry { std::shared_ptr snapshot; std::vector> levels; detail::ComputedView view; std::vector awaiting; std::vector failed; [[nodiscard]] std::span viewRowOf() const; [[nodiscard]] std::optional sourceRowOf(RowId const& key) const; [[nodiscard]] std::size_t bytes() const; };` (`viewRowOf` and the key index are built once, at the first keyed query, under a `std::once_flag`; held only by `std::shared_ptr`, never copied or moved) + - `class ViewCache { public: explicit ViewCache(ViewCacheOptions options = {}); [[nodiscard]] std::expected, TableError> find(ViewKey const& key, std::shared_ptr const& snapshot, std::span const> levels, std::function()> const& compute); [[nodiscard]] ViewCacheStats stats() const; };` + - `AnswerContext::views` and `TableRuntime::views` (`ViewCache*`; null computes every time) + +- [ ] **Step 1: Write the failing view cache tests** in `tests/table/test_table_view_cache.cpp` + +- `"view cache: a cached view is the view"` (invariant 29): for seeds 0–199, `std::mt19937{seed}` builds a 300-row table of 4 columns (`Integer`, `Text` base; `Decimal`, `Text` at level `"recipe"` read through a `CellLevelCache` holding 70% of rows), a sort of 1–2 random keys and a random filter of 1–2 entries; through one `ViewCache` entry the test builds page 0, a keyed reply of 10 random keys and page 1, and `CHECK`s each `PageRows` equal (`rows`, `total`, `totalKind`, `undecided`, `indices`, `offset`) to `apply`'s over the same `LeveledSource`. Its second half stores into the sorted level and builds page 0 again, which must equal `apply`'s over the new snapshot. Mutation: an entry valid whatever its order levels' generations, which serves the old order. + +Add: +- `"view cache: an arrival on an order level computes again; on a display level it does not"`: one entry sorted on recipe column `r1`; a store into the level (generation changes) → the next `find` recomputes (`computations == 2`); with the sort on a base column and only display level `r2` changing → `computations == 1`, `hits == 1`. Mutation: as the case above. +- `"view cache: a request on a newer snapshot does not join an older computation"`: request A `find`s key K with snapshot S1 and its `compute` blocks on a latch; request B `find`s K with snapshot S2 (one `updateRow` later): B computes on its own (`computations == 2`, `waits == 0`) and its entry's `view` is the one over S2; after A is released, a third `find` with S2 hits B's entry. The same holds for a different order-level generation with the same snapshot. Mutation: single flight joined on the key alone, which hands B the view over S1. +- `"view cache: fifty requests compute once"` (S7/C1 shape): 50 threads `find` the same key while `compute` waits on a latch: `computations == 1`, `waits == 49`, every result the same entry. Mutation: no single flight. +- `"view cache: the bound evicts least recently used"`: `maxBytes` fitting two entries of 10,000 rows; three keys → the first evicted, `bytes <= maxBytes`. Mutation: no bound. +- `"view cache: a keyed fetch sorts nothing"` (S9 gate): after page 0, a keyed query of 20 keys: `computations` unchanged. Mutation: a keyed query bypassing the cache. + +- [ ] **Step 2: Write the failing bridge tests** in `tests/net/test_push_unloaded_rows.cpp` + +The file's rig (anonymous namespace), built as `tests/net/test_push_two_clients.cpp` builds its server, socket and client sessions: a `RemoteServer` over a 4-thread pool with `ServerConfig{.tableWork = }` serving `bench::WideTableModel` (`small`: 1,000 × 30, `recipePreloaded = 0`, recipe gate held), the model's level cache `attach`ed; one or more clients, each a `ui::Session` with a watcher and a `CountingChannel` (a `ui::ModelChannel` decorator logging `(actionId, bodyJson)` of every `execute`). Records are `"WideTableModel.small"` and `"WideTableModel.wide"` (A3). The row-scope cases use a test model `UrModel` in the same file: table `rows`, registered `table::rowScoped`, whose function returns `table::scoped(, )` with `scope` = the principal, principal `p1` seeing rows 0–499 and `p2` rows 250–749, and one `CellLevelCache` for `r` shared by both scopes; a second registration `UrEmptyScope` is the same with `scope` left empty. + +```cpp +// Mutation: announce bypassing the derived-record walk. +TEST_CASE("announce: a derived record is included, many keys are whole, an unregistered record is refused", + "[net][push][levels]") { + // A record D registered derivedFrom the table record; a client watching D. + // announce(small, 3 keys) → the client's event names `small` with the 3 keys and D whole-type. + // announce(small, 4,097 keys) → `small` whole-type. + // announce("NoSuchRecord", {}) → unexpected, and no event reaches any client within 200 ms. +} +``` + +Add: +- `"table over the bridge: the first page arrives before the level"` (row (a) bridge): with the gate held forever, `Table.small` sorted on `id`, `columns` = `id`, `d000` and `t001`: the reply arrives; `unloadedRows` names indices 0–199 (keys 1–200) with `{"recipe"}`; every recipe cell is `null`. Mutation: the answer path waiting for its loader. +- `"table over the bridge: a level loads only for what a query needs"` (row (c) bridge): a query naming no recipe column → `loaderCalls()` empty; naming one → the logged keys are exactly the page's 200; sorting on one → every candidate row (1,000). Mutation: `levelsNeeded` returning every level. +- `"table over the bridge: two clients load a value once"`: two clients ask the same page naming a recipe column; release the gate; `loadedKeys == 200` and each key once in the log. Mutation: `want` not checking the cache (800). +- `"table over the bridge: an updates effect invalidates without a call"`: after the level loaded, `EditWideRow` on the row keyed 5 (declares `updatesTable` on `wide` and `small`): the next query names that row unloaded and wants it; the loader loads key 5 once. Mutation: `fanOut` not handing the change to observers. +- `"table over the bridge: no value crosses a scope"`: `UrModel`'s `rows` for `p1` and `p2`, each naming `r`: each reply holds only its scope's rows; after release the loader was called once for each of rows 250–499. Mutation: a view shared across scopes (an empty `scope` in the key). +- `"table over the bridge: a row-scoped table announces without keys"`: an arrival for `UrModel.rows` reaches `p1`'s watch with `keys` absent; no event `p1` receives names a key outside 0–499; `UrEmptyScope`'s first query answers `SourceContract`. Mutation: the record registered without `privateKeys`. +- `"table over the bridge: a level's layout is the cache's"` (`test_push_tables.cpp`): `WideTableModel.scoped` for `"extended"` (recipe columns `d000`…`d005`, `t001`…`t006`) and for a default principal such as `"alice"` (`t001`…`t006` only), each naming every recipe column it has, with `recipePreloaded = 0` and the gate released: each reply's recipe cells equal the generator's values under their own column ids, and each key was loaded once for both. Mutation: reading the value by the caller's column order (alice's `t001` then holds `d000`'s value). Mutation: reading by the caller's column order. +- `"table over the bridge: failed keys are not loaded again inside the delay"` (L8): `small` sorted on `d000` (so `recipe` is an order level; `Page::failed` names order levels only, spec "Partial pages"), the loader returning no value for any key: once the 1,000 order keys have failed, ten page requests cause zero further loader calls, and each reply has `awaiting` empty, `failed == {"recipe"}`, and every `unloadedRows[i].failed == {"recipe"}`; the failed keys were announced once; then one request with `retry: true` calls the loader for each of the 1,000 keys once, in `ceil(1000 / 64) = 16` calls. Mutation: a failed key wanted again by every request. +- `"table over the bridge: one sort for many watchers"` (C1, C2, gate parts, `test_push_tables.cpp`): fifty clients watching one sorted view of `wide` refetch after one announcement: `ViewCacheStats::computations` rises by one; each computation ran on a `tableWork` thread (the answer path records `std::this_thread::get_id()` into a test hook). Mutation: no single flight. +- C3 gate case `"table over the bridge: fifty sorts stay inside the bound"`: fifty clients with fifty different sorts: `ViewCacheStats::bytes <= maxBytes`. + +- [ ] **Step 3: Run to verify failure** — `cmake --build --preset cl-debug --target morph_table_tests` and, on Linux, `cmake --build --preset linux-everything --target morph_net_tests`: `view_cache.hpp` not found; `announce`, `attach` not members. + +- [ ] **Step 4: Implement `view_cache.hpp`** + +`find`: under a mutex, look the key up; an entry whose `snapshot` pointer and every `levels` generation equal the arguments is a hit (its recency bumped); a stale entry is dropped (`invalidations`). On a miss, a request joins a running computation only when that computation's key, snapshot pointer and every level generation equal its own, and then waits on its `std::shared_future` (`waits`); otherwise it computes inline, outside the mutex, and publishes through its own `std::shared_future` (running computations are kept per `(key, snapshot, generations)`). The cache builds the `ViewEntry` from the `ViewComputation`, the snapshot and the generations, in a `std::shared_ptr`; when two computations of one key finish, the entry whose snapshot and generations a later request matches is the one it finds (an insert replaces the key's entry). A failed computation is not cached. Over `maxBytes` after an insert, least recently used entries go. `ViewEntry::bytes()` is `4 × (view + undecided)` plus, once built, `4 × source rows` for `viewRowOf` and 12 bytes a row for the key index (a vector of `(hash, source row)` sorted by hash). + +- [ ] **Step 5: Implement `detail::answerQuery` through the cache** (A1) + +The `ViewKey` is `{record, served.scope, canonical JSON of validated sort and filters}`; the levels passed are the `order` levels with their snapshots' `generation()`. `compute` runs `computeView` over the `LeveledSource` and fills `awaiting`/`failed`. `pageRows` takes the entry's view and, for a keyed query, its `viewRowOf()`; a keyed key is found with `sourceRowOf`. `retain` is not in the key (Task 17b merges it per request). A table registered `rowScoped` whose function returns an empty `scope` answers `SourceContract` ("a row-scoped table needs a scope"). + +- [ ] **Step 6: Implement `announce` and `attach`** in `remote.hpp` + +- `announce`: refuses (`"unregistered record: "`) when `effects::detail::resolved().recordNamed(record)` is null; otherwise builds `Changes` with `addRecordChange` and `addDerivedChanges`, applies the `kMaxChangeKeys` rule, and posts to the strand a fan-out with no caller (`fanOut(changes, 0, {})`'s path) that does not call the observers. Callable from any thread. +- `attach`: stores the observer and calls `observer->bind(sink)`, `sink` capturing `std::weak_ptr` and calling `announce` when the server is alive, nothing otherwise. +- `fanOut` (an execute's or a time fact's changes) calls each observer's `changed(changes)` first, on the strand, then queues events; an announce reaches the private fan-out step that skips observers. +- `ServerConfig::views` sizes the server's one `ViewCache`; the server puts it in the `TableRuntime` it installs around a dispatch (A2), and `QueryTable::invoke` passes `AnswerContext{.views = currentTableRuntime()->views}` (null when no runtime is installed, as under `LocalBackend`). + +- [ ] **Step 7: Run the targets** — `ctest --test-dir build/cl-debug -R "^(view cache|table)" --output-on-failure` and `ctest --test-dir build/linux-everything -R "^(table over the bridge|announce|tables)" --output-on-failure`: all pass. + +- [ ] **Step 8: Benchmarks** — S7 (ten page requests of one sorted view: requests two to ten), S9 (a keyed query of 20 keys: first and next) in `bench_table.cpp`; C1–C3 timings (strand's longest task, views computed per second) in `test_push_unloaded_rows.cpp` under `"[.benchmark][net]"`. Record into table.md. + +- [ ] **Step 9: Update `docs/spec/core/backend.md` "Fan-out"** (`ServerConfig::tableWork`, `views`; `announce`; `attach` and the observer call before events). Docs build green. + +- [ ] **Step 10: Mutations, then commit** + +```bash +git add include/morph/table include/morph/core tests docs/spec/core/backend.md +git commit -m "feat(net): announce arrivals and cache views" +``` + +--- + +### Task 7: The window under changes (table.md step 2d) + +Two commits: **7a** `PageWindow`; **7b** the session's read and watch, the query cache's open, partial and opaque entries, and the bridge's end-to-end case. + +**Spec:** "The window under changes" (all), "Partial pages" (the window's members), "The client's caches", "Which client cache a screen uses", "Arrival" steps 4–6; Tests rows "An event that precedes its page reply", "An ordinary edit on the same record", "Invalidation faster than a round trip", "A reply holds the named columns only" (window half), "A row the window never held", "An open entry hears every change", "A partial entry is not merged into", "An opaque entry", "A read with no entry", (b) window, (d) window and bridge; invariants 8, 9, 10, 30; gates K1–K4, K6, K7, E1–E4. + +**Files (7a):** +- Modify: `include/morph/table/query.hpp` (`PageFetch::{columns, revision}`; `RowsFetch`; `Refresh`; `RefreshPlan`; `detail::WindowCore`; `PageWindow`; the declaration `template <> class PageWindow>;`) +- Modify: `include/morph/table/description.hpp` (the definition of `PageWindow`) +- Test: `tests/table/test_table_server.cpp`; `tests/table/bench_table.cpp` (K1–K3) + +**Files (7b):** +- Modify: `include/morph/ui/interpreter.hpp`, `src/ui/interpreter.cpp` (`CallOptions::stop`; `Session::read`, `Session::watchRecord`, `WatchHolder`; open, partial and opaque entries) +- Modify: `include/morph/ui/document.hpp` (`ActionRecordInfo::{open, partial, opaque}`), `include/morph/ui/author.hpp` (copies them), `include/morph/core/effects.hpp` (`ActionRecord::{open, partial}`; `opaque` is Task 2's, A4), `include/morph/core/detail/result_walker.hpp` (holder members), `include/morph/table/query.hpp` (`Page::openMembers`, `Page::partialMember`) +- Modify: `tests/ui_interpreter_support.hpp` (`Harness(ui::Catalog catalog)`) +- Create: `tests/test_ui_interpreter_cache.cpp` (in `morph_tests`) +- Test: `tests/net/test_push_unloaded_rows.cpp` +- Docs: `docs/spec/core/registry.md` (`openMembers`, `partialMember`, an opaque holder), `docs/spec/ui/interpreter.md` "Queries and the cache" (open, partial, opaque entries; `Session::read`, `watchRecord`) + +**Interfaces:** +- Consumes: Task 4's `UnloadedRow`, `UnloadedLevels`, `GenericPage`, keyed replies; Task 6's `changeKeys`/`rowIdOf` (Task 5), `announce`. +- Produces (7a), in `query.hpp` unless noted: + - `struct PageFetch { PageRequest request; std::vector columns; std::uint64_t query = 0; std::uint64_t revision = 0; bool operator==(PageFetch const&) const = default; };` + - `struct RowsFetch { std::vector keys; std::vector columns; std::uint64_t query = 0; std::uint64_t sequence = 0; };` + - `enum class Refresh : std::uint8_t { Nothing, Rows, Pages };` `struct RefreshPlan { Refresh kind = Refresh::Nothing; RowsFetch rows; };` + - `PageWindow`: `explicit PageWindow(std::int64_t pageSize = 200, std::size_t maxPages = 8, RowKey key = {}, std::size_t maxKeys = 1000, std::size_t maxColumns = 96);` `std::vector visible(std::int64_t first, std::int64_t count, std::span columns = {});` `std::vector superseded() const;` `void cancelled(PageFetch const& fetch);` `RefreshPlan refresh(std::span keys);` `RefreshPlan owed();` `void invalidate();` `bool accept(PageFetch const& fetch, Page reply);` (keeps the existing `bool`: `false` when ignored) `std::vector patch(RowsFetch const& fetch, Page reply);` `UnloadedLevels unloadedLevels(std::int64_t index) const;` `std::int64_t undecided() const noexcept;` `std::span awaiting() const noexcept;` `std::span failed() const noexcept;` `std::uint64_t revision() const noexcept;` and the existing `fail`, `reset`, `query`, `row`, `total`, `totalKind`, `error`, `pagesHeld`, `pending` + - `PageWindow` (in `description.hpp`): the same members with `GenericPage` in place of `Page` and no key function parameter (`explicit PageWindow(std::int64_t pageSize = 200, std::size_t maxPages = 8, std::size_t maxKeys = 1000, std::size_t maxColumns = 96);`), and in place of `row`: `RowId const* key(std::int64_t index) const;` `Cell const* cell(std::int64_t index, std::string_view column) const;` (null when the row or the column is not held) `bool holds(std::int64_t index, std::string_view column) const;` +- Produces (7b): + - `CallOptions::stop` (`::core::async::StopToken`; a stop sends the wire's `cancel` where the server takes one) + - `async::Completion Session::read(std::shared_ptr const& channel, std::string const& actionId, Value const& body, ::core::async::StopToken stop = {});` + - `class WatchHolder` (move-only; destroying it releases its count of the record) + - `async::Completion Session::watchRecord(std::string const& record, std::function handler);` + - `effects::ActionRecord::{std::vector open; std::vector partial;}` beside Task 2's `opaque`, and `ui::ActionRecordInfo::{open, partial, opaque}` (all in `operator==`) + - `static constexpr std::array Page::openMembers() noexcept { return {"awaiting", "distinct"}; }` and `static constexpr std::string_view Page::partialMember() noexcept { return "unloadedRows"; }` (`distinct` exists once Task 8a lands; until then the walk skips a named member the holder lacks) + +#### 7a — `PageWindow` + +The rules no test alone fixes, which both specialisations share through `detail::WindowCore` (all bookkeeping that does not touch a row's content: pages, keys per row, unloaded levels per row, staleness, revisions, fetches in flight, totals, sequences): + +- **State.** `_query` (`reset` advances), `_revision` (`invalidate` advances), per held page `{keys, unloaded levels per row, stale, complete, columns held (generic) with the visible-call stamp of each}`, at most one fetch in flight per page (the window remembers it, so `accept` knows whether it was a whole-page or a column fetch), totals `{total, totalKind, undecided, awaiting, failed}` with `_totalsRevision`, `_sequence`, `_latestAsked: RowId → sequence`, `_latestKeyed` (the sequence of the latest `RowsFetch`), `_owed` (keys recorded while a page fetch is in flight). +- **`visible(first, count, columns)`** covers pages `first/pageSize − 1` to `(first + max(count,1) − 1)/pageSize + 1`, skipping pages past an exact total and pages with a fetch in flight. Not held, or held stale → a whole-page fetch `{offset p·size, limit size}` with `columns`, tagged `revision = _revision`. Held, current and short (a server cut it) → the rest of the page, with the columns held. Held, current, complete but lacking some of `columns` (generic only) → a column fetch of the same rows (`{p·size, rows held}`) naming only the missing columns, in the order given. Each returned fetch is recorded in flight. A typed window ignores `columns` and always fetches `{}`. +- **`accept`.** Ignored (`false`) when `fetch.query != _query` or the fetch is not the one in flight for its page; the in-flight record is removed either way when it matches. A column fetch: when the reply's `keys` equal the page's row for row, its cells are added (each row's unloaded levels become the union of held and reply levels) and over `maxColumns` the columns with the oldest visible-call stamp go; otherwise `invalidate()`. A whole-page fetch: the page's rows, keys and unloaded levels are replaced (a rest-of-page fetch appends as today) and `stale = fetch.revision != _revision`. With at most one fetch in flight per page and a reply accepted only as that fetch, a reply older than the held content cannot arrive: the spec's "a reply older than the held content is ignored" holds by construction, and the window keeps no content revision. A page reply with `fetch.revision >= _totalsRevision` sets `undecided`, `awaiting`, `failed` and `_totalsRevision`, and sets `total`/`totalKind` only when the reply carries a `total` (an absent total neither replaces a known one nor counts as a difference); when a present `total`, its `totalKind`, or `undecided` differ from the window's, every other held page is marked stale. Pages over `maxPages` go farthest first. +- **`superseded()`** returns the fetches in flight for pages outside the range `visible` last covered; **`cancelled`** removes a matching in-flight record. +- **`refresh(keys)`.** While any page fetch is in flight, the keys are appended to `_owed`. Then: no row held → `Nothing`; no key function (typed) or `awaiting()` not empty → `invalidate()`, `Pages`; otherwise the keys are the event's keys (deduplicated, held or not) plus, per run of consecutive held view rows (across adjacent held pages), the key of the run's first row unless already present; more than `maxKeys` → `invalidate()`, `Pages`; else `Rows` with `RowsFetch{keys, columns: the columns of the latest visible call (typed: {}), _query, ++_sequence}`, each key's `_latestAsked` set to the sequence and `_latestKeyed` to it. +- **`owed()`** builds a plan from `_owed` by the same rules as `refresh` without recording again, then clears `_owed` when no page fetch is in flight; `Nothing` when `_owed` is empty. +- **`patch(fetch, reply)`.** Ignored (empty result) when `fetch.query != _query`. A reply row whose key has `_latestAsked > fetch.sequence` is ignored, missing or not. `invalidate()` and an empty result when: a requested key the window holds (and no later fetch asked for) has no row; a reply row's key is held at another view row than `reply.indices`; a reply row's key is not held and its view row lies inside a held page; `fetch.sequence == _latestKeyed` and the reply's `total`, `totalKind` or `undecided` differ from the window's. Otherwise each held row with a reply row at its view row is replaced (its unloaded levels with the reply's); in a generic window, a column the page holds that the reply lacks is dropped from that page, so the next `visible` fetches it again rather than showing a cell older than its row. The replaced view rows are returned ascending. `patch` never adds, removes or reorders held rows, and never sets the totals. +- **`invalidate()`** advances `_revision` and marks every held page stale; rows stay shown, fetches stay in flight. + +- [ ] **Step 1: Write the failing window tests** in `tests/table/test_table_server.cpp` + +Helpers in the file: `using Win = PageWindow;` `GenericPage pageOf(std::int64_t offset, std::vector keys, std::vector columns, std::int64_t total, std::int64_t undecided = 0)` (one integer cell per column, the key's value); `GenericPage keyedOf(std::vector> keyAtIndex, columns, total, undecided = 0)`; `std::vector ids(std::initializer_list)`. + +```cpp +// Mutation: refresh not recording keys while a page fetch is in flight. +TEST_CASE("window: an event that precedes its page reply is owed", "[table][window]") { + Win window{10, 8}; + std::vector const cols{"a"}; + auto const fetches = window.visible(0, 5, cols); + REQUIRE(fetches.size() == 2); + CHECK(window.refresh(ids({3, 4})).kind == Refresh::Nothing); // nothing held yet + GenericPage reply = pageOf(0, {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, cols, 20); + reply.unloadedRows = {{3, {"recipe"}, {}}, {4, {"recipe"}, {}}}; + REQUIRE(window.accept(fetches[0], std::move(reply))); + auto const plan = window.owed(); + REQUIRE(plan.kind == Refresh::Rows); + CHECK(std::ranges::is_permutation(plan.rows.keys, ids({3, 4, 0}))); // the keys and the run's probe + CHECK(window.unloadedLevels(3).levels.size() == 1); +} + +// Mutation: refresh skipping held rows with no unloaded level. +TEST_CASE("window: an ordinary edit on a held row is a keyed fetch, a move invalidates", "[table][window]") { + Win window{10, 8}; + std::vector const cols{"a"}; + for (auto const& fetch : window.visible(0, 5, cols)) { + REQUIRE(window.accept(fetch, pageOf(fetch.request.offset, fetch.request.offset == 0 + ? std::vector{0,1,2,3,4,5,6,7,8,9} : std::vector{10,11,12,13,14,15,16,17,18,19}, + cols, 20))); + } + auto const plan = window.refresh(ids({7})); + REQUIRE(plan.kind == Refresh::Rows); + CHECK(std::ranges::is_permutation(plan.rows.keys, ids({7, 0}))); // one run, rows 0–19: one probe + auto const replaced = window.patch(plan.rows, keyedOf({{0, 0}, {7, 7}}, cols, 20)); + CHECK(replaced == std::vector{0, 7}); + auto const again = window.refresh(ids({7})); + auto const before = window.revision(); + CHECK(window.patch(again.rows, keyedOf({{0, 0}, {7, 12}}, cols, 20)).empty()); + CHECK(window.revision() == before + 1); +} +``` + +Add, each a `TEST_CASE` with its mutation: +- `"window: invalidation faster than a round trip advances one reply per trip"`: hold page 0; `invalidate`; `visible` → one fetch `f1` (revision r1); `invalidate` again (r2); `visible` → nothing (page 0 in flight); `accept(f1, rows A)`: `cell(0,"a")` shows A, the page is still stale, and `visible` returns exactly one fetch tagged r2. Mutation: `accept` ignoring every older revision. +- `"window: a column scrolled in is fetched for held pages and merged by key"` (row "A reply holds the named columns only", window half; gate K7): two complete pages with `{a}`; `visible(0, 5, {a, b})` returns two column fetches naming only `{b}`, offsets 0 and 10, limits 10; accepting replies with the same keys adds `b` (`cell(3,"b")` non-null, `cell(3,"a")` unchanged); a column reply whose `keys` differ invalidates. Mutation: a column fetch naming every column. +- `"window: over maxColumns the columns farthest from view go first"`: `maxColumns = 2`; visible `{a,b}`, then `{c}`: after the column reply, page holds `{b,c}` or `{a,c}` — assert `c` held and exactly 2 held, and the one dropped is the column least recently named (`a` when the second call was `{b, c}`). Mutation: dropping the newest. +- `"window: a page reply with another total adopts it and marks other pages stale"`: page size 10, pages 0 and 1 held with total 30; `visible(15, 10, cols)` fetches page 2; its reply says total 35: `total() == 35`, and the next `visible(15, 10, cols)` refetches pages 0 and 1 whole. Mutation: totals adopted without marking. +- `"window: a reply without a total keeps the known one"`: the existing `"table: the window keeps a known total when a reply omits it"` still passes, and a page reply with `total` absent marks no page stale. Mutation: an absent total counted as a difference. +- `"window: a patch drops the columns its reply lacks"`: page 0 holds `{a, b}`; a keyed reply naming only `{a}` for key 3 is patched: `cell(3, "a")` is the reply's, `holds(3, "b")` is false, and `visible(0, 5, {a, b})` returns a column fetch naming `{b}`. Mutation: `patch` keeping the old `b` cells. +- `"window: a row the window never held"` (four invalidating shapes and one not): held pages 0–1 of 100 rows; a keyed reply with (i) an unheld key at view row 5, (ii) the probe at view row 1, (iii) total 101 on the latest fetch, (iv) a requested held key missing — each invalidates (`revision()` +1); (v) an unheld key at view row 80 with the total unchanged → no invalidation, nothing replaced. Mutation: no probes (shape ii undetected: assert the plan's keys contain the run's first key). +- `"window: an older keyed reply never overwrites a newer one"`: fetch F1 for key 7, then F2 for key 7; patch F2's reply (value 2), then F1's reply (value 1): `cell(7,"a")` holds 2, and F1's reply with key 7 missing does not invalidate. Mutation: no per-key sequence. +- `"window: the totals of an old keyed reply are not compared"`: F1 then F2; F1's reply carries total 30 (window 20): no invalidation. Mutation: comparing totals on every keyed reply. +- `"window: an awaited order or too many keys refetches pages"`: a page reply with `awaiting {"recipe"}` → `refresh` returns `Pages` and advances the revision; `maxKeys = 3` and an event of 3 keys plus 1 probe → `Pages`. Mutation: the key count compared before adding probes. +- `"window: superseded fetches are those left behind"` (K6 at window level): `visible(0,…)` → pages 0,1; `visible(50'000, 20)` → `superseded()` returns the fetches of pages 0 and 1; after `cancelled` for each, their late replies are ignored. Mutation: `superseded` returning every fetch. +- `"window: a typed window without a key function refreshes pages"`: `PageWindow{10, 8}` holding a page: `refresh` → `Pages`. Mutation: probing without keys. +- `"window: unloaded levels agree with the page after accept"` (row (b) window): the reply's `unloadedRows {{2,{"recipe"},{"recipe"}}}` → `unloadedLevels(2).levels == {"recipe"}`, `.failed == {"recipe"}`, `unloadedLevels(1)` empty. Mutation: `accept` dropping `unloadedRows`. +- `"window: a view that empties shows no rows and asks no more"` (Review Focus 4): pages 0–1 held, total 20; a keyed reply on the latest fetch with no rows and total 0 invalidates; the whole-page reply of page 0 with no rows and total 0 is accepted: `row`/`key(0)` null, `total() == 0`, `visible(0, 5, cols)` returns nothing, `pending()` false. Mutation: an empty reply leaving the page incomplete (fetching it forever). +- Gate K2 `"window: a refresh of 900 keys over eight runs asks 908"`: 8 pages held in eight runs (pages 0,2,4,…,14 of 100,000 rows, each its own run), an event of 900 keys: `plan.rows.keys.size() == 908`. Mutation: probing every held row. + +Bridge-free typed regression: the existing `PageWindow`/`PageWindow` cases keep passing with the new constructor (defaults unchanged). + +- [ ] **Step 2: Run to verify failure** — `cmake --build --preset cl-debug --target morph_table_tests`: compile errors naming `refresh`, `RowsFetch`, `revision`, `PageWindow` incomplete. + +- [ ] **Step 3: Implement `detail::WindowCore`, `PageWindow` and `PageWindow`** per the rules above. `query.hpp` declares `template <> class PageWindow>;` before any use so that a translation unit without `description.hpp` cannot instantiate the primary template for generic rows (it fails as incomplete instead). The generic page stores cells per column (`std::map, std::less<>>` per page). + +- [ ] **Step 4: Run the target** — `ctest --test-dir build/cl-debug -R "^(window|table)" --output-on-failure`: all pass. + +- [ ] **Step 5: Benchmarks** (`"[.benchmark][window]"`): K1 (`accept` of a 200 × 300 generic page), K2 timing (the gate case's refresh), K3 (`patch` of 200 rows). Record into table.md's "Benchmarks: client cache and window". + +- [ ] **Step 6: Mutations, then commit** + +```bash +git add include/morph/table tests/table +git commit -m "feat(table): the page window under changes" +``` + +#### 7b — reading and watching without entries, open, partial and opaque entries, end to end + +- [ ] **Step 7: Write the failing cache tests** in `tests/test_ui_interpreter_cache.cpp` + +The catalog: `corpusCatalog()` wrapped so that `T_Page` (model `Lims_SampleModel`) has `records = {ActionRecordInfo{.record = "T_Row", .keyField = "code", .paths = {"rows"}, .open = {"awaiting"}, .partial = {"unloadedRows"}}}` and `T_TablePage` has `{.record = "T_Row", .opaque = true}`; `Harness h{catalog};` documents query them as `kList` in `tests/test_push_cache.cpp` does. + +```cpp +// Mutation: the cache ignoring `open`. +TEST_CASE("cache: an open entry is marked by a change to a key it does not hold", "[push][cache][table]") { + Harness h{tableCatalog()}; + auto screen = h.mount(kPageList); // one query of T_Page + h.reply(0, R"({"rows":[{"code":1}],"awaiting":["recipe"],"unloadedRows":[]})"); + h.session.applyChanges(changesOf("T_Row", {"99"})); + h.settle(); + CHECK(h.dispatcher.calls.size() == 2); // refetched + h.reply(1, R"({"rows":[{"code":1}],"awaiting":[],"unloadedRows":[]})"); + h.session.applyChanges(changesOf("T_Row", {"99"})); + h.settle(); + CHECK(h.dispatcher.calls.size() == 2); // not open any more: key 99 is not held +} +``` + +Add: +- `"cache: a partial entry is not merged into"`: reply with `unloadedRows [{"index":0,"levels":["recipe"]}]`; an `updates T_Row` reply of `{code:1, qty:5}` marks the entry (one more call) and leaves `rows[0].qty` as it was; with `unloadedRows []` the same reply merges (`rows[0].qty == 5`, no call). Mutation: the cache ignoring `partial`. +- `"cache: an opaque entry is marked by every change and never merged"`: a `T_TablePage` entry and a distinct-values-shaped entry of the same action are both marked by a keyed change naming key 99, and an `updates` reply marks without merging. Mutation: the cache looking for a key member. +- `"cache: a read makes no entry"`: `session.read(channel, "T_Page", body)` resolves with the reply; `cacheSize()` unchanged; a stop before the reply settles the completion as cancelled and passes `CallOptions::stop` to the channel. Mutation: `read` going through `acquire`. +- `"cache: a watch without an entry joins the watch set"`: `watchRecord("T_Row", handler)` sends one `watch` naming `T_Row` and resolves after its reply; `watchSet()` names `T_Row` while the holder lives and not after it is destroyed (one `unwatch`); a `resync` event calls the handler once with `keys` absent; a `changes` event for `T_Row` calls it with the event's keys. Mutation: holders not counted in the watch set. + +In `tests/net/test_push_unloaded_rows.cpp`, the bridge half of (d) with gate E3: + +```cpp +// Mutations: refresh asking for every held key; refresh dropping the keys the window does not hold. +TEST_CASE("table over the bridge: exactly the arrived rows update, without polling", "[net][push][levels]") { + // WideTableModel::configure: small with recipePreloaded = 0, recipeBatch = 20; the recipe gate held. + // No refreshEvery, no clock advanced. The CountingChannel counts page requests (no `keys`) and keyed ones apart. + // 1. watchRecord("WideTableModel.small") resolves. PageWindow w{200, 8}; w.visible(0, 30, {"id","d000"}) + // returns two fetches (pages 0 and 1: no total is known yet); each is sent with Session::read and accepted. + // pageRequests == 2, keyedRequests == 0; every row's unloadedLevels names "recipe". + // 2. Settle 200 ms: still 2 and 0. + // 3. recipeGate().release(1): one loader call of 20 keys runs (the display wants keep each want's keys in + // order, newest want first, so they are the first 20 keys of one page). Let L = model.loaderCalls()[0]. + // The cache stores, then announces L; the watch handler calls w.refresh(keys). + // 4. The plan is Rows; its keys are L plus the probe of the one held run (key 1, the first held row) — CHECK + // is_permutation(plan.rows.keys, L ∪ {1}). One keyed request is sent: pageRequests == 2, keyedRequests == 1. + // 5. w.patch returns the view rows of L, plus row 0 when key 1 was a probe, ascending; for each key of L its + // row's unloadedLevels is empty and cell(row, "d000") is the generator's value; a row not in L still names + // "recipe". +} +``` + +Add `"table over the bridge: the same arrival through the query cache"` (E4): a document query on `Table.small` (a page read by a `rows` table) holds the same page; the arrival marks that one entry and it refetches once: one page request per marked entry. + +- [ ] **Step 8: Run to verify failure** — `cmake --build --preset cl-debug --target morph_tests` (and `morph_net_tests` on `linux-everything`): compile errors naming `read`, `watchRecord`, `open`. + +- [ ] **Step 9: Implement the catalog's members** + +- The result walk records, for each visited `Object` holder with `openMembers()` or `partialMember()`, its path and those members' paths (`ActionResult` gains `std::vector holders`, `struct HolderMembers { std::string path; std::vector open; std::string partial; };`). A named member the holder's reflection lacks is skipped. +- Resolution gives each matched `ActionRecord` the `open` and `partial` paths of every holder whose path is a prefix of (or equal to the parent of) the record's path. `Table.`'s opaque slot is Task 2's (A4). `ui::author` copies `open`, `partial` and `opaque` into `ActionRecordInfo`. + +- [ ] **Step 10: Implement the cache rules** in `src/ui/interpreter.cpp` + +A helper `bool memberSet(Value const& value, std::string_view path)`: set when the value at `path` is a non-empty array, an object, `true`, or a number other than zero. `markRecordChange` marks an entry whose held record is `opaque` or has any `open` path set, whatever the keys. `applyRecordEffect` treats `updates` as `changes` for an entry whose record is `opaque` or has any `open` or `partial` path set (the discarded-reply downgrade already marks only). + +- [ ] **Step 11: Implement `read` and `watchRecord`** + +- `CallOptions::stop`; `BridgeDispatcher::Channel::execute` uses `RawHandler::execute(…, stop)` when `stop.stop_possible()`. +- `read` sends through `channel->execute(actionId, toJson(body), {.stop = stop})`, parses the reply into a `Value` on the owner, and settles; no entry is made, no effects applied; a stopped call settles cancelled and its reply is dropped. +- `watchRecord` counts the record in `_recordRefs` as an entry does (a count leaving zero sends a `watch` for it at once, through `flushWatches`), stores the handler under a holder id, and resolves when that watch's reply arrives (at once when the record was already watched and acknowledged, or there is no watcher). `applyChanges` calls each handler whose record a change names, after marking entries; `reconnected` (reconnect and `resync`) calls each with `RecordChange{record, nullopt}`. `WatchHolder`'s destructor removes the handler and counts the record out. + +- [ ] **Step 12: Run the targets** — `ctest --test-dir build/cl-debug -R "^(cache|stale_mark|own_updates|changes_reply)" --output-on-failure` and `ctest --test-dir build/linux-everything -R "^table over the bridge" --output-on-failure`: all pass, the existing push-cache cases included. + +- [ ] **Step 13: Benchmarks** — K4 (a 4,096-key event against an entry of 200 rows; storing a decoded 200 × 300 page) in `tests/bench_interpreter.cpp`; E1 (first page of sorted 100,000 × 300 `wide` with `recipe` unloaded, request to accepted page, against no level) and E2 (loader return to patched rows for 20 rows, p50/p99) in `test_push_unloaded_rows.cpp` under `"[.benchmark][net]"`. Record into table.md's "Benchmarks: client cache and window" and "Benchmarks: end to end". + +- [ ] **Step 14: Docs** — `docs/spec/core/registry.md` (a holder's `openMembers()`/`partialMember()`; `ActionRecord::open`, `partial`, `opaque`; `Table.` holds its record opaquely); `docs/spec/ui/interpreter.md` "Queries and the cache" (the three entry rules; `Session::read`; `watchRecord` and its holder). Docs build green. + +- [ ] **Step 15: Mutations, then commit** + +```bash +git add include/morph src/ui tests docs/spec +git commit -m "feat(ui): read and watch without entries; open, partial and opaque cache entries" +``` + +--- + +## Review Focus + +Up to five input classes the spec implies and no row of its Tests table exercises; each has its test in the owning task. + +1. **An invalidation between a want and its load's return, followed by another want.** A query that follows the invalidation's event while the stale load still runs must start a fresh load, not wait on one that will be dropped; otherwise the row stays unloaded until some later request. Test: Task 5, `"level cache: a want after an invalidation does not wait for the stale load"`. +2. **A loaded cell that goes back to `Unloaded`** (a server dropping what it loaded). The row moves to the unloaded tail of its sort, or becomes undecided and leaves a filtered view, with `onUndecidedChange`. Test: Task 3a, `"table: a cell dropped back to unloaded moves to the tail or leaves the view"`. +3. **Keys that are awkward JSON.** A text key with a quote, a backslash and non-ASCII, and an integer key above 2^53, through `changeKeys`/`rowIdOf` (and so through an announcement), must come back as the same `RowId`. Test: Task 5, `"table: change keys round-trip awkward keys"`. +4. **A view that empties.** Every row leaving the view (an edit, a filter's level arriving): the window must show no rows, a total of 0, and stop fetching. Test: Task 7a, `"window: a view that empties shows no rows and asks no more"`. +5. **A row entering from the undecided set under `ReorderPolicy::Deferred`.** It takes its true place at once and is not held; displayed rows keep their indices. Test: Task 3a, `"table: under deferred reorder an undecided row takes its true place at once"`. + +## Self-review notes + +- **Coverage.** Spec rows (a)–(d), the cache rows, the window rows and the client-cache rows of "Tests" each map to a named case above. Gate K6's "at most three sent" and the timing rules of "When a fetch is sent" are `WindowRows`'s (Task 9); here the window's half (`superseded`) is tested. W3, W4, S1, S2, S4 timings are recorded, not gated, per "How figures are taken". L9 is not in step 2c's gate list; "A cache's size is deep" covers its behaviour. +- **Placement follows the index's Cross-part decisions:** `Page::{unloadedRows, undecided, awaiting, failed}` and `UnloadedRow` in Task 3b (decision 3); `effects::IChangeObserver`/`AnnounceSink`, `changeKeys`, `rowIdOf` in Task 5 (decision 4); `TableQuery::columns` and `maxColumns` in Task 1 (decision 2); `PageWindow`'s `key`, `cell`, `holds`, `revision` here in Task 7 (decision 8). diff --git a/docs/superpowers/plans/2026-11-17-one-table-c-engine-config.md b/docs/superpowers/plans/2026-11-17-one-table-c-engine-config.md new file mode 100644 index 000000000..75f1386de --- /dev/null +++ b/docs/superpowers/plans/2026-11-17-one-table-c-engine-config.md @@ -0,0 +1,974 @@ +# The One Table, Part C — the Engine's Configuring Additions: Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Give `morph::table` value-set filters and distinct values, column groups with quantifiers and `together`, values with units over a run-time unit registry, and a column layout with widths, pinning and one fitting rule (table.md Delivery order step 3). + +**Architecture:** Everything lands in the engine headers (`include/morph/table/{data_source,sort,filter,query,engine}.hpp`) plus one UI parser (`include/morph/ui/measure.hpp`). Filters stay compile-once / evaluate-per-cell: a `oneOf` becomes one hashed set, a group becomes per-cell entry bit masks found by one value-index lookup, `Together` is a bipartite matching by augmenting paths, and every evaluation is three-valued (`Match`) over unloaded cells. A `Measure` key is a pair (dimension place, exact base amount) built from tables indexed by `UnitId`, so key building never looks a unit up by name. + +**Tech Stack:** C++23, header-only morph, glaze, Catch2 (`morph_table_tests`), MSVC preset `cl-debug` (timings on `cl-release`). + +**Spec:** `docs/spec/table/table.md` — "Filters from the header" (value-set operator, distinct values, date buckets), "Column groups", "Values with units" (all subsections), "Columns: order, visibility, pinning, width" (the engine half), "Semantics of an unloaded cell" ("Unloaded cells in filters", "Unique values"), Invariants 1, 22–25, Tests rows "`oneOf`", "Distinct values", "Column groups", "Values with units", "Unit numbers across registries", "Column layout" (engine half), "The generic page" (measure half), Benchmarks D1–D6, G1–G7, M1–M5, Neighbouring specs row `../forms/views.md` (step 3), Delivery order step 3. Roadmap: `docs/superpowers/plans/2026-11-17-one-table.md`, Task 8. +Part C of 7; index: 2026-11-17-one-table.md + +## Global Constraints + +Every line of the roadmap's Global Constraints applies. The ones this part touches, verbatim: + +- AGENTS.md "Comments and documentation": a comment or spec states what the code does now and why; no history, no issue numbers, no commit hashes. +- Every public symbol has complete Doxygen `@param`/`@tparam`/`@return`; the docs build runs `WARN_AS_ERROR = FAIL_ON_WARNINGS` (`cmake -S . -B build/doc -G Ninja -DMORPH_BUILD_DOCUMENTATION=ON -DMORPH_BUILD_TESTS=OFF -DMORPH_BUILD_EXAMPLES=OFF && cmake --build build/doc --target doc`). +- morph is public: no company, product or customer names in code, tests, specs or commit messages; examples use the neutral `lims` domain. +- Commits are authored `yaraslau ` and end with the session's attribution lines. +- AGENTS.md "Verify rather than assert": every test names the mutation it was seen to fail on, and the task runs that mutation once and records the failing output in its commit message body. +- Limits, verbatim: `QueryLimits::maxSetValues = 10'000`, `maxQuerySetValues = 20'000`, `maxQuerySetBytes = 1 MiB` (1'048'576 bytes), `maxGroupColumns = 16`; `DistinctRequest::limit = 200`. +- Timings are never CI gates. Count gates (D3, D5, D6, G5, G7, M5) are ordinary tests; timings (D1–D4, G1–G4, G6, G7, M1–M4) are `[.benchmark]` cases in `tests/table/bench_table.cpp`, run on `cl-release`, and written into table.md's "Benchmarks: configuring", "Benchmarks: column groups" and "Benchmarks: values with units" tables in the same commit. +- No GUI application is launched. + +Commands (the roadmap's standard loop): + +```bash +cmake --preset cl-debug +cmake --build --preset cl-debug --target morph_table_tests +ctest --test-dir build/cl-debug -R "" --output-on-failure +# timings only: +cmake --preset cl-release && cmake --build --preset cl-release --target morph_table_tests +build/cl-release/tests/morph_table_tests.exe "[.benchmark][]" +``` + +## Review Focus + +1. **An empty or repeating value set.** `oneOf: []` (a menu with nothing ticked) matches no row in `include` and rejects no row in `exclude`; `["alpha", "ALPHA"]` on a folded text column is one member, matches the same rows as `["alpha"]`, and counts as two toward `maxSetValues`. Test goes to Task 8a (`"table: an empty or repeating oneOf"`). +2. **A values-list search that is not text.** On a decimal column a search `"1.5"` finds `11.5` and `1.5` (substring of the value's text), an accented search `"É"` finds `"école"` and `"Ecole"` under the collator's fold, and an `offset` past the end returns no values while `distinct` still counts them. Test goes to Task 8a (`"distinct: search on numbers and accents, and an offset past the end"`). +3. **A group naming a column twice, `exactly 0`, and `together` over too many entries.** `columns: ["a", "a"]` is `InvalidSpec` (a matching that may use a column twice is not `Together`); `exactly 0` passes exactly the rows where no column satisfies; `together` with 4 entries over 3 columns is `InvalidSpec`. Test goes to Task 8b (`"groups: duplicate columns, exactly zero and too many together entries"`). +4. **Negative and zero amounts with units.** `-300 g` sorts before `-0.2 kg`; `0 g`, `0 kg` and a plain `0` in a mass-only column: the first two tie and are one distinct value, the third is `InvalidValue` as an operand; `-0.5 %` sorts in the ratio group after every mass in both directions. Test goes to Task 8c (`"measure: negative and zero amounts"`). +5. **A stored layout that repeats itself.** An `order` naming an id twice keeps its first place; a width of `0` or `-5` is dropped; `pinned: 40` over three visible columns is 3; a group listing a column twice keeps it once. Test goes to Task 8d (`"column layout: a stored layout that repeats itself"`). + +--- + +## Decisions + +- **`x-units` is not emitted.** table.md 2534-2536 has a typed row's schema list the column's units as `x-units`. That schema is `forms::schemaJson()`, built from the type alone, while the units are the application's run-time registry (2500-2502), and 2505 says a client learns units "from the table's description … and from nowhere else". 8c therefore marks a `MeasureValue` property `x-measure` only, and the units reach a client in `TableDescription::units` (filled by `DescribeTable`). The spec's `x-units` sentence should be changed to say that. Reading a document row's `{num, den, unit}` in `ValueRowsSource` is Task 11a's, and dropping a stored display unit that names a removed unit is Task 12c's (the index assigns both). + +## Order inside Part C + +As the index's cross-part decision 17 fixes: 8a needs Task 3; **8b needs 8a** (its pooled values list extends `distinctValues`, and its `Together` value index reuses 8a's `ValueSet`); **8c needs Tasks 1, 2 and 5** (the generic page, the describe reply, the level cache); 8d is independent. 8a, 8b and 8c all touch `filter.hpp` and `query.hpp`: they merge one at a time, in the order 8a, 8b, 8c, each rebased on the last. + +## Interfaces consumed from other parts + +These are the names this part relies on. Where the roadmap does not pin a detail, the assumption is stated; the implementer checks it against the landed code first and adapts only the call, never the name another task produced. + +- **Task 3 (Part B):** `struct Unloaded {}` (a `Cell` alternative after `QuantityCell`); `CellState::Unloaded`; `enum class Match : std::uint8_t { No, Yes, Unknown }`; `static Match CompiledFilter::evaluate(Column const&, KeyColumn const&, std::uint32_t)`; `passes == (evaluate == Yes)`; `static Match CompiledFilter::evaluateGroup(Group const&, std::span keys, std::uint32_t row)` (keys by column index; the plain-group rule, which 8b replaces with quantifiers); `Match CompiledFilter::evaluateRow(std::span, std::uint32_t) const`; `FilterRun::take() &&` and `FilterRun::takeUndecided() &&`, whose group stage calls `evaluateGroup`; `PageRows::{TotalKind totalKind; std::int64_t undecided; …}`; `Engine::undecidedRowCount()`; `TableErrorCode::SourceContract` (appended last); `ColumnInfo::level` (a member after `format`). +- **Task 1 (Part A):** `TableDescription{…, groups, levels, units, …}`, `ColumnGroupInfo{id, label, columns}` with `operator== = default`, `GenericPage`, `writeGenericPage(PageRows const&, DataSource const&, std::span)`, `readGenericPage(std::string_view, TableDescription const&)`. Task 1 leaves `TableDescription::units` out; **8c adds it after `levels`**. `UnitInfo` does not exist before 8c. +- **Task 2 (Part A):** `ServedTable{description, source, levels, mayRead, scope, services, limits}`; `DescribeTable::invoke` returning `describe(served.description)`; `table::detail::answerQuery(ServedTable const&, TableQuery const&, AnswerContext const& = {})` (index decision 5). +- **Task 4 (Part B):** `TableQuery::keys` (`std::optional>`, after `columns`); `apply` = `validate` -> `detail::computeView` -> `detail::pageRows`. +- **Task 5 (Part B):** the spec's block verbatim: `CellLevelCache`, `LevelCacheOptions` (no `units` member; 8c adds it), `LevelCacheStats::{loaderCalls, invalidatedKeys}`, `LevelSource::snapshot()`, `LevelSnapshot::read`, `invalidateAll()`; `cellBytes(Cell const&)` in `data_source.hpp` (`sizeof(Cell)` for a `Measure`, which owns no heap). +- **Task 7 (Part B):** `Page::openMembers()` already returns `{"awaiting", "distinct"}`; 8a only adds the `distinct` member it names. +- **Task 0 (Part A):** `bench::WideTableSource`, `bench::WideTableOptions{rows, columns, seed, loadDelay}` in `tests/table/wide_table_model.{hpp,cpp}`. `bench::WideColumn{info, role, sparse, level, group}` from `WideTableSource::layout()`: `m000`...`m009` have role `Measure` and `ColumnKind::Decimal` until 8c; `t004`...`t006` carry `group = "g3"`, `t011`...`t026` `group = "g16"`. + +--- + +### Task 8a: `oneOf` and distinct values + +**Spec:** "Filters from the header" (the value-set operator; "Distinct values" with every bullet); "Unique values"; invariants 22, 23; Tests rows "`oneOf`", "Distinct values"; gates D3, D5, D6; timings D1–D4. + +**Files:** +- Modify: `include/morph/table/filter.hpp` (`FilterEntry::oneOf`, `CompiledFilter::Op::OneOf`, `ValueSet`, `FilterCounters`, `detail::readEntry`, `compileEntry`, `CompiledFilter::matches`) +- Modify: `include/morph/table/query.hpp` (`QueryLimits::{maxSetValues, maxQuerySetValues, maxQuerySetBytes}`, `DateBucket`, `DistinctRequest`, `DistinctValue`, `DistinctValues`, `distinctValues`, `TableQuery::distinct`, `PageRows::distinct`, `Page::distinct`, `validate`, `apply`, `detail::entryTooLong`) +- Modify: `include/morph/table/engine.hpp` (`Engine::distinctValues`) +- Modify: `include/morph/table/description.hpp` (`GenericPage::distinct`, its writer and reader) +- Create: `tests/table/test_table_distinct.cpp`; modify `tests/table/test_table_filter.cpp`, `tests/table/test_table_query.cpp`, `tests/table/bench_table.cpp`, `tests/CMakeLists.txt` (add `table/test_table_distinct.cpp` to `morph_table_tests`) + +**Interfaces:** +- Consumes: Task 3's `Match`, `CellState::Unloaded`, `CompiledFilter::evaluate`, `PageRows::undecided`; Task 1's `GenericPage`. +- Produces: + ```cpp + // filter.hpp + struct FilterEntry { /* existing members, then: */ std::optional> oneOf; }; + enum class CompiledFilter::Op : std::uint8_t { Eq, …, NotEmpty, OneOf }; // OneOf appended last + struct FilterCounters { std::uint64_t setLookups = 0; std::uint64_t cellReads = 0; std::uint64_t augmentations = 0; }; + class ValueSet; // the members of one oneOf, parsed for one column; one lookup per cell + // CompiledFilter::Entry gains: std::shared_ptr set; (set for OneOf only) + static Match CompiledFilter::evaluate(Column const&, KeyColumn const&, std::uint32_t row, FilterCounters* counters); + // query.hpp + enum class DateBucket : std::uint8_t { Value, Year, Month, Day }; // JSON "value", "year", "month", "day" + struct DistinctRequest { std::string column; std::string group; std::string search; + DateBucket bucket = DateBucket::Value; std::int64_t offset = 0; std::int64_t limit = 200; }; + struct DistinctValue { std::string text; std::string operand; std::int64_t count = 0; }; + struct DistinctValues { std::vector values; std::int64_t distinct = 0; + TotalKind distinctKind = TotalKind::Exact; std::int64_t blanks = 0; + std::int64_t invalid = 0; bool partial = false; }; + [[nodiscard]] std::expected + distinctValues(DataSource const& source, TableQuery const& query, DistinctRequest const& request, + ApplyOptions const& options = {}); + // TableQuery::distinct, PageRows::distinct, Page::distinct, GenericPage::distinct: + std::optional distinct; // on TableQuery + std::optional distinct; // on PageRows, Page, GenericPage + // engine.hpp + [[nodiscard]] std::expected Engine::distinctValues(DistinctRequest const& request) const; + ``` + +- [ ] **Step 1: Write the failing `oneOf` tests** in `tests/table/test_table_filter.cpp`. Add helpers `oneOf(std::vector)` beside `eq(…)`, and, to the header comment's mutation list, "`ValueSet` matching nothing: `oneOf matches what its eq entries match` fails". + +```cpp +FilterEntry oneOf(std::vector v) { return FilterEntry{.oneOf = std::move(v)}; } + +TEST_CASE("table: oneOf matches what its eq entries match", "[table][filter]") { + Filtered const text{.source = tabletest::column(ColumnKind::Text, + {std::string{"Alpha"}, std::string{"beta"}, std::string{"Gamma"}, Cell{}, + std::string{"alpha"}}), + .services = {}}; + CHECK(text.by(on("c", {oneOf({"alpha", "BETA"})})) == Rows{0, 1, 4}); + CHECK(text.by(on("c", {oneOf({"alpha", "BETA"})})) == text.by(on("c", {eq("alpha"), eq("BETA")}))); + CHECK(text.by(on("c", {}, Combine::Any, {oneOf({"alpha", "BETA"})})) == Rows{2, 3}); + CHECK(text.by(on("c", {oneOf({"alpha", "BETA"}), isEmpty()})) == Rows{0, 1, 3, 4}); + FilterEntry exact = oneOf({"alpha"}); + exact.caseSensitive = true; + CHECK(text.by(on("c", {exact})) == Rows{4}); + + Filtered const ints{.source = tabletest::column(ColumnKind::Integer, + {std::int64_t{1}, std::int64_t{2}, std::int64_t{3}, std::string{"x"}}), + .services = {}}; + CHECK(ints.by(on("c", {oneOf({"2", "1.5"})})) == Rows{1}); + CHECK(ints.by(on("c", {oneOf({"2", "zz"})})).error().code == TableErrorCode::InvalidValue); +} + +TEST_CASE("table: oneOf is unknown on an unloaded cell", "[table][filter]") { + auto const source = tabletest::column(ColumnKind::Text, {std::string{"Alpha"}, Cell{Unloaded{}}, std::string{"b"}}); + auto const page = apply(*source, TableQuery{.sort = {}, .filters = on("c", {oneOf({"alpha"})}), .page = {}}); + REQUIRE(page); + CHECK(page->rows == std::vector{0}); + CHECK(page->undecided == 1); + CHECK(page->totalKind == TotalKind::AtLeast); +} + +TEST_CASE("table: an empty or repeating oneOf", "[table][filter]") { // Review Focus 1 + Filtered const text{.source = tabletest::column(ColumnKind::Text, + {std::string{"Alpha"}, std::string{"beta"}, Cell{}}), .services = {}}; + CHECK(text.by(on("c", {oneOf({})})) == Rows{}); + CHECK(text.by(on("c", {}, Combine::Any, {oneOf({})})) == Rows{0, 1, 2}); + CHECK(text.by(on("c", {oneOf({"alpha", "ALPHA"})})) == text.by(on("c", {oneOf({"alpha"})}))); +} + +TEST_CASE("table: a oneOf of any size is one lookup a cell", "[table][filter]") { // gate D5 + // 100,000 rows of "v"; a oneOf of the 10,000 texts "v0".."v9999". + // For every row r: CompiledFilter::evaluate(compiled.columns()[0], keys, r, &counters). + CHECK(counters.setLookups == 100'000); + CHECK(passed == 50'000); +} + +TEST_CASE("table: oneOf is the OR of its eq entries over random tables", "[table][filter]") { // invariant 23 + // std::mt19937 seed 23; 200 tables of 50 rows over kinds Text, Integer, Decimal, Date, Bool; + // cells drawn from 8 values plus empty and one invalid text; a oneOf of 1..6 drawn values. + CHECK(f.by(on("c", {oneOf(values)})) == f.by(on("c", eqEntries(values)))); // include, combine any + CHECK(f.by(on("c", {}, Combine::Any, {oneOf(values)})) == + f.by(on("c", {}, Combine::Any, eqEntries(values)))); // exclude +} +``` + +(`Filtered`, `on`, `eq`, `isEmpty` are the file's existing helpers; the D5 case builds key columns as `Filtered::by` does and keeps the pointers to call `evaluate` itself.) + +- [ ] **Step 2: Write the failing limit tests** in `tests/table/test_table_query.cpp`: + +```cpp +TEST_CASE("table query: value sets are bounded per entry and per query", "[table][query]") { + // texts(n, bytes): n distinct strings of exactly `bytes` bytes. + auto const columns = std::vector{tabletest::col("a", ColumnKind::Text), tabletest::col("b", ColumnKind::Text), + tabletest::col("c", ColumnKind::Text)}; + auto query = [](std::vector>> sets) { /* one oneOf include per column */ }; + CHECK(validate(query({{"a", texts(10'000, 16)}}), columns)); + CHECK(validate(query({{"a", texts(10'001, 16)}}), columns).error().code == TableErrorCode::LimitExceeded); + CHECK(validate(query({{"a", texts(10'001, 16)}}), columns).error().column == "a"); + CHECK(validate(query({{"a", texts(10'000, 16)}, {"b", texts(10'000, 16)}}), columns)); + CHECK(validate(query({{"a", texts(10'000, 16)}, {"b", texts(10'000, 16)}, {"c", texts(1, 16)}}), columns) + .error().code == TableErrorCode::LimitExceeded); + CHECK(validate(query({{"a", texts(9'000, 120)}}), columns).error().code == TableErrorCode::LimitExceeded); // 1,080,000 B + CHECK(validate(query({{"a", texts(1, 257)}}), columns).error().code == TableErrorCode::LimitExceeded); // maxTextLength +} + +TEST_CASE("table query: a 10,000-value set fits the wire bound", "[table][query]") { // gate D6 + auto const q = /* one oneOf of 10,000 distinct 16-byte texts on column "a" */; + std::string json; + REQUIRE_FALSE(glz::write_json(q, json)); + CHECK(json.size() <= 250'000); + CHECK(json.size() < QueryLimits{}.maxQuerySetBytes); + CHECK(validate(q, columns)); +} +``` + +- [ ] **Step 3: Write the failing distinct-value tests** in `tests/table/test_table_distinct.cpp`. Header comment lists the mutations: "counting over the view (the column's own filter kept): `counts are those of the rows the other filters leave` fails"; "grouping text by raw text instead of its fold: `values are grouped by the key` fails". The shared source: + +```cpp +// status (Text), city (Text), qty (Integer); keys 0..5 +// 0: "open", "Berlin", 1 +// 1: "Open", "Paris", 2 +// 2: "closed", "Berlin", 3 +// 3: empty, "Berlin", 4 +// 4: "open", "Paris", "x" (invalid integer) +// 5: "hold", "Rome", 5 + +TEST_CASE("distinct: counts are those of the rows the other filters leave", "[table][distinct]") { + TableQuery q; + q.filters.columns["city"] = ColumnFilter{.combine = Combine::Any, .include = {oneOf({"Berlin", "Paris"})}, .exclude = {}}; + q.filters.columns["status"] = ColumnFilter{.combine = Combine::Any, .include = {eq("closed")}, .exclude = {}}; + auto const d = distinctValues(*source, q, DistinctRequest{.column = "status"}); + REQUIRE(d); + REQUIRE(d->values.size() == 2); + CHECK(d->values[0].text == "closed"); CHECK(d->values[0].operand == "closed"); CHECK(d->values[0].count == 1); + CHECK(d->values[1].text == "open"); CHECK(d->values[1].operand == "open"); CHECK(d->values[1].count == 3); + CHECK(d->distinct == 2); CHECK(d->distinctKind == TotalKind::Exact); + CHECK(d->blanks == 1); CHECK(d->invalid == 0); CHECK_FALSE(d->partial); + // invariant 22: the sum is the row count of the other filters + CHECK(1 + 3 + d->blanks + d->invalid == 5); +} + +TEST_CASE("distinct: invalid cells are counted and listed under no value", "[table][distinct]") { + TableQuery q; + q.filters.columns["city"] = ColumnFilter{.combine = Combine::Any, .include = {eq("Paris")}, .exclude = {}}; + auto const d = distinctValues(*source, q, DistinctRequest{.column = "qty"}); + REQUIRE(d); + REQUIRE(d->values.size() == 1); + CHECK(d->values[0].text == "2"); CHECK(d->values[0].operand == "2"); CHECK(d->values[0].count == 1); + CHECK(d->invalid == 1); +} + +TEST_CASE("distinct: an unloaded cell or an undecided row sets partial", "[table][distinct]") { + // the same source with row 0's status Unloaded{}, then (second source) row 1's city Unloaded{} + CHECK(withUnloadedStatus->partial); CHECK(withUnloadedStatus->values[1].count == 2); // "open": rows 1, 4 + CHECK(withUndecidedRow->partial); CHECK(withUndecidedRow->values[1].count == 2); // row 1 undecided by city +} + +TEST_CASE("distinct: search, paging and the cap", "[table][distinct]") { + CHECK(names(distinctValues(*source, {}, {.column = "status", .search = "OP"})) == std::vector{"open"}); + auto const page = distinctValues(*source, {}, {.column = "status", .offset = 1, .limit = 1}); + CHECK(names(page) == std::vector{"hold"}); // ascending: closed, hold, open + CHECK(page->distinct == 3); + ApplyOptions capped; + capped.limits.totalCap = 2; + auto const atCap = distinctValues(*source, {}, {.column = "status"}, capped); + CHECK(atCap->distinct == 2); CHECK(atCap->distinctKind == TotalKind::AtLeast); CHECK(atCap->values.size() == 3); + ApplyOptions small; + small.limits.maxLimit = 2; + CHECK(distinctValues(*source, {}, {.column = "status", .limit = 500}, small)->values.size() == 2); +} + +TEST_CASE("distinct: search on numbers and accents, and an offset past the end", "[table][distinct]") { // Review Focus 2 + // decimal column [11.5, 1.5, 2]; text column ["école", "Ecole", "x"] with tabletest's accent-folding collator + CHECK(names(distinctValues(*decimals, {}, {.column = "c", .search = "1.5"})) == std::vector{"1.5", "11.5"}); + CHECK(names(distinctValues(*texts, {}, {.column = "c", .search = "É"}, foldingOptions)).size() == 1); // one folded value + auto const past = distinctValues(*decimals, {}, {.column = "c", .offset = 10}); + CHECK(past->values.empty()); CHECK(past->distinct == 3); +} + +TEST_CASE("distinct: dates by year, month and day", "[table][distinct]") { + // Date column (days) for 2025-12-31, 2026-01-15, 2026-01-20, 2026-02-01 + auto const years = distinctValues(*dates, {}, {.column = "d", .bucket = DateBucket::Year}); + CHECK(years->values[0].text == "2025"); CHECK(years->values[0].operand == "2025-01-01..2025-12-31"); CHECK(years->values[0].count == 1); + CHECK(years->values[1].text == "2026"); CHECK(years->values[1].operand == "2026-01-01..2026-12-31"); CHECK(years->values[1].count == 3); + auto const months = distinctValues(*dates, {}, {.column = "d", .bucket = DateBucket::Month}); + CHECK(names(months) == std::vector{"2025-12", "2026-01", "2026-02"}); + CHECK(months->values[1].operand == "2026-01-01..2026-01-31"); CHECK(months->values[1].count == 2); + auto const days = distinctValues(*dates, {}, {.column = "d", .bucket = DateBucket::Day}); + CHECK(days->values[1].text == "2026-01-15"); CHECK(days->values[1].operand == "2026-01-15..2026-01-15"); + // A DateTime column: Day bucket of 2026-01-15T10:00:00Z has operand "2026-01-15T00:00:00Z..2026-01-15T23:59:59Z". + CHECK(distinctValues(*source, {}, {.column = "status", .bucket = DateBucket::Year}).error().code == TableErrorCode::InvalidSpec); +} + +TEST_CASE("distinct: requests are validated as queries are", "[table][distinct]") { + CHECK(distinctValues(*source, {}, {.column = "status", .group = "g"}).error().code == TableErrorCode::InvalidSpec); + CHECK(distinctValues(*source, {}, {}).error().code == TableErrorCode::InvalidSpec); + CHECK(distinctValues(*source, {}, {.column = "nope"}).error().code == TableErrorCode::UnknownColumn); + ApplyOptions hidden; + hidden.mayRead = [](std::string_view id) { return id != "status"; }; + CHECK(distinctValues(*source, {}, {.column = "status"}, hidden).error().code == TableErrorCode::ColumnNotReadable); + CHECK(distinctValues(*source, {}, {.column = "status", .search = std::string(257, 'a')}).error().code == TableErrorCode::LimitExceeded); + CHECK(distinctValues(*source, {}, {.column = "status", .offset = 10'000'001}).error().code == TableErrorCode::LimitExceeded); + CHECK(distinctValues(*source, {}, {.column = "status", .offset = -1}).error().code == TableErrorCode::LimitExceeded); + TableQuery keyed; + keyed.keys = std::vector{std::int64_t{1}}; + keyed.distinct = DistinctRequest{.column = "status"}; + CHECK(validate(keyed, source->columns()).error().code == TableErrorCode::InvalidSpec); +} + +TEST_CASE("distinct: the column's level is an order level of the request", "[table][distinct]") { + // the shared source with status declared at level "slow" (tabletest::col then .level = "slow") + TableQuery q; + q.distinct = DistinctRequest{.column = "status"}; + q.columns = std::vector{"city"}; + auto const needed = levelsNeeded(q, source->columns(), {}); + CHECK(needed.order == std::vector{"slow"}); // Mutation: levelsNeeded ignoring query.distinct +} + +TEST_CASE("distinct: server mode answers through apply, client mode through the engine", "[table][distinct]") { + TableQuery q; // the city filter of the first case + q.distinct = DistinctRequest{.column = "status"}; + auto const page = apply(*source, q); + REQUIRE(page); + CHECK(page->rows.empty()); + REQUIRE(page->distinct); + CHECK(*page->distinct == *distinctValues(*source, q, *q.distinct)); // DistinctValues and DistinctValue are ==-comparable + Engine engine{source}; + REQUIRE(engine.setFilter(q.filters)); + CHECK(*engine.distinctValues(*q.distinct) == *distinctValues(*source, q, *q.distinct)); +} + +TEST_CASE("distinct: invariant 22 over random tables", "[table][distinct]") { + // std::mt19937 seed 22; 100 tables of 300 rows, columns c0 (Text), c1 (Integer), c2 (Text); + // random filters on all three; for column c0 with partial == false: + CHECK(sumOfCounts + d->blanks + d->invalid == *apply(*source, queryWithoutC0Filter)->total); +} +``` + +Also in `test_table_description.cpp`: `"generic page: a distinct reply round-trips"` — a `GenericPage` with no rows and `distinct` holding the first case's values reads back equal. + +- [ ] **Step 4: Run the new cases; they fail to compile** (`oneOf`, `distinctValues`, `DistinctRequest` undeclared). + +Run: `cmake --build --preset cl-debug --target morph_table_tests` +Expected: compile errors naming `oneOf` and `distinctValues`. + +- [ ] **Step 5: Implement the value-set operator** in `filter.hpp`: + - `FilterEntry::oneOf` per the Interfaces block, Doxygen as the spec's block ("Matches a cell whose value equals any of these. Each is parsed for the column's kind, as `eq`'s operand is."). + - `readEntry` counts `oneOf` as one operator (`Op::OneOf`); `operatorDefined(kind, OneOf)` is `operatorDefined(kind, Eq)`. + - `class ValueSet` holds the parsed members for one column kind: `std::unordered_set` (Date, DateTime, Bool), `std::unordered_set` (Integer, Key, Decimal, Quantity; hash of the canonical `numerator`/`denominator`), `std::unordered_set` (Number), and a transparent-hash `std::unordered_set` of folded texts, or raw texts when `caseSensitive` (Text). Public: `[[nodiscard]] bool contains(KeyColumn const& keys, std::uint32_t row, FilterCounters* counters) const` (one probe; increments `counters->setLookups` when non-null) and `[[nodiscard]] std::size_t size() const noexcept`. Members are parsed by the existing `parseOperand`; one that does not parse is `InvalidValue` naming the column, as `eq` is. An Integer/Key column drops a member that is not a whole number at compile time (it can never match, as `eq "1.5"` cannot). + - `matches` for `OneOf`: `Unloaded` → handled by Task 3's `evaluate` (`Unknown`); `Empty`/`Invalid` → `false`; `Valid` → `entry.set->contains(…)`. + - The existing `evaluate(Column const&, KeyColumn const&, std::uint32_t)` gains a trailing `FilterCounters* counters = nullptr`, passed down to `matches`. + +- [ ] **Step 6: Implement the limits** in `query.hpp`: `QueryLimits::maxSetValues = 10'000`, `maxQuerySetValues = 20'000`, `maxQuerySetBytes = std::size_t{1} << 20U`. `detail::entryTooLong` checks every `oneOf` member against `maxTextLength`. `detail::checkFilters` sums, over every `oneOf` of `include` and `exclude` of every column and group, the member count and the members' byte sizes; one entry over `maxSetValues` is `LimitExceeded` naming its column (a group's: empty), the sums over `maxQuerySetValues` or `maxQuerySetBytes` are `LimitExceeded` with an empty column. + +- [ ] **Step 7: Implement distinct values** in `query.hpp` (`DateBucket` and its `glz::meta` "value", "year", "month", "day"; the three structs with `operator== = default`; `TableQuery::distinct`, `PageRows::distinct`, `Page::distinct`). `distinctValues` follows the spec's bullets; the choices they leave open: + - **Validation** (also run by `validate` on `query.distinct`): exactly one of `column`, `group` is non-empty, else `InvalidSpec`; `group` non-empty is `Unavailable` ("pooled values need column groups") until Task 8b; the column passes `detail::checkColumn`; `search.size() > maxTextLength`, `offset < 0`, `offset > maxOffset`, `limit < 0` are `LimitExceeded`; `limit` is clamped to `maxLimit`; a `bucket` other than `Value` on a kind other than `Date`/`DateTime` is `InvalidSpec`; a `Custom` column is `UnsupportedOperator` (it has no values to tick); `query.keys` and `query.distinct` both set is `InvalidSpec`. + - **Rows:** compile `query.filters` with `filters.columns[request.column]` erased; run it as `apply` does (a `FilterRun` over every source row). `Yes` rows are candidates; any `Unknown` row sets `partial`. + - **Grouping key:** the column's `KeyColumn` (built for the column alone): Text by `folded`, the exact/int/real key otherwise; a bucket groups `Date` days or `DateTime` seconds (UTC) by the bucket's start. `Empty` → `blanks`, `Invalid` → `invalid`, `Unloaded` → `partial`, not counted. + - **A value's `text` and `operand`:** from its first candidate row in source order. Text: the raw text (both). Integer/Key/Decimal/Quantity: `units::detail::formatRationalDecimal` of the key (both). Number: `std::format("{}", key)` (both). Bool: `"true"`/`"false"`. Date `Value`: the cell's own string when the source gave text, else ISO `YYYY-MM-DD` (both). DateTime `Value`: the cell's own string, else `YYYY-MM-DDTHH:MM:SSZ`. Buckets: `text` is `YYYY`, `YYYY-MM` or `YYYY-MM-DD`; `operand` is the bucket's first and last instant joined by `..` (`2026-01-01..2026-01-31`; DateTime ends `T00:00:00Z` and `T23:59:59Z`), which the menu splits into a `between`. + - **Search:** `collator.fold(search)` is a substring of the value's folded `text`, folded once per distinct value, never per row. + - **Order and paging:** values sort by `compareKeys` (ascending) on their first rows, ties by first row; `distinct` is the count after search, reported as `min(count, totalCap)` with `AtLeast` when `count > totalCap`; then `offset`/`limit` slice `values`. + - `levelsNeeded` (Task 3) adds, when `query.distinct` names a `column`, that column's level to `order`: the values list reads it for every candidate row. + - `apply` with `query.distinct` set returns `PageRows{.rows = {}, .distinct = distinctValues(…)}` (same options). The typed `apply` copies `distinct` into `Page`. `detail::answerQuery` (Task 2) takes the same branch before `computeView`, so a `Table.` request with `distinct` answers a `GenericPage` with no rows and `distinct` set. `Page::openMembers()` already names `"distinct"` (Task 7). + - `GenericPage::distinct` is written and read as the JSON member `"distinct"` with glaze's default shape of `DistinctValues`. + +- [ ] **Step 8: Implement `Engine::distinctValues(request) const`** in `engine.hpp`: validation as above with default `QueryLimits`; the engine's current `filter()` spec with the column's entry erased; the engine's current snapshot; key columns the engine holds for that snapshot reused, a missing one built on the calling thread and not stored. + +- [ ] **Step 9: Run the new cases and the target** + +Run: `ctest --test-dir build/cl-debug -R "oneOf|distinct|value sets|wire bound" --output-on-failure`, then `ctest --test-dir build/cl-debug -R "table" --output-on-failure` +Expected: all pass. + +- [ ] **Step 10: Mutations.** (a) `ValueSet::contains` returning `false`: `"table: oneOf matches what its eq entries match"` fails. (b) `distinctValues` keeping the column's own filter: `"distinct: counts are those of the rows the other filters leave"` fails (`closed` missing). (c) grouping text by `raw` instead of `folded`: the same case fails (`open` count 2, an `Open` value appears). Paste each failing line into the commit body; revert. + +- [ ] **Step 11: Timings D1–D4.** Add `TEST_CASE("table benchmark: distinct values and value sets", "[.benchmark][table][distinct]")` to `bench_table.cpp`, generated with `std::mt19937{7}`: 100,000 rows, a text column of 50,000 distinct values; D1 first 200 with keys built; D2 with search `"abc"`; D3 with filters on two other columns, one of them `Unloaded` in 30% of rows — the case asserts `partial` before printing; D4 compile a 10,000-value `oneOf` and filter the 100,000 rows. Run on `cl-release` (command in Global Constraints) and write the medians and spreads into table.md "Benchmarks: configuring" rows D1–D4 (replacing "(P)" budgets by the recorded baseline plus its spread, as the Performance section says). + +- [ ] **Step 12: Docs build** (new public symbols) — command in Global Constraints; expected: exits 0. + +- [ ] **Step 13: Commit** `feat(table): value sets and distinct values` (body: the three mutations' failing lines). + +--- + +### Task 8b: Column groups with quantifiers + +**Spec:** "Column groups" (declaring, the filter, the truth tables, deciding `Together`, unloaded cells, the menu's pooled values list); "Unloaded cells in filters" (group rule); invariant 24 and invariant 1 for groups; Tests row "Column groups"; Tests row "Distinct values" (a group's "(Blanks)"); gates G5, G7; timings G1–G4, G6, G7; Neighbouring specs row `../forms/views.md` (`x-group`, step 3). + +**Files:** +- Modify: `include/morph/table/data_source.hpp` (`ColumnGroupInfo` moved here, `DataSource::groups()`, `VectorSource::setGroups`, `detail::checkGroups`) +- Modify: `include/morph/table/filter.hpp` (`GroupMode`, `ColumnCount`, `GroupFilter::{group, mode, match, exclude}`, `CompiledFilter::Group`, `compileGroup`, `compileFilter`'s groups parameter, `CompiledFilter::evaluateGroup` with quantifiers and counters, glaze metas) +- Modify: `include/morph/table/query.hpp` (`checkFilters` for groups, `apply` passing `source.groups()`, pooled `distinctValues`) +- Modify: `include/morph/table/engine.hpp` (`compile` passing `_source->groups()`; `fitToColumns` keeping a declared group's filter) +- Modify: `include/morph/table/description.hpp` (remove `ColumnGroupInfo`'s definition; it now comes from `data_source.hpp`) +- Modify: `include/morph/forms/forms.hpp` (`mergeSchemaExtras` writes `x-group`), `include/morph/forms/views.hpp` (`deriveColumns` copies `x-group`) +- Modify: `tests/table/wide_table_model.{hpp,cpp}` (declared groups `g3`, `g16`) +- Test: `tests/table/test_table_filter.cpp`, `tests/table/test_table_distinct.cpp`, `tests/table/test_table_columns.cpp`, `tests/table/bench_table.cpp` +- Docs: `docs/spec/forms/views.md` (`x-group` on a schema property, copied into the `v-columns` entry) + +**Interfaces:** +- Consumes: 8a's `ValueSet`, `FilterCounters`, `distinctValues`; Task 3's `Match`, `evaluate(Column const&, …)`. +- Produces: + ```cpp + // data_source.hpp (ColumnGroupInfo as Task 1 defined it: id, label, columns) + [[nodiscard]] virtual std::span DataSource::groups() const { return {}; } + void VectorSource::setGroups(std::vector groups); + // filter.hpp + enum class GroupMode : std::uint8_t { Columns, Together }; // JSON "columns", "together" + struct ColumnCount { enum class Kind : std::uint8_t { Any, All, AtLeast, Exactly } kind = Kind::Any; std::int64_t n = 1; }; + struct GroupFilter { std::string group; std::vector columns; GroupMode mode = GroupMode::Columns; + ColumnCount match; Combine combine = Combine::Any; + std::vector include; std::vector exclude; }; + [[nodiscard]] std::expected + compileFilter(FilterSpec const& spec, std::span columns, Services const& services, + std::span groups = {}); // and compileFilterGuarded likewise + static Match CompiledFilter::evaluateGroup(Group const& group, std::span keys, + std::uint32_t row, FilterCounters* counters = nullptr); // Task 3's, extended + ``` + `ColumnCount` JSON is `"any"`, `"all"`, `{"atLeast": n}` or `{"exactly": n}` (custom glaze read/write). + +- [ ] **Step 1: Write the failing truth-table test** in `test_table_filter.cpp`. Helper `Match groupMatch(VectorSource const&, GroupFilter, std::uint32_t row)` compiles `FilterSpec{.groups = {g}}` with `source.groups()`, builds the key columns as `Filtered::by` does, and returns `CompiledFilter::evaluateGroup(compiled->groups()[0], pointers, row)`. Header mutation line: "`Together` evaluated as `atLeast` with n = entries: the truth-table case fails on row F, F, X". + +```cpp +// companyA, companyB, companyC (Text), declared group {"company", "Company", {companyA, companyB, companyC}} +// 0: F, S, X 1: F, F, X 2: S, F, F 3: F, X, Y 4: X, Y, Z +TEST_CASE("groups: the spec's truth table", "[table][filter][groups]") { + auto const by = [&](GroupFilter g) { return f.by(FilterSpec{.columns = {}, .groups = {std::move(g)}}); }; + CHECK(by({.group = "company", .include = {eq("F")}}) == Rows{0, 1, 2, 3}); + CHECK(by({.group = "company", .mode = GroupMode::Together, .include = {eq("F"), eq("S")}}) == Rows{0, 2}); + CHECK(by({.group = "company", .include = {oneOf({"F", "S", "T"})}}) == Rows{0, 1, 2, 3}); + CHECK(by({.group = "company", .match = {.kind = ColumnCount::Kind::AtLeast, .n = 2}, .include = {oneOf({"F", "S"})}}) == Rows{0, 1, 2}); + CHECK(by({.group = "company", .match = {.kind = ColumnCount::Kind::Exactly, .n = 1}, .include = {eq("F")}}) == Rows{0, 3}); + CHECK(by({.group = "company", .match = {.kind = ColumnCount::Kind::All, .n = 1}, .include = {oneOf({"F", "S"})}}) == Rows{2}); + CHECK(by({.columns = {"companyA", "companyB", "companyC"}, .include = {eq("F")}}) == + by({.group = "company", .include = {eq("F")}})); // the plain group is `any` + CHECK(by({.group = "company", .include = {}, .exclude = {eq("X")}}) == Rows{2}); // "none of the companies is X" + std::vector seventeen; // only the 17th entry can match + for (int i = 1; i <= 16; ++i) seventeen.push_back(eq("A" + std::to_string(i))); + seventeen.push_back(eq("F")); + CHECK(by({.group = "company", .include = seventeen}) == Rows{0, 1, 2, 3}); // Mutation: a 16-bit entry mask + // dates: "two of the three are after 2026-05-01" over three Date columns is atLeast 2 with a gt entry +} + +TEST_CASE("groups: three-valued over unloaded cells", "[table][filter][groups]") { + // Rows of three Text cells, U = Cell{Unloaded{}}; each CHECK is groupMatch(source, filter, row) == expected. + struct Case { GroupFilter filter; std::array cells; Match expected; }; + // AtLeast 2, eq F: {F,F,U} Yes {X,X,U} No {F,X,U} Unknown + // Any, eq F: {F,U,U} Yes {X,X,U} Unknown {X,X,X} No + // All, eq F: {F,F,U} Unknown {F,X,U} No + // Exactly 1, eq F: {F,F,U} No {F,X,U} Unknown {X,X,U} Unknown {F,X,X} Yes {X,X,X} No + // Together F,S: {F,S,U} Yes {F,F,U} Unknown {U,U,X} Unknown {X,X,U} No + // include {}, exclude eq X: {F,X,U} No {F,S,U} Unknown {F,S,F} Yes + // isEmpty, Any: {X,X,U} Unknown (an unloaded cell is not empty) +} + +TEST_CASE("groups: duplicate columns, exactly zero and too many together entries", "[table][filter][groups]") { // Review Focus 3 + CHECK(f.by(FilterSpec{.columns = {}, .groups = {{.columns = {"companyA", "companyA"}, .include = {eq("F")}}}}) + .error().code == TableErrorCode::InvalidSpec); + CHECK(by({.group = "company", .match = {.kind = ColumnCount::Kind::Exactly, .n = 0}, .include = {eq("F")}}) == Rows{4}); + CHECK(f.by(FilterSpec{.columns = {}, .groups = {{.group = "company", .mode = GroupMode::Together, + .include = {eq("A"), eq("B"), eq("C"), eq("D")}}}}) + .error().code == TableErrorCode::InvalidSpec); + CHECK(f.by(FilterSpec{.columns = {}, .groups = {{.group = "company", .mode = GroupMode::Together, .include = {eq("A")}}}}) + .error().code == TableErrorCode::InvalidSpec); // fewer than 2 entries + CHECK(f.by(FilterSpec{.columns = {}, .groups = {{.group = "company", .match = {.kind = ColumnCount::Kind::AtLeast, .n = 4}, + .include = {eq("F")}}}}).error().code == TableErrorCode::InvalidSpec); + CHECK(f.by(FilterSpec{.columns = {}, .groups = {{.group = "company", .columns = {"companyA", "companyB"}, + .include = {eq("F")}}}}).error().code == TableErrorCode::InvalidSpec); + CHECK(f.by(FilterSpec{.columns = {}, .groups = {{.group = "nope", .include = {eq("F")}}}}).error().code == + TableErrorCode::UnknownColumn); +} + +TEST_CASE("groups: a declared group is checked where the columns are read", "[table][filter][groups]") { + // setGroups with: one column; 17 columns; mixed kinds (Text + Integer); a column in two groups. + // apply(*source, {}) on each is TableErrorCode::SourceContract; a well-formed declaration applies. +} + +TEST_CASE("groups: the JSON shape reads and writes", "[table][filter][groups]") { + auto const json = R"({"group":"company","mode":"together","match":{"atLeast":2},"combine":"all","include":[{"eq":"F"},{"eq":"S"}],"exclude":[{"eq":"X"}]})"; + GroupFilter g; + REQUIRE_FALSE(glz::read_json(g, std::string{json})); + CHECK(g.mode == GroupMode::Together); CHECK(g.match.kind == ColumnCount::Kind::AtLeast); CHECK(g.match.n == 2); + for (auto const* m : {R"("any")", R"("all")", R"({"atLeast":3})", R"({"exactly":0})"}) { /* read then write gives m back */ } +} + +TEST_CASE("groups: invariant 24 over random tables", "[table][filter][groups]") { + // std::mt19937 seed 24; 300 tables of 40 rows, k = 2..5 Text columns over {A..E, empty}; + // random mode, match (n in 0..k), combine, 1..k include entries (eq, oneOf, ne, startsWith). + // Oracle: Columns mode counts columns whose cell satisfies the entries one by one (CompiledFilter::matches); + // Together tries every injective assignment of entries to columns. + CHECK(f.by(spec) == oracleRows); +} + +TEST_CASE("groups: invariant 1 over random completions", "[table][filter][groups]") { // gate G7's oracle + // std::mt19937 seed 1; 200 tables of 30 rows, k = 3, about a third of the cells Unloaded; + // for each row, every completion of its unloaded cells over {A..E, empty}: + // evaluate == Yes => the oracle passes every completion; == No => it passes none; + // == Unknown => it passes some and fails some. +} + +TEST_CASE("groups: a row costs k lookups and at most one augmentation an entry", "[table][filter][groups]") { // gate G5 + // 1,000 rows, a group of 16 Text columns, every cell valid: "v<0..31>" from std::mt19937{5}; no exclude entries. + // FilterCounters over every row. + CHECK(absentSetCounters.setLookups == 16 * 1'000); // `any`, oneOf {"zz1", "zz2"}: no row holds it, no early stop + CHECK(heldSetCounters.setLookups >= 1'000); // `any`, oneOf {"v0", "v1"}: stops at the first match + CHECK(heldSetCounters.setLookups < 16 * 1'000); + CHECK(togetherCounters.setLookups == 16 * 1'000); // Together, 16 eq entries "v0".."v15": reads every column + // per row, a fresh FilterCounters: a row whose cells leave some entry unmatched has 0 augmentations, + // every other row at most 16. + CHECK(maxAugmentationsOnUncoveredRows == 0); + CHECK(maxAugmentationsOnCoveredRows <= 16); +} +``` + +- [ ] **Step 2: Write the failing pooled-values test** in `test_table_distinct.cpp`: + +```cpp +TEST_CASE("distinct: a group's values are pooled", "[table][distinct][groups]") { + // the truth-table source plus row 5: empty, F, U(nloaded); and a column filter companyA notEmpty is NOT set + auto const d = distinctValues(*source, {}, DistinctRequest{.group = "company"}); + // F counted once per row that has it in any column + CHECK(countOf(d, "F") == 5); // rows 0, 1, 2, 3, 5 + CHECK(countOf(d, "S") == 2); // rows 0, 2 + CHECK(d->blanks == 1); // row 5: the rows an isEmpty entry under any admits + CHECK(d->partial); // row 5's unloaded cell + TableQuery own; // the group's own filter is removed, others stay + own.filters.groups.push_back({.group = "company", .include = {eq("Z")}}); + CHECK(countOf(distinctValues(*source, own, {.group = "company"}), "F") == 5); + CHECK(distinctValues(*source, {}, {.group = "nope"}).error().code == TableErrorCode::UnknownColumn); +} +``` + +- [ ] **Step 3: Write the failing `x-group` test** in `test_table_columns.cpp`: a row type `GroupRow{std::string companyA, companyB, companyC, name;}` with `static constexpr auto columnGroups() noexcept` returning one `{"company", {"companyA","companyB","companyC"}}` (`table::ColumnGroup{std::string_view id; std::span columns;}`, declared in `data_source.hpp`); `"derived columns carry x-group"`: the schema property of `companyB` holds `"x-group": "company"`, `deriveColumns` copies it into `companyB`'s entry, `name` has none. Mutation: the copy list in `deriveColumns` without `"x-group"`. + +- [ ] **Step 4: Run; the new cases fail to compile** (`GroupMode`, `ColumnCount`, `setGroups` undeclared). + +- [ ] **Step 5: Implement declarations and compile.** + - Move `ColumnGroupInfo` (unchanged members, `operator== = default` and Doxygen) from `description.hpp` to `data_source.hpp`; add `struct ColumnGroup` (the typed row's constexpr declaration), `DataSource::groups()`, `VectorSource::setGroups`, and `[[nodiscard]] std::optional detail::checkGroups(std::span, std::span)`: each group has 2 to 16 columns, all present and of one kind, and no column is in two groups; otherwise `SourceContract` naming the group's id. `apply`, `distinctValues` and `Engine::setFilter` run it before compiling. + - `GroupFilter` gains members in the order of the Interfaces block (existing designated initializers `.columns, .combine, .include` stay valid). `glz::meta` for `GroupMode` and the custom read/write for `ColumnCount`. + - `compileGroup(GroupFilter const&, columns, services, groups)`: exactly one of `group` and `columns` is non-empty, else `InvalidSpec`; `group` resolves through `groups` (`UnknownColumn` naming it when absent); a column named twice is `InvalidSpec`; `Together` takes 2 to k include entries, else `InvalidSpec`; `AtLeast`/`Exactly` take `0 <= n <= k`, else `InvalidSpec`. `CompiledFilter::Group` keeps `k` (the named column count), `mode`, `match`, and per column the include and exclude entries that compiled for it; a column on which `combine: all` cannot compile every include entry, or `any` none, never satisfies (it counts as a known *no*). An include or exclude entry that compiles on none of the columns is the error the existing rule returns. A group with no include and no exclude entry compiles to nothing. + - Per column, the compile builds a **value index**: one hash map from a cell key to the bit mask (`std::uint32_t`: `Columns` mode takes up to `maxEntries` = 32 entries; only `Together` is capped at 16) of the include entries that are `eq` or `oneOf` on it (folded text unless `caseSensitive`), and the list of the other include entries with their bit. `exclude` is compiled the same way into its own index. + - `checkFilters`: a group's `include + exclude` count against `maxEntries`, every exclude entry against `maxTextLength`; `group` and `columns` resolve and pass `checkColumn` (a declared group through `source.groups()`, which `validate` receives as a new trailing `std::span groups = {}` parameter). + +- [ ] **Step 6: Extend `CompiledFilter::evaluateGroup(Group const&, keys, row, counters)`** with this algorithm (the spec fixes it; G5 counts it): + +```text +mask(c): cell = keys[c] at row; counters->cellReads += 1 + Unloaded -> unknown column + Empty / Invalid -> entries matching by the column-filter rules (isEmpty / notEmpty only) + Valid -> index lookup (counters->setLookups += 1) | each condition entry tested + +empty include: the include part is Yes, and neither `mode` nor `match` is consulted (an exclude-only group + compiles in either mode); only the exclude part below decides. + +Columns mode (each column known-satisfies when combine any: mask != 0; all: mask == every compiled include bit +and the column compiled every include entry): read columns in order, keeping y (satisfying), q (unknown), +r (columns not yet read); stop as soon as decided: + any Yes y>=1 No y+q+r == 0 + all Yes y == k No y+q+r < k + atLeast Yes y>=n No y+q+r < n + exactly Yes y==n, q==0, r==0 No y>n or y+q+r < n + otherwise, after the last column: Unknown + +Together mode (E = include entries, full = (1< No (no augmentation) + known pass: Kuhn matching over known masks; for e in 0..E-1: + counters->augmentations += 1; if !augment(e) { fail; break } + all augmented -> Yes + unknownCols empty -> No + wildcard pass: the same matching with mask(c) = full for every unknown column + (not counted) -> Unknown if it covers every entry, else No + augment(e): for each column c with bit e in mask(c), not visited: visit c; + if owner[c] is none or augment(owner[c]): owner[c] = e; return true + return false + +exclude (both modes): any known cell matching any exclude entry -> No; otherwise any unknown column -> Unknown; + otherwise Yes. A group with exclude entries reads every column: once the include part is decided its reading + stops, and the columns it did not read are still read for the exclude index (one exclude-index lookup per valid + cell, counted in setLookups like the include index's). Empty and invalid cells do no index lookup. +group result: No if include-part or exclude-part is No; else Unknown if either is Unknown; else Yes. +``` + + `FilterRun`'s group stage and `evaluateRow` already call `evaluateGroup` (Task 3); they pass no counters. + +- [ ] **Step 7: Engine and pooled values.** `Engine::compile` passes `_source->groups()`; `fitToColumns` keeps a `GroupFilter` whose `group` the source still declares and drops one whose `group` it no longer declares, then erases groups whose `columns` and `group` are both empty. `distinctValues` with `request.group`: the rows passing every filter with every `GroupFilter` whose `group == request.group` removed; per candidate row, the set of distinct keys over the group's loaded valid cells (a row counts once per value); `blanks` counts rows with at least one empty cell, `invalid` rows with at least one invalid cell; an unloaded cell sets `partial` and the row's loaded cells still count. A value's text and operand come from the first row and column (in group order) that holds it. Remove 8a's `Unavailable` branch. `levelsNeeded` adds the level of every column of `request.group` to `order`; test `"distinct: a group's levels are order levels of the request"` in `test_table_distinct.cpp` (group `company` with `companyB` at level `"slow"`: `levelsNeeded(q with distinct{.group = "company"}, …).order == {"slow"}`; mutation: the group's columns left out). + +- [ ] **Step 8: `x-group`.** `mergeSchemaExtras` writes `"x-group": ""` on each property a row's `columnGroups()` names; `deriveColumns` adds `"x-group"` to the keys it copies. `docs/spec/forms/views.md`: one line under the copied keys, and the `v-columns` row's key list gains `"x-group"?`. + +- [ ] **Step 9: Fixture.** `bench::WideTableSource::groups()` returns, from `layout()`'s `group` attributes, `{"g3", "Group of 3", {t004, t005, t006}}` and `{"g16", "Group of 16", {t011 ... t026}}` (`wideColumns(30)` keeps only `t004`...`t006`, so the 30-column source declares `g3` alone). + +- [ ] **Step 10: Run** `ctest --test-dir build/cl-debug -R "groups|distinct|x-group|filter" --output-on-failure`; expected: all pass, and the existing group cases (`"columns combine by AND, and a group matches on any of its columns"`, `"an all-group uses only the columns that evaluate every entry"`) unchanged and passing. + +- [ ] **Step 11: Mutations.** (a) `Together` decided as `atLeast` with n = entry count: the truth table fails on row `F, F, X`. (b) `exactly` deciding `Yes` while `q > 0`: the three-valued case `{F,X,U}` fails. (c) the known pass reading unknown columns as matching every entry: the three-valued case `{F,F,U}` reports `Yes`. (d) skipping the covered-mask check: G5 fails (augmentations on rows that cannot match). Paste each; revert. + +- [ ] **Step 12: Timings G1–G4, G6, G7.** `TEST_CASE("table benchmark: column groups", "[.benchmark][table][groups]")` on 100,000 rows generated with `std::mt19937{7}` (a group of 3 and one of 16 Text columns, keys built): G1 `any` with a 2-value `oneOf`; G2 with a 1,000-value `oneOf`; G3 `atLeast 2` and `exactly 1`; G4 `together` 2 entries over 3 columns and 16 over 16; G6 pooled distinct values, first 200; G7 G1 and G4 with a third of the cells `Unloaded` — the case first asserts the undecided count equals the oracle's from Step 1's invariant-1 test on a 1,000-row slice. Write the figures into table.md "Benchmarks: column groups". + +- [ ] **Step 13: Docs build; commit** `feat(table): column groups with quantifiers`. + +--- + +### Task 8c: Values with units + +**Spec:** "Values with units" (all subsections); "Rows on the wire" (the `Measure` row of the cell table); invariant 25; Tests rows "Values with units", "Unit numbers across registries", "The generic page" (measure half); Failure modes row "A level cache read under a registry of another lineage"; gate M5; timings M1–M4. + +**Files:** +- Modify: `include/morph/table/data_source.hpp` (`UnitInfo`, `UnitId`, `kUnknownUnit`, `UnitRegistry`, `UnitRegistryBuilder`, `addUnitsOf`, `Measure`, `Cell`'s last alternative, `ColumnKind::Measure`, `ColumnInfo::dimensions`, `Services::units`, `displayText`, `unitsFor`) +- Modify: `include/morph/table/sort.hpp` (`KeyColumn` for `Measure`, `compareKeys`) +- Modify: `include/morph/table/filter.hpp` (operands with units, `ValueSet` of measures) +- Modify: `include/morph/table/query.hpp` (`MeasureValue`, `kindOf`, `toCell`, `RowsSource`'s registry, typed `apply`, `distinctValues` on measures, `DistinctValue::{dimension, unit}`, the `SourceContract` check) +- Modify: `include/morph/table/description.hpp` (`TableDescription::units` after `levels`; the generic page's `Measure` cell; `registryOf`), `include/morph/core/registry.hpp` (`DescribeTable::invoke` fills `units`) +- Modify: `include/morph/table/level_cache.hpp` (`LevelCacheOptions::units`, the lineage check) +- Modify: `include/morph/forms/forms.hpp` (`x-measure` on a `MeasureValue` property), `include/morph/forms/views.hpp` (`inferColumnKind` → `"measure"`) +- Create: `include/morph/ui/measure.hpp` (`ui::TypedMeasure`, `ui::parseMeasure`, `ui::measureOperand`); `tests/table/test_table_measure.cpp` (in `morph_table_tests`; add to `tests/CMakeLists.txt`) +- Modify: `tests/table/test_table_description.cpp`, `tests/table/bench_table.cpp`, `tests/table/wide_table_model.{hpp,cpp}` (its 10 measure columns become `ColumnKind::Measure` over the fixture registry below) + +**Interfaces:** +- Consumes: 8a's `ValueSet`, `distinctValues`; Task 3's `Unloaded`, `SourceContract`; Task 1's generic page; Task 5's `CellLevelCache`, `LevelCacheOptions`, `LevelCacheStats`. +- Produces: + ```cpp + struct UnitInfo { std::string id; std::string symbol; std::string dimension; math::Rational toBase; std::uint32_t decimals = 0; }; + using UnitId = std::uint32_t; + inline constexpr UnitId kUnknownUnit = std::numeric_limits::max(); // never a live unit + class UnitRegistry { + public: + [[nodiscard]] std::optional find(std::string_view id) const; // counts findCalls() + [[nodiscard]] UnitInfo const& info(UnitId unit) const; // precondition: contains(unit) + [[nodiscard]] bool contains(UnitId unit) const noexcept; // false for a tombstone or kUnknownUnit + [[nodiscard]] std::span dimensions() const; + [[nodiscard]] std::span unitsOf(std::string_view dimension) const; + [[nodiscard]] std::uint64_t lineage() const noexcept; + [[nodiscard]] std::uint64_t findCalls() const noexcept; // relaxed counter, for gate M5 + }; + class UnitRegistryBuilder { + public: + UnitRegistryBuilder(); // seeds {"", "", "ratio", 1, 6} + explicit UnitRegistryBuilder(std::shared_ptr previous); + std::expected add(UnitInfo unit); + bool remove(std::string_view id); + [[nodiscard]] std::shared_ptr build() &&; + }; + template + std::expected addUnitsOf(UnitRegistryBuilder& builder, std::span units, + std::string_view (*dimensionOf)(E)); + struct Measure { math::Rational amount; UnitId unit = 0; }; + using Cell = std::variant; + // ColumnKind::Measure appended after Custom, JSON "measure"; ColumnInfo::dimensions (std::vector, last member) + // Services::units: std::shared_ptr + [[nodiscard]] std::vector unitsFor(UnitRegistry const& registry, std::span columns); + // query.hpp + struct MeasureValue { math::Rational amount; std::string unit; }; // JSON {"num","den","unit"}; schema x-measure + struct DistinctValue { /* 8a members */ std::string dimension; std::string unit; }; + // description.hpp + [[nodiscard]] std::expected, TableError> registryOf(TableDescription const& description); + // writeGenericPage / readGenericPage gain a trailing `UnitRegistry const* units = nullptr` + // level_cache.hpp + // LevelCacheOptions::units: std::function()> (empty: no units) + // ui/measure.hpp + namespace morph::ui { + struct TypedMeasure { math::Rational amount; std::string unit; }; // unit: the unit's id + [[nodiscard]] std::optional parseMeasure(std::string_view text, std::span units, + LocaleFacts const& locale, std::string_view chosen = {}); + [[nodiscard]] std::string measureOperand(TypedMeasure const& measure); // "0.3 kg"; "0.003" for the ratio unit + } + ``` + +The test registry used below (built once in the file): `""` (seeded), `kg` (`"kg"`, mass, 1, 3), `g` (`"g"`, mass, 1/1000, 0), `percent` (`"%"`, ratio, 1/100, 2). Helper `m(std::string_view amount, std::string_view unit) -> Cell` is `Measure{dec(amount), *registry->find(unit)}`. Columns: `col("w", ColumnKind::Measure)` with `dimensions = {"mass", "ratio"}` unless a case says otherwise; `ApplyOptions::services.units = registry`. + +- [ ] **Step 1: Write the failing order and filter tests** in `test_table_measure.cpp` (header mutation line: "a key that ignores the unit's factor: `300 g sorts between 0.2 kg and 0.5 kg` fails"): + +```cpp +// rows: 0: 0.5 kg 1: 0.3 % 2: 300 g 3: 0.2 kg 4: empty 5: 0.2 % +TEST_CASE("measure: 300 g sorts between 0.2 kg and 0.5 kg in both directions", "[table][measure]") { + CHECK(sorted(SortDirection::Ascending) == std::vector{3, 2, 0, 5, 1, 4}); + CHECK(sorted(SortDirection::Descending) == std::vector{0, 2, 3, 1, 5, 4}); // groups stay, amounts reverse +} + +TEST_CASE("measure: filters compare base amounts within a dimension", "[table][measure]") { + CHECK(rowsOf(gt("250 g")) == Rows{0, 2}); // drops 0.2 kg and both percents + CHECK(rowsOf(eq("300 g")) == rowsOf(eq("0.3 kg"))); + CHECK(rowsOf(eq("0.3 kg")) == Rows{2}); // 300 g; not 0.3 %: they are not equal + CHECK(rowsOf(ne("250 g")) == Rows{0, 1, 2, 3, 5}); // other dimensions included; empty never matches + CHECK(rowsOf(eq("0.003")) == Rows{1}); // no unit: a ratio, equal to 0.3 % + CHECK(rowsOf(oneOf({"300 g", "0.2 percent"})) == Rows{2, 5}); + CHECK(rowsOf(between("0.25 kg", "600 g")) == Rows{0, 2}); + CHECK(error(eq("250 lb")).code == TableErrorCode::InvalidValue); CHECK(error(eq("250 lb")).column == "w"); + CHECK(error(between("1 kg", "50 percent")).code == TableErrorCode::InvalidValue); + CHECK(errorOnMassOnly(eq("0.003")).code == TableErrorCode::InvalidValue); // dimensions = {"mass"} +} + +TEST_CASE("measure: invalid cells and the registry contract", "[table][measure]") { + // Measure{1, kUnknownUnit}; a ratio cell in a mass-only column: both Invalid, sort with empty after valid + CHECK(sortedMassOnly(SortDirection::Ascending).back() == /* the empty row */); + ApplyOptions none; // services.units null + CHECK(apply(*source, sortBy("w"), none).error().code == TableErrorCode::SourceContract); +} + +TEST_CASE("measure: negative and zero amounts", "[table][measure]") { // Review Focus 4 + // mass+ratio column: -300 g, -0.2 kg, 0 g, 0 kg, -0.5 % + CHECK(sortedAsc == std::vector{0, 1, 2, 3, 4}); // -300 g < -0.2 kg < 0 g = 0 kg (by row), then ratio + CHECK(sortedDesc == std::vector{2, 3, 1, 0, 4}); // mass reversed, ties by source row, ratio last + CHECK(distinctCount("0 g") == 2); // 0 g and 0 kg one value + CHECK(errorOnMassOnly(eq("0")).code == TableErrorCode::InvalidValue); +} + +TEST_CASE("measure: 0.3 kg and 300 g are one distinct value", "[table][measure]") { + // rows: 0.3 kg, 300 g, 0.3 % + REQUIRE(d->values.size() == 2); + CHECK(d->values[0].count == 2); CHECK(d->values[0].dimension == "mass"); CHECK(d->values[0].unit == "kg"); + CHECK(d->values[0].text == "0.3 kg"); CHECK(d->values[0].operand == "0.3 kg"); + CHECK(d->values[1].dimension == "ratio"); CHECK(d->values[1].text == "0.3 %"); CHECK(d->values[1].operand == "0.3 percent"); +} + +TEST_CASE("measure: invariant 25 over random amounts", "[table][measure]") { + // std::mt19937 seed 25; 500 rows of random amounts in kg, g, percent, ""; sort ascending and descending; + // within a dimension the order equals the order by amount * toBase (exact Rational), ties by source row; + // eq on any one row's operand matches exactly the rows of equal base amount and equal dimension. +} + +TEST_CASE("measure: keys look units up by number only", "[table][measure]") { // gate M5 + auto const before = registry->findCalls(); + // build the Measure KeyColumn over 100,000 cells (KeyColumn::forColumn + buildKeys) + CHECK(registry->findCalls() == before); +} +``` + +- [ ] **Step 2: Write the failing registry tests:** + +```cpp +TEST_CASE("units: a unit id is a token", "[table][measure]") { + UnitRegistryBuilder b; + for (std::string_view bad : {"k g", "1kg", "-kg", "+kg", ".kg", ",kg", "", "abcdefghijklmnopqrstuvwxyz0123456"}) // last: 33 bytes + CHECK(b.add({.id = std::string{bad}, .symbol = "x", .dimension = "mass", .toBase = dec("1"), .decimals = 0}).error().code + == TableErrorCode::InvalidSpec); + CHECK(b.add(kg)); CHECK_FALSE(b.add(kg)); // a second unit of one id + CHECK(b.add({.id = "z", .symbol = "z", .dimension = "mass", .toBase = dec("0"), .decimals = 0}).error().code == TableErrorCode::InvalidSpec); + auto const r = std::move(b).build(); + CHECK(r->find("") == UnitId{0}); CHECK(r->info(0).dimension == "ratio"); + CHECK(std::ranges::equal(r->dimensions(), std::vector{"ratio", "mass"})); +} + +TEST_CASE("units: numbers survive a successor and tombstones stay", "[table][measure]") { + auto const first = /* "", kg, g, percent */; + UnitRegistryBuilder next{first}; + CHECK(next.remove("g")); + auto const mg = next.add({.id = "mg", .symbol = "mg", .dimension = "mass", .toBase = dec("0.000001"), .decimals = 0}); + auto const second = std::move(next).build(); + CHECK(second->find("kg") == first->find("kg")); + CHECK_FALSE(second->find("g")); + CHECK_FALSE(second->contains(*first->find("g"))); + CHECK(*mg == 4); // after "", kg, g, percent; g's 2 never reused + CHECK(second->lineage() == first->lineage()); + CHECK(std::move(UnitRegistryBuilder{}).build()->lineage() != first->lineage()); + CHECK(std::ranges::equal(second->unitsOf("mass"), std::vector{*first->find("kg"), *mg})); // largest factor first + // a Measure{1, g's number} under `second` is an invalid cell +} + +TEST_CASE("units: a level cache keeps values under a successor and drops them under another lineage", "[table][measure]") { + // CellLevelCache over one Measure column, LevelCacheOptions::units returning `current`; + // load key 1 (Measure 300 g), snapshot reads it, stats().loaderCalls == 1. + current = successor; // same lineage + CHECK(read(1) == LevelRead::Loaded); CHECK(cache.stats().loaderCalls == 1); + current = fresh; // another lineage + CHECK(read(1) == LevelRead::Unloaded); CHECK(cache.stats().invalidatedKeys >= 1); +} + +TEST_CASE("units: addUnitsOf reads a unit enum", "[table][measure]") { + // tests' own enum TmUnit { kg, g, percent } with UnitTraits (meta + relations g->kg 1/1000); + // dimensionOf maps kg, g -> "mass", percent -> "ratio". + // registry holds kg (toBase 1, decimals from meta), g (1/1000); percent is refused (InvalidSpec): + // a ratio unit must reach the dimension's base "" through relations, and TmUnit has none to it. +} +``` + + `addUnitsOf` rule pinned by this test: within a dimension the base is the dimension's existing base unit (`""` for ratio) or else the first enumerator of `units` in that dimension; every other unit's `toBase` is the product of `UnitRelation::fromTo` along a relation path to the base (searched in both directions, the reverse taking the reciprocal); a unit with no path is `InvalidSpec`; `symbol` is `UnitMeta::display`, `decimals` `UnitMeta::defaultDecimals`. + +- [ ] **Step 3: Write the failing wire, typed-row and parser tests.** In `test_table_description.cpp`: `"generic page: a measure round-trips"` — a page of one Measure column holding `Measure{3/10, kg}` writes `[3, 10, "kg"]`, reads back equal through `registryOf(description)`; reading with a null registry is `SourceContract`; an unknown id in the JSON reads as `Measure{…, kUnknownUnit}` (an invalid cell). In `test_table_measure.cpp`: + +```cpp +TEST_CASE("measure: typed rows carry their unit by id", "[table][measure]") { + struct MRow { std::int64_t id; MeasureValue weight; }; // file-scope in the real test, for glaze + // typed apply with services.units: column "weight" is ColumnKind::Measure; sort ascending by weight over + // {0.5 kg, 300 g, "lb"(unknown)} gives 300 g, 0.5 kg, then the unknown unit as invalid. + // schemaJson() property weight has "x-measure": true; deriveColumns gives kind "measure". +} + +TEST_CASE("measure: what a user types", "[table][measure]") { + auto const en = ui::localeFacts("en"); auto const de = ui::localeFacts("de"); + CHECK(ui::parseMeasure("0.3kg", units, en) == ui::TypedMeasure{dec("0.3"), "kg"}); + CHECK(ui::parseMeasure("0.3 kg", units, en) == ui::TypedMeasure{dec("0.3"), "kg"}); + CHECK(ui::parseMeasure("300 g", units, en) == ui::TypedMeasure{dec("300"), "g"}); + CHECK(ui::parseMeasure("0.3 KG", units, en) == ui::TypedMeasure{dec("0.3"), "kg"}); // then without case + CHECK(ui::parseMeasure("5 %", units, en) == ui::TypedMeasure{dec("5"), "percent"}); // by symbol + CHECK(ui::parseMeasure("0,3 kg", units, de) == ui::TypedMeasure{dec("0.3"), "kg"}); + CHECK_FALSE(ui::parseMeasure("0.3 kg", units, de)); // '.' is de's group separator + CHECK(ui::parseMeasure("12", units, en, "g") == ui::TypedMeasure{dec("12"), "g"}); + CHECK(ui::parseMeasure("12", units, en) == ui::TypedMeasure{dec("12"), ""}); + CHECK(ui::parseMeasure("-1.5 g", units, en) == ui::TypedMeasure{dec("-1.5"), "g"}); + CHECK_FALSE(ui::parseMeasure("0.3 lb", units, en)); + CHECK(ui::measureOperand({dec("0.3"), "kg"}) == "0.3 kg"); + CHECK(ui::measureOperand({dec("0.003"), ""}) == "0.003"); +} +``` + +- [ ] **Step 4: Run; the new cases fail to compile** (`UnitRegistryBuilder`, `Measure`, `ColumnKind::Measure` undeclared). + +- [ ] **Step 5: Registry and cell** (`data_source.hpp`): the Interfaces block; `static_assert(sizeof(Measure) <= sizeof(QuantityCell))` beside `Cell`. `UnitRegistry` stores units in a vector indexed by `UnitId` (a tombstone slot is kept), a `std::unordered_map` for `find`, dimensions in first-registration order (kept from `previous`), and per dimension its unit ids sorted by `toBase` descending. `lineage()` is drawn from a process-wide `std::atomic` when a builder starts from nothing and copied from `previous` otherwise. The builder refuses (`InvalidSpec`): an id of 0 or more than 32 bytes, holding white space, or beginning with a digit, `+`, `-`, `.` or `,`; an id already live; `toBase <= 0`; an empty `dimension`. The ratio unit `""` is seeded only by the default constructor and cannot be removed. `displayText(Measure)` is the amount's decimal text (`formatRationalDecimal`); a unit's symbol is the display layer's. `unitsFor` returns the live units of every dimension a `Measure` column of `columns` allows (all dimensions for an empty `dimensions`), in registry order. `ColumnKind::Measure` is appended after `Custom` with JSON `"measure"`. + +- [ ] **Step 6: Keys and order** (`sort.hpp`). `KeyColumn` gains `std::vector places` (per row) and, built once in `forColumn` from `services.units`, `std::vector unitPlaces` and `std::vector unitFactors` indexed by `UnitId` (`-1` for a tombstone or a dimension the column does not allow; the place is the dimension's index in `info.dimensions`, or in `registry.dimensions()` when that is empty). Reading a cell into a `Measure` column: `Measure` → place and `checkedMul(amount, factor)` into `exact` (overflow or `-1` → `Invalid`); `Rational`/`int64` → the ratio unit `""`; `std::string` → parsed as an operand (Step 7's grammar); anything else `Invalid`. `compareKeys` for `Measure`, after the rank rule: + +```text +if places[l] != places[r]: return places[l] < places[r] ? -1 : 1 // direction not applied +result = sign(exact[l] <=> exact[r]); return dir == Descending ? -result : result +``` + +- [ ] **Step 7: Filters** (`filter.hpp`). `operatorDefined(Measure, op)` is the numeric kinds' rule. An operand on a `Measure` column is `` or ` ` (one space; the decimal by `parseDecimal`); no id is the ratio unit `""`. The unit must be live, its dimension allowed by the column, and `checkedMul` must fit; otherwise `InvalidValue` naming the column. `CompiledFilter::Entry` gains `std::array places`; `between`'s two places must be equal (`InvalidValue`). `matches` on a valid `Measure` cell: `Ne` → `places != entry.places[0] || exact != operand`; every other comparison → `false` when the places differ, else `compare` on `exact`. `ValueSet` holds `(place, base amount)` pairs for a `Measure` column. + +- [ ] **Step 8: Typed rows, distinct, wire, level cache, parser.** + - `query.hpp`: `MeasureValue` with `glz::meta` writing `{"num","den","unit"}`; `kindOf()` is `Measure`; the cell readers become `Cell (*)(Row const&, UnitRegistry const*)` and `toCell(MeasureValue)` resolves the id (`find`), giving `kUnknownUnit` when absent; `RowsSource` takes `std::shared_ptr units` as its last constructor parameter and the typed `apply` passes `options.services.units`. `apply`, `distinctValues` and `Engine::setSort`/`setFilter` return `SourceContract` when a column they sort, filter or list values of is `Measure` and `services.units` is null; `writeGenericPage` does when a column it writes is `Measure` and its `units` is null. `distinctValues` on a `Measure` column groups by `(place, base amount)`; a value's `text` is ` ` of its first row (no space and no symbol for `""`), `operand` ` `, `dimension` and `unit` that row's. + - `description.hpp`: `TableDescription::units` (`std::vector`, after `levels`, so `describe`'s revision covers it); `DescribeTable::invoke` (Task 2) sets `units = unitsFor(*served.services.units, columnInfos(description))` before `describe` when a column is `Measure` and the model left `units` empty, and `detail::answerQuery` passes `served.services.units.get()` to `writeGenericPage`. `registryOf(description)` builds a registry from `description.units` (`""` already seeded and skipped). The page writer writes `Measure` as `[num, den, ""]` through `units`; the reader resolves the id through `units`; a `Measure` column with `units == nullptr` is `SourceContract`. + - `level_cache.hpp`: the cache records `units()->lineage()` with each stored value set; `snapshot()` first compares the current registry's lineage with the recorded one and, when they differ, runs `invalidateAll()` (counted in `invalidatedKeys`). + - `forms.hpp`: a `MeasureValue` property gets `"x-measure": true`. `views.hpp`: `inferColumnKind` returns `"measure"` for an `x-measure` property. + - `ui/measure.hpp`: `parseMeasure` reads an optional `-`, digits, at most one `locale.numeric.decimalSeparator` followed by digits, optional spaces, then a unit: an exact match of a unit's `symbol` or `id` first, then the same compared ASCII-case-insensitively, first in `units` order; no unit text takes `chosen`. Anything else (a group separator included) is `std::nullopt`. `measureOperand` writes `formatRationalDecimal(amount)`, then `" " + unit` unless `unit` is empty. + +- [ ] **Step 9: Fixture.** `bench::WideTableSource` owns a registry of three dimensions and ten units — mass `kg` 1, `g` 1/1000, `mg` 1/1'000'000, `t` 1000; volume `l` 1, `ml` 1/1000, `m3` 1000; ratio `""` 1, `percent` 1/100, `permille` 1/1000 — exposes it as `std::shared_ptr units() const`, and `m000`...`m009` become `ColumnKind::Measure` (role `Measure`, `dimensions` empty) holding `Measure` cells drawn over those units from the same `cellBits(seed, id, row)`; `"wide table: the column mix is the spec's"` keeps counting them as the 10 measures. The call sites that now need the registry (each passes `source.units()` / `.get()`): + - `bench::WideTableModel`'s three `ServedTable`s `wide`, `small`, `scoped` (Part A, Task 2): `services.units = source->units()`, so `DescribeTable` fills `units` and `answerQuery` writes measures; + - Part A's `"generic page"` cases that write a `WideTableSource{1'000, 30, 7}` page with an empty column list (the 30 layout holds `m000`): `writeGenericPage(…, source.units().get())`; + - `tests/table/bench_table.cpp`: the `[.benchmark][wire]` cases (W1, W2, W7) and every `apply`/`Engine` over a `WideTableSource` whose query sorts, filters or writes an `m` column; + - the rest found by `grep -rn "WideTableSource\|WideTableModel" tests` — each `ApplyOptions`/`EngineOptions`/`ServedTable` built over the fixture gets `services.units = source.units()`, whether or not it touches an `m` column today, so a later query cannot hit `SourceContract`. + +- [ ] **Step 10: Run** `cmake --build --preset cl-debug --target morph_table_tests morph_tests`, then `ctest --test-dir build/cl-debug -R "measure|units|generic page|wide table|over the bridge" --output-on-failure`, then `-R "table|views"`; expected: all pass. `morph_net_tests` (the wide model over a socket) is POSIX-only: run `-R "push"` on `linux-everything` before merging. + +- [ ] **Step 11: Mutations.** (a) `KeyColumn` storing `amount` instead of `amount * factor`: `"300 g sorts between…"` fails. (b) `compareKeys` negating the place comparison under `Descending`: the same case's descending line fails. (c) `ne` returning `false` across dimensions: `"filters compare base amounts…"` fails on `ne("250 g")`. (d) `forColumn` dropped and `find(info(unit).id)` per cell: M5 fails. (e) the level cache skipping the lineage check: the level-cache case fails. Paste each; revert. + +- [ ] **Step 12: Timings M1–M4.** `TEST_CASE("table benchmark: values with units", "[.benchmark][table][measure]")`: 100,000 rows from `std::mt19937{7}`, one `Measure` column over the fixture registry (three dimensions, ten units) and a `Quantity` column of the same amounts converted by factor; M1 keys and sort of each; M2 `gt "250 g"` against the quantity column's `gt "0.25"`; M3 compile 1,000 operands with units, and `ui::parseMeasure` of 1,000 typed texts; M4 `distinctValues` on the measure column against a decimal column. Write the figures into table.md "Benchmarks: values with units". + +- [ ] **Step 13: Docs build; commit** `feat(table): values with units`. + +--- + +### Task 8d: Column layout — widths, pinning and the fitting rule + +**Spec:** "Columns: order, visibility, pinning, width" (the `ColumnLayout` paragraph and "A layout survives a changed table"); "Configuring a table" (the `layout` value's shape); "Column groups" (a user-formed group in the layout, fitted with it); "Display" (`"units"` in the layout); invariant 26 (engine half); Tests row "Column layout" (engine half; gate D8 is Task 10's). + +**Files:** +- Modify: `include/morph/table/engine.hpp` (`ColumnLayoutValue`, `ColumnLayout`) +- Modify: `tests/table/test_table_columns.cpp` (new cases), `tests/table/test_table_engine.cpp` (the existing `rebuild` expectation) + +**Interfaces:** +- Consumes: `ColumnGroupInfo` (Task 1's, in `description.hpp`; 8b moves it unchanged to `data_source.hpp`, which `description.hpp` includes, so `engine.hpp` includes `description.hpp` if 8d lands first and `data_source.hpp` after 8b). +- Produces: + ```cpp + /// The stored shape of a layout: {"order", "hidden", "pinned", "widths", "groups", "units"}. + struct ColumnLayoutValue { + std::vector order; + std::vector hidden; // in display order + std::size_t pinned = 0; + std::map widths; // column id -> width in units, > 0 + std::vector groups; // user-formed groups + std::map> units; // column -> dimension -> unit id + [[nodiscard]] bool operator==(ColumnLayoutValue const&) const = default; + }; + class ColumnLayout { + public: + explicit ColumnLayout(std::vector ids, std::vector declaredHidden = {}); + // existing: order(), visible(), setVisible(), move(), reset() + bool setWidth(std::string const& columnId, std::optional width); // false: unknown id or width <= 0 + [[nodiscard]] std::map const& widths() const noexcept; + void setPinned(std::size_t n); // clamped to visible().size() + [[nodiscard]] std::size_t pinned() const noexcept; // the first n of visible() + [[nodiscard]] ColumnLayoutValue value() const; + void apply(ColumnLayoutValue const& value); // the fitting rule + void rebuild(std::vector ids, std::vector declaredHidden = {}); + }; + ``` + +- [ ] **Step 1: Write the failing tests** in `test_table_columns.cpp` (header mutation lines: "`apply` ignoring `hidden`: `a stored layout is fitted to the table` fails"; "`rebuild` appending a new column last: `a new column takes its declared place` fails"): + +```cpp +TEST_CASE("column layout: a stored layout is fitted to the table", "[table][layout]") { + ColumnLayout layout{{"id", "name", "a", "new", "b"}}; + layout.apply(ColumnLayoutValue{ + .order = {"b", "ghost", "name", "id", "a"}, .hidden = {"ghost", "a"}, .pinned = 9, + .widths = {{"name", 24}, {"ghost", 10}}, + .groups = {{.id = "g1", .label = "Company", .columns = {"a", "ghost", "b"}}, + {.id = "g2", .label = "X", .columns = {"ghost", "name"}}}, + .units = {{"a", {{"mass", "g"}}}, {"ghost", {{"mass", "kg"}}}}}); + auto const v = layout.value(); + CHECK(v.order == std::vector{"b", "name", "id", "a", "new"}); // "new" after its declared predecessor "a" + CHECK(v.hidden == std::vector{"a"}); + CHECK(layout.visible() == std::vector{"b", "name", "id", "new"}); + CHECK(v.pinned == 4); + CHECK(v.widths == std::map{{"name", 24}}); + REQUIRE(v.groups.size() == 1); + CHECK(v.groups[0].columns == std::vector{"a", "b"}); + CHECK(v.units == std::map>{{"a", {{"mass", "g"}}}}); +} + +TEST_CASE("column layout: every column hidden shows the first declared column", "[table][layout]") { + ColumnLayout layout{{"id", "name"}}; + layout.apply(ColumnLayoutValue{.order = {"name", "id"}, .hidden = {"name", "id"}}); + CHECK(layout.visible() == std::vector{"id"}); +} + +TEST_CASE("column layout: a new column takes its declared place", "[table][layout]") { + ColumnLayout layout{{"new0", "new1", "id", "name", "new2"}}; + layout.apply(ColumnLayoutValue{.order = {"name", "id"}}); + CHECK(std::vector(layout.order().begin(), layout.order().end()) == + std::vector{"new0", "new1", "name", "new2", "id"}); // new0 first (no placed predecessor), + // new1 after new0, new2 after name + ColumnLayout grown{{"a", "b", "c"}}; + grown.move("c", 0); + grown.rebuild({"a", "b", "x", "c"}); + CHECK(std::vector(grown.order().begin(), grown.order().end()) == + std::vector{"c", "a", "b", "x"}); +} + +TEST_CASE("column layout: widths and pinning", "[table][layout]") { + ColumnLayout layout{{"a", "b", "c"}, {"c"}}; + CHECK(layout.visible() == std::vector{"a", "b"}); // declared hidden + CHECK(layout.setWidth("a", 30)); + CHECK_FALSE(layout.setWidth("zzz", 30)); + CHECK_FALSE(layout.setWidth("b", 0)); + CHECK(layout.widths() == std::map{{"a", 30}}); + CHECK(layout.setWidth("a", std::nullopt)); + CHECK(layout.widths().empty()); + layout.setPinned(5); + CHECK(layout.pinned() == 2); + CHECK(layout.setVisible("b", false)); + CHECK(layout.pinned() == 1); // clamped as visibility changes + layout.setWidth("a", 12); + layout.reset(); + CHECK(layout.visible() == std::vector{"a", "b"}); + CHECK(layout.widths().empty()); + CHECK(layout.pinned() == 0); +} + +TEST_CASE("column layout: the stored shape round-trips as JSON", "[table][layout]") { + std::string json; + REQUIRE_FALSE(glz::write_json(fitted.value(), json)); + ColumnLayoutValue back; + REQUIRE_FALSE(glz::read_json(back, json)); + CHECK(back == fitted.value()); + CHECK_FALSE(glz::read_json(back, std::string{R"({"order":["a"],"hidden":[],"pinned":1,"widths":{"a":24}})"})); // groups, units absent +} + +TEST_CASE("column layout: a stored layout that repeats itself", "[table][layout]") { // Review Focus 5 + ColumnLayout layout{{"a", "b", "c"}}; + layout.apply(ColumnLayoutValue{.order = {"b", "a", "b", "c"}, .pinned = 40, + .widths = {{"a", 0}, {"b", -5}, {"c", 8}}, + .groups = {{.id = "g", .label = "G", .columns = {"a", "a", "c"}}}}); + CHECK(layout.value().order == std::vector{"b", "a", "c"}); + CHECK(layout.value().widths == std::map{{"c", 8}}); + CHECK(layout.pinned() == 3); + CHECK(layout.value().groups[0].columns == std::vector{"a", "c"}); +} +``` + +- [ ] **Step 2: Update** `"table: a column layout keeps one column visible and resets"` in `test_table_engine.cpp`: after `rebuild({"b", "c", "d"})` the order is `{"c", "d", "b"}` (d after its declared predecessor c) and `visible()` is `{"c", "d"}`. + +- [ ] **Step 3: Run; the new cases fail to compile** (`ColumnLayoutValue` undeclared), and the updated engine case fails on the order line. + +- [ ] **Step 4: Implement** in `engine.hpp`. `glz::meta` is not needed for `ColumnLayoutValue` (reflection), and reading leaves an absent member at its default. One function fits every input; `apply(value)` runs it with the current declared ids, `rebuild(ids, declaredHidden)` replaces the declared ids and declared hidden set and runs it on `value()`, `reset()` sets order to the declared ids, hidden to the declared hidden set, clears widths and `pinned`, and keeps `groups` and `units`: + +```text +fit(value, D = declared ids): + order = ids of value.order that are in D, first occurrence only + for d in D, in declared order, if d not in order: + p = nearest id before d in D that is in order + insert d right after p, or at the front when there is none + hidden = ids of value.hidden in D, listed in `order`'s order (an unmentioned column is shown) + widths = entries of value.widths with id in D and width > 0 + groups = each group with its columns filtered to D, duplicates removed; groups with < 2 columns dropped + units = entries of value.units with column id in D + if D is not empty and no column of order is visible: show D[0] + pinned = min(value.pinned, number of visible columns) +``` + + `setVisible` and `rebuild` re-clamp `pinned`; `pinned()` counts the first visible columns of `order`. + +- [ ] **Step 5: Run** `ctest --test-dir build/cl-debug -R "column layout" --output-on-failure`; expected: all pass, the two existing engine layout cases included. + +- [ ] **Step 6: Mutations.** (a) `fit` taking `hidden` as empty: `"a stored layout is fitted to the table"` fails. (b) the new-column loop appending at the end: `"a new column takes its declared place"` fails. (c) `pinned` not clamped: `"widths and pinning"` fails at `pinned() == 2`. Paste each; revert. + +- [ ] **Step 7: Docs build; commit** `feat(table): widths, pinning and fitting in the column layout`. + +--- + +## Self-review + +- **Spec coverage.** Value-set operator (8a Steps 1, 5, 6); its limits and wire bound D6 (8a Step 2); distinct values — which rows, which values, unloaded, order and bound, dates, levels (the column is read for every candidate, through `apply`'s key building), client mode, server mode, validation, caching (8a Steps 3, 7, 8); column groups — declaring, user-formed groups (fitted in 8d), the filter and its JSON, both truth tables, `Together`, excludes, the pooled values list, `x-group` (8b); values with units — registry, lineage, tombstones, unit-id rule, ratio, wire, typed rows, order, filters, `parseMeasure`, values list (8c); display units per dimension are stored by 8d and drawn by Task 15; the layout (8d). The menu halves, D7, D8 and D9 are Tasks 10 and 12, as the roadmap assigns. Neighbouring specs for step 3: `../forms/views.md` (8b Step 8); `engine.md`'s fitting-rule row is step 4, not here. +- **Type consistency.** `FilterCounters` is introduced in 8a and used by 8b; `ValueSet` in 8a, extended to measures in 8c; `DistinctValue` gains `dimension`/`unit` in 8c only; `ColumnGroupInfo` moves to `data_source.hpp` in 8b and is consumed by 8d; `GroupFilter`'s member order keeps the existing designated initializers valid. +- **Proportion.** Code blocks are tests and signatures; the only bodies are the group evaluation, the measure comparison and the fitting rule, which the spec fixes and the tests alone would not determine. diff --git a/docs/superpowers/plans/2026-11-17-one-table-d-screen.md b/docs/superpowers/plans/2026-11-17-one-table-d-screen.md new file mode 100644 index 000000000..ef396b4c5 --- /dev/null +++ b/docs/superpowers/plans/2026-11-17-one-table-d-screen.md @@ -0,0 +1,925 @@ +# The One Table, Part D: the Table on a Screen — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Put every table of a screen on the one pipeline: `TableRows` with its two implementations, the model-backed `table` node lowered onto a `PageWindow`, persisted state loaded before the first rows, the column window in the mount and both backends, and `rows` tables ordered by the engine. + +**Architecture:** `include/morph/ui/table_rows.hpp` holds `TableRows`, `EngineRows`, `WindowRows` and `ValueRowsSource`; `include/morph/table/state_store.hpp` holds the store interface and the state document's reader, writer and fitting; `include/morph/ui/table_state.hpp` and `table_state_model.hpp` hold the preferences-backed store and its server model. The interpreter lowers a table node onto a `TableRows` and hands it to the mount through a new `ui::Table::source`; the mount realizes rows by view index and, from Task 10, columns through `ColumnExtents`, cell pools and a per-frame budget. The interpreter's own sort and page are removed in Task 11. + +**Tech Stack:** C++23, CMake presets `cl-debug` (core, `morph_tests`, `morph_table_tests`, `morph_net_tests`) and `cl-qt-debug` with `-DMORPH_BUILD_QT_QUICK=ON -DMORPH_BUILD_TUI=ON` (for `morph_qt_quick_tests`, `morph_tui_tests`), Catch2, glaze, Qt 6 Quick offscreen. + +**Spec:** `docs/spec/table/table.md`: "The pipeline", "Sources", "A document's rows as a source", "The client table model", "The renderer" (all subsections), "Every surface on the pipeline" → "The document `table` node on the engine", "What the interpreter does for a table", "The status line" (counts), "Persisted state" (interface, "The framework's stores", load before the first rows, saving sort and filters), "Where sort and filter run"; the matching rows of "Tests" and "Invariants" (16 to 21, 26 to 28, 31); gates U1–U8, K5, K8, F1–F10, T1, T3–T5, D8, W6, B1; "Neighbouring specs" rows for steps 4, 5 and 6; "Delivery order" steps 4, 5, 6. Roadmap: `docs/superpowers/plans/2026-11-17-one-table.md` (Tasks 9, 10, 11). + +Part D of 7; index: 2026-11-17-one-table.md + +## Global Constraints + +- Everything in the roadmap's Global Constraints and its standard task loop applies; this plan lists only what is specific to its tasks. +- Limits, verbatim: `Virtualization{threshold = 2000, overscan = 10, columnThreshold = 40, cellThreshold = 20'000, columnOverscan = 4, realizePerFrame = 240}`; `SessionOptions::tableStateTimeout = 300 ms`, `tableStateDebounce = 500 ms`, `tableSettle = 80 ms`, `tableAwaitedRefetch = 250 ms`, `maxRangeRows = 10'000`; `TableStateOptions::segmentBytes = 12 KiB`; a `save` part and a `load` head are at most 1 MiB, an envelope 8 MiB; a key member is 1 to 256 bytes; `PageWindow(pageSize = 200, maxPages = 8, maxKeys = 1000, maxColumns = 96)`; a status-line message lasts 6 s or until the next gesture on the table; client mode is bounded at 20,000 rows or 1,000,000 cells. +- Comments and specs state what the code does now and why; no history, no issue numbers (AGENTS.md). +- Count gates are ordinary tests. Timings carry the tag `[.benchmark]`, never gate CI, and are written into table.md's Performance tables with machine and load. +- No GUI application is launched. Qt Quick behaviour is tested only through the offscreen `morph_qt_quick_tests` (`QT_QPA_PLATFORM=offscreen`). +- Neutral domain only: test models and documents use `Lims_SampleModel`, table id `samples`. +- Qt Quick and terminal suites need their options; configure once with + `cmake --preset cl-qt-debug -DMORPH_BUILD_QT_QUICK=ON -DMORPH_BUILD_TUI=ON` and build/run there: + `cmake --build --preset cl-qt-debug --target morph_qt_quick_tests morph_tui_tests`, + `ctest --test-dir build/cl-qt-debug -L qt_quick -R "" --output-on-failure` (the `qt_quick` label sets `QT_QPA_PLATFORM=offscreen`), `ctest --test-dir build/cl-qt-debug -R "" --output-on-failure` for `morph_tui_tests`. + +## Interfaces assumed from Parts A–C + +The roadmap names these; it does not pin their exact signatures. This plan uses the spec's signatures where the spec gives them and the shapes below where it does not. An implementer finding a different shape in the landed code uses the landed one and fixes this list. + +| From | Used as | +|---|---| +| Task 0, 2 | `bench::WideTableOptions`, `bench::WideTableSource`; `bench::WideTableModel` (registered type `bench::kWideModelType`, `"WideTableModel"`, set up with `WideTableModel::configure(WideTableOptions wide, WideTableOptions small)`), so the fixture's records are `WideTableModel.wide` and `WideTableModel.small`; the library `morph_wide_table_model` linked into `morph_tests`, `morph_net_tests` and `morph_qt_quick_tests`; B5's browser harness `tests/qt_quick/wasm/` run with `emrun`. | +| Task 1 | `TableQuery::columns` (`std::optional>`, absent: every column; cross-part decision 2), `table::TableColumn`, `table::TableDescription` (with `capabilities`, as the spec's block has it), `table::GenericRow`, `table::GenericPage`; `table::readGenericPage(std::string_view, TableDescription const&) -> std::expected`; glaze metadata for `TableDescription` so a reply's JSON reads with `glz::read_json`. | +| Task 2 | Action ids `Table.` and `Table..describe`; the record `.` (here `Lims_SampleModel.samples`); its catalog entry marks `Table.` opaque (`ActionRecordInfo::opaque`). | +| Task 4 | `TableQuery::keys` (`std::optional>`), `table::UnloadedLevels{std::span levels; std::span failed;}` (views into the window: copy before the window changes); `GenericPage::{unloadedRows, undecided, awaiting, failed, indices}`. | +| Task 5 | `table::rowIdOf(glz::raw_json const&) -> std::optional` (and its inverse `changeKeys(std::span) -> std::vector`, not used here). `effects::RecordChange::keys` absent is a whole-type change. | +| Task 7 | `Session::read` and `Session::watchRecord` and `PageFetch`, `RowsFetch`, `Refresh`, `RefreshPlan` and the `PageWindow` members exactly as table.md "The window under changes" and "The client's caches" declare them; `PageWindow(pageSize, maxPages, maxKeys, maxColumns)` (no key function), `bool accept(PageFetch const&, GenericPage)` (`false`: ignored), `std::vector patch(RowsFetch const&, GenericPage)`, and the accessors of cross-part decision 8: `RowId const* key(std::int64_t)`, `Cell const* cell(std::int64_t, std::string_view column)`, `bool holds(std::int64_t, std::string_view column)`, `std::uint64_t revision()`; plus `unloadedLevels(index)`, `undecided()`, `awaiting()`, `failed()`, `total()`, `totalKind()`, `error()`, `pending()`. `WindowRows` reads rows only through these. | +| Task 8d | `ColumnLayout::value() -> table::ColumnLayoutValue{order, hidden, pinned, widths, groups, units}` and `apply(ColumnLayoutValue const&)` (cross-part decision 7); its JSON is the spec's `layout` shape. | +| Task 15 | `ui::formatCell(table::Cell const&, table::TableColumn const&, ui::FormatContext const&, DisplayUnits const& = {}) -> std::string` (cross-part decision 9), which replaces this plan's stand-in. | + +## Review Focus + +1. **A description that changes revision while state is held** (roadmap item 1). A sort key, a filter and the fetched columns naming column `c` when a page reply carries a new `revision` whose description lacks `c`: the client refetches the description once, drops the key and the filter (reported once each), and the next query names neither `c` nor its level. A view selection and a cell draft do not exist before Tasks 17a and 24; those tasks extend this test. Test: Task 9d. +2. **The table unmounted while its state load or a fetch is in flight.** A screen closed before a held load completes, then the load released: nothing is applied, no query is sent, no handler runs on a destroyed table, and the debounced save of a changed state goes out at once on unmount. Test: Task 9d. +3. **A stored document that is not JSON, or of a version the framework does not know.** The table starts from its defaults, the problem is reported once at the table's path, nothing is written until the user changes something, and then the framework's version overwrites it. Test: Task 9b. +4. **An empty model-backed view.** Before the first reply, and while rows are undecided, the node's `empty` content is not shown and the status line says the table is loading or counts "0 so far, N loading"; after a reply with `total: 0, totalKind: exact, undecided: 0` it is shown and the count reads "0 rows". Test: Task 9d. +5. **Rows with a duplicate key in a `rows` table.** Three rows with keys 1, 2, 1: the table shows two rows, the first occurrence of key 1 in the engine's order, the duplicate is reported as the mount reports it today (`site::kDuplicateKey`, once per duplicate), and a sort never brings the second row back. Test: Task 11a. + +--- + +## File map + +| File | Task | Holds | +|---|---|---| +| `include/morph/ui/table_rows.hpp` (new) | 9a, 11a | `SortChain`, `TableTotal`, `TableRowRef`, `TableCountsView`, `TableRows`, `EngineRows`, `WindowRows`, `WindowRowsOptions`, `WindowRowsStats`, `cellValue`, `countsText`, the stand-in cell text; `ValueRowsSource` (11a) | +| `include/morph/table/state_store.hpp` (new) | 9b | `TableStateKey`, `StoredTableState`, `SaveOutcome`, `SaveResult`, `ITableStateStore`, `MemoryTableStateStore`, `StateDocument`, `readState`, `writeState`, `fitState`, `FitReport` | +| `include/morph/ui/table_state.hpp` (new) | 9c | `PreferencesTableStateStore`, `TableStates` (the session's kept states, load batching, save debounce) | +| `include/morph/ui/table_state_model.hpp` (new) | 9c | `TableStateModel` (`morph.tableState`), its request types, `TableStateOptions`, `registerTableState` | +| `include/morph/ui/view.hpp` | 9d, 10a | `Table::source`, `Table::status`, `TableStatus`; `Virtualization` additions, `TableColumn::id`, `Table::arrangement` | +| `include/morph/ui/mount.hpp` | 9d, 10a | rows by view index from `TableRows`; `ColumnExtents`, `windowingOf`, the column window, cell pools, the frame budget, focus by key and column | +| `include/morph/ui/backend.hpp` | 9d, 10a | `TableWidget::setStatus`, `setColumnWindow`, `setOnHorizontalRange`, `setOnSelectRange`, `setFocusCell`, `setCounts` | +| `include/morph/ui/testing/recording_backend.hpp` | 9d, 10a | records the new table calls; `scrollTableColumns`, `selectTableRange` | +| `include/morph/ui/interpreter.hpp`, `src/ui/interpreter.cpp` | 9d, 11b | `SessionOptions` members; the lowering of both table kinds onto `TableRows`; `RtScope::onMissingRowField` | +| `src/qt_quick/table.cpp`, `src/qt_quick/qml/MorphTable.qml` | 10b | the column window, pinned columns, pools, the status line | +| `src/tui/table_widget.cpp`, `.hpp` | 10b | the column range, the status line | +| `tests/ui_table_support.hpp` (new) | 9a | the test catalog, description and page builders, `GatedTableStateStore` | +| `tests/test_ui_table_model.cpp` (new), `tests/test_ui_table_state.cpp` (new) | 9 to 11 | in `morph_tests` | +| `tests/table/test_table_state.cpp` (new) | 9b | in `morph_table_tests` | + +--- + +### Task 9: `TableRows` and the model-backed table on a screen (table.md step 4) + +Four commits, each its own reviewer gate. + +#### 9a — `TableRows`, `EngineRows` and `WindowRows` + +**Files:** +- Create: `include/morph/ui/table_rows.hpp`, `tests/ui_table_support.hpp`, `tests/test_ui_table_model.cpp` +- Modify: `tests/CMakeLists.txt` (add `test_ui_table_model.cpp`, and in 9c `test_ui_table_state.cpp`, to `add_executable(morph_tests …)`), `include/morph/ui/interpreter.hpp` (`SessionOptions` members below) + +**Interfaces:** +- Consumes: Task 7's `PageWindow`, `Session::read`, `Session::watchRecord`; Task 1's description and generic page; Task 5's `rowIdOf`; `table::Engine` (existing: `setSort`, `setFilter`, `viewRowCount`, `rowIdAt`, `sourceRowOf`, `onViewChange`, `pending`). +- Produces (namespace `morph::ui`, every member as table.md "The client table model" declares it): + +```cpp +using SortChain = std::vector; +struct TableTotal; // rows, kind, known +struct TableRowRef; // key, row +struct TableCountsView; // view, source, undecided, pending, awaiting, failed +class TableRows; // total, row, unloadedLevels, visible, keysOf, setSort, setFilter, counts, onChange + +/// A row's key and its record, owned (cross-part decision 28). +struct RowRecord { table::RowId key; Value row; }; +/// What EngineRows reads a row's key and record through: its source's snapshot and a source row. +using RecordOf = std::function; +class EngineRows final : public TableRows { +public: + EngineRows(std::shared_ptr source, RecordOf recordOf, table::EngineOptions options); + [[nodiscard]] table::Engine& engine() noexcept; + [[nodiscard]] std::size_t recordsBuilt() const noexcept; // calls of recordOf, for the memo's test +}; + +struct WindowRowsOptions { + std::shared_ptr channel; // the model's channel + std::string table; // the table id: actions "Table.
", "Table.
.describe" + std::string record; // ".
", watched for the window's life + Value body; // the node's `body`, the query's fixed part + std::int64_t pageSize = 200; + std::size_t maxPages = 8; + std::size_t maxKeys = 1000; + std::size_t maxColumns = 96; + bool restoring = false; // sends no query until `release` +}; +struct WindowRowsStats { + std::size_t pageFetches = 0; // page and column fetches sent + std::size_t rowsFetches = 0; // keyed fetches sent + std::size_t cancelled = 0; + std::size_t cellsConverted = 0; // cells turned into a row record's fields (K8) +}; +class WindowRows final : public TableRows { +public: + WindowRows(Session& session, WindowRowsOptions options); + void setDescription(table::TableDescription description); // fits sort, filters and columns to it + [[nodiscard]] table::TableDescription const* description() const noexcept; + void onStaleDescription(std::function handler); // a reply named another revision + void release(); // leaves the restoring state: the first query + void retry(); // the query again with `TableQuery::retry` + [[nodiscard]] std::optional const& error() const noexcept; + [[nodiscard]] WindowRowsStats const& stats() const noexcept; +}; + +/// A cell as the document reads it. +[[nodiscard]] Value cellValue(table::Cell const& cell, table::ColumnInfo const& column); +/// The status line's counts part. +[[nodiscard]] std::string countsText(TableCountsView const& counts, std::int64_t selected, FormatContext const& format); +namespace detail { +/// The text of a cell until `ui::formatCell` lands (Task 15 replaces every call). +[[nodiscard]] std::string standInCellText(table::Cell const& cell, table::TableColumn const& column); +} +``` + +- `SessionOptions` gains `std::chrono::milliseconds tableSettle{80}`, `std::chrono::milliseconds tableAwaitedRefetch{250}` (this commit) and, in 9c, `std::shared_ptr tableState`, `std::chrono::milliseconds tableStateTimeout{300}`, `std::chrono::milliseconds tableStateDebounce{500}`, `std::string application` (the `application` of every table-state key). + +Decisions the signatures do not carry: + +- `EngineRows` memoises `RecordOf`'s result per source row of the engine's `viewSnapshot()`; the memo is dropped for the rows an `Updated` names and whole on a new snapshot of `Reset`. `row(index)` returns a `TableRowRef` whose references point into the memo and hold until the next `onChange`. +- Decoding replies: `Session::read` resolves with a `Value`. `WindowRows` decodes a page as `table::readGenericPage(ui::detail::canonicalJson(reply), *description())` and a description as `glz::read_json(ui::detail::canonicalJson(value))`. This writes the reply back to text once before parsing it; the cost is part of F1 and is recorded beside W2's decode. A `Value` overload of `readGenericPage` that skips the text is a later optimisation and is not asked of Part A here. +- `cellValue`: empty and unloaded → `null`; `int64` in `Integer`/`Key` → `Int`; in `Date` → the `String` `YYYY-MM-DD`; in `DateTime` → `Timestamp` of seconds × 1000 ms; `Rational` → `Decimal{rational}` (its places kept); `double` → `Decimal` parsed by `table::parseDecimal` from `std::to_chars`'s shortest text, a non-finite one `null`; text → `String`; `bool` → `Bool`; `QuantityCell` → `Decimal` of `amount × toCanonical`; a `Measure` (once Task 8c lands) → `Quantity{amount, unit id}`. +- `standInCellText` is `table::displayText(cell)`, and the empty string for an unloaded cell. +- `countsText`, numbers grouped with `format.numeric`'s separator: nothing known (`!view.known`) → `""`; `undecided > 0` → `" so far, loading"`; `selected > 0` → `" of selected"`, `` followed by `" or more"` under `AtLeast`; `AtLeast` → `" or more"`; otherwise `"1 row"` or `" rows"`. +- `WindowRows` row records: `row(index)` returns a record holding one field per column of the latest `visible(…, columns)` set for which `PageWindow::holds(index, column)`, each read with `cell(index, column)` and its key with `key(index)`, memoised per key until the row's cells change; `cellsConverted` counts each conversion. A record of a 300-column table of which twenty are named converts twenty cells. +- `WindowRows` timing: a fetch for a page adjacent to a held page goes at once; any other waits until `visible` has stayed within one page for `tableSettle`, measured on `SessionOptions::scheduler`; every `superseded()` fetch is cancelled through its stop token and handed to `cancelled`. While `awaiting()` is not empty, the first refetch after an event goes at once and the next no sooner than `tableAwaitedRefetch` after it. +- `WindowRows` order of work: `watchRecord` first; the first page fetch only after it resolves, after `setDescription`, and (with `restoring`) after `release`. Its handler maps a change with `keys` to `refresh` of each key read by `rowIdOf` (a key that does not read invalidates), and a change without `keys` to `invalidate()`; after every `accept` it sends `owed()` and calls `visible` again. +- A reply whose `revision` differs from the held description's calls the `onStaleDescription` handler once per revision and is still accepted. + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_table_model.cpp`, over `Harness` and the helpers of `tests/ui_table_support.hpp`: `tableCatalog()` (the corpus catalog plus `Table.samples` and `Table.samples.describe` on `Lims_SampleModel`), `samplesDescription(int columns)` (`id` Integer key, `name` Text, then `c3…` Decimal), `describeJson(TableDescription)`, `pageJson(offset, count, total, columns, TotalKind kind = Exact, std::int64_t undecided = 0, std::vector awaiting = {})` whose row keys are `offset + i` and names `"row-"`. + +```cpp +// Mutation: send a fetch per `visible` call (K6 fails: 40 page calls). +TEST_CASE("window rows: a scroll that passes sends no fetch until it rests", "[ui][table]") { + Harness h; WindowRows rows = h.windowRows("samples"); // helper: options, description, release + h.answerFirstPage(rows, /*total*/ 100'000); + std::size_t const before = h.dispatcher.count("Table.samples"); + for (std::int64_t at = 0; at < 100'000; at += 2'500) { rows.visible(at, 30, h.columns); h.advance(10ms); } + CHECK(h.dispatcher.count("Table.samples") == before); // inside tableSettle each time + h.advance(80ms); + CHECK(h.dispatcher.count("Table.samples") <= before + 3); // K6: at most three sent + CHECK(rows.stats().cancelled + 3 >= rows.stats().pageFetches - 1); + rows.visible(99'990, 30, h.columns); // the next page from a held one: at once +} +// Mutation: a refetch per event (ten fetches). +TEST_CASE("window rows: an awaited order is throttled", "[ui][table]") { + // first page replies with awaiting = {"recipe"}; ten keyed events within 100 ms + CHECK(fetchesAfterEvents == 1); // one at once + h.advance(250ms); CHECK(fetchesAfterEvents == 2); // one after tableAwaitedRefetch +} +// Mutation: build the record from every column of the page. +TEST_CASE("window rows: a row of 300 columns converts the twenty it is asked for", "[ui][table]") { + // description of 300 columns; visible(0, 50, first twenty ids); reply holds those twenty + // the window first holds 40 columns (an earlier visible() named c1…c40 and the reply carried them), then + // visible(0, 50, the first twenty ids): the record converts only the twenty named, not the forty held + REQUIRE(rows.row(0).has_value()); + CHECK(rows.stats().cellsConverted == 20); // K8; the mutation converts 40 + static_cast(rows.row(0)); CHECK(rows.stats().cellsConverted == 20); // memoised +} +// Mutation: send the first page before `release`. +TEST_CASE("window rows: a restoring window sends no query until released", "[ui][table]") { + CHECK(h.dispatcher.count("Table.samples") == 0); + rows.setSort({{.column = "name", .dir = table::SortDirection::Descending}}); + rows.release(); + CHECK(h.dispatcher.count("Table.samples") == 1); // T3: one query, carrying the sort + CHECK(h.dispatcher.calls.back().body.contains(R"("column":"name")")); +} +// Mutation: countsText ignoring `undecided`. +TEST_CASE("status line: counts for exact, lower-bound and undecided totals", "[ui][table]") { + using enum table::TotalKind; + FormatContext const en = formatFor("en"); // ui_table_support.hpp: numeric facts from ui::localeFacts(tag); + // a default FormatContext has an empty groupSeparator + CHECK(countsText({.view = {9093, Exact, true}}, 0, en) == "9,093 rows"); + CHECK(countsText({.view = {100000, AtLeast, true}}, 0, en) == "100,000 or more"); + CHECK(countsText({.view = {12, AtLeast, true}, .undecided = 40}, 0, en) == "12 so far, 40 loading"); + CHECK(countsText({.view = {9093, Exact, true}}, 3, en) == "3 of 9,093 selected"); + CHECK(countsText({.view = {1, Exact, true}}, 0, en) == "1 row"); + CHECK(countsText({}, 0, en).empty()); +} +// Mutation: EngineRows::keysOf completing on a later turn. +TEST_CASE("engine rows: every row is present, visible does nothing, keysOf completes at once", "[ui][table]") {} +// Mutation: no memo (recordsBuilt grows by the view size on every read). +TEST_CASE("engine rows: a row's record is built once until the row changes", "[ui][table]") { + static_cast(rows.row(0)); static_cast(rows.row(0)); CHECK(rows.recordsBuilt() == 1); + patchRow(source, 0); settle(); static_cast(rows.row(0)); CHECK(rows.recordsBuilt() == 2); +} +``` + +- [ ] **Step 2: Run to verify they fail** +Run: `cmake --build --preset cl-debug --target morph_tests` → FAIL: `ui/table_rows.hpp` not found. + +- [ ] **Step 3: Implement** the interfaces above in `include/morph/ui/table_rows.hpp` and the `SessionOptions` members in `interpreter.hpp`, with complete Doxygen. + +- [ ] **Step 4: Run to verify they pass** +Run: `ctest --test-dir build/cl-debug -R "window rows|engine rows|status line" --output-on-failure` → all pass; then the whole `morph_tests` target green. + +- [ ] **Step 5: Mutations** (each named above): apply, see the named case fail, revert; paste each failing line into the commit body. + +- [ ] **Step 6: Commit** `feat(ui): table rows over the engine and over a page window`. + +#### 9b — The store interface and the state document + +Needs Task 8c: `fitState` reads `TableDescription::units` directly. + +**Files:** +- Create: `include/morph/table/state_store.hpp`, `tests/table/test_table_state.cpp` +- Modify: `tests/CMakeLists.txt` (`table/test_table_state.cpp` in `morph_table_tests`), `tests/table/bench_table.cpp` (T1) + +**Interfaces:** +- Consumes: `table::ColumnLayoutValue` and `ColumnLayout::apply` (8d), `table::TableDescription`, `table::FilterSpec`, `table::compileFilter` (existing compile path in `filter.hpp`). +- Produces (namespace `morph::table`): `TableStateKey`, `StoredTableState`, `SaveOutcome`, `SaveResult`, `ITableStateStore` exactly as table.md "Persisted state" declares them (`shares()` included, default `false`), and: + +```cpp +class MemoryTableStateStore final : public ITableStateStore { +public: + explicit MemoryTableStateStore(exec::IExecutor& executor); // completions delivered there + // revisions are "1", "2", … per key +}; + +inline constexpr int kStateVersion = 1; +struct StateDocument { // part "state" + int version = kStateVersion; + std::optional layout; + std::optional sort; + std::optional filters; + std::optional view; + CarriedState carried; // what the caller's description lacks, written back as it was +}; +/// Parts of a stored state that name a column or group the caller's description lacks. Not applied; merged back +/// by `writeState` into the places they had. +struct CarriedState { + struct Column { + std::string id; + std::string after; // the id it followed in the stored `order`; empty: it was first + bool hidden = false; + std::optional width; + std::map units; // dimension -> unit id, as stored + }; + std::vector columns; // in stored order + std::vector> sort; // each key with its index in the stored chain + std::map filters; // by the absent column's id + std::vector groupFilters; // naming an absent group + std::vector groups; // user groups that name an absent column + [[nodiscard]] bool operator==(CarriedState const&) const = default; +}; +struct FitDrop { + enum class What : std::uint8_t { Column, SortKey, Filter, GroupFilter, Group, Unit }; + What what; + std::string id; // the column, group or "column/dimension" of a unit + bool carried = false; // kept in `carried` (the description lacks it) or gone + [[nodiscard]] bool operator==(FitDrop const&) const = default; +}; +struct FitReport { std::vector dropped; }; // in the order of the fitting list +[[nodiscard]] std::expected readState(std::string_view json); // migrates forward +[[nodiscard]] std::string writeState(StateDocument const& document); +[[nodiscard]] StateDocument fitState(StateDocument document, TableDescription const& description, FitReport& report); +``` + +Decisions: +- `readState` refuses text that is not JSON and a `version` above `kStateVersion` with a message; a lower version is migrated one version at a time (none exist yet, so the migration table is empty and its loop is tested by a version-0 fixture added when a version 2 exists — not now). +- `fitState` applies table.md's fitting list in order: columns as `ColumnLayout::apply` fits them; sort keys on missing or unsortable columns dropped; filters on missing columns or groups, filters that do not compile for the column's kind, and units not in `description.units` dropped; user groups left with fewer than two columns dropped with their filter. A drop because the description lacks the column or group moves the entry into `carried`; any other drop is gone. A user group that names an absent column is carried whole (it is whole again when the column returns); one left with fewer than two present columns for any other reason is gone with its filter. +- `writeState` merges `carried` back: a column into `order` right after its `after` id, or first when `after` is empty, or at the end when `after` is itself absent from the written order; its hidden flag, width and units with it; a sort key at its stored index, clamped to the chain's length; filters, group filters and groups appended. Reading that text with the same description carries the same entries again. +- The lowering reports, once each at the table's path, the drops the spec says are reported: `Filter`, `GroupFilter` and `Unit`. `Column`, `SortKey` and `Group` drops are not reported. + +- [ ] **Step 1: Write the failing tests** + +```cpp +// Mutation: fitState returning the document unchanged. +TEST_CASE("table state: stale state is fitted", "[table][state]") { + // id, name, weight (a Measure column of dimension "mass"; units lists g and kg), declared group g1 = {name, alias} + TableDescription const d = describedSamples(); + StateDocument const stored = readState(R"({"version":1, + "layout":{"order":["id","gone","name","weight"],"hidden":["id","name","weight"],"pinned":2,"widths":{"gone":9}, + "units":{"weight":{"mass":"zz"}}}, + "sort":[{"column":"gone","direction":"asc"},{"column":"name","direction":"desc"}], + "filters":{"columns":{"gone":{"include":[{"eq":"x"}]},"weight":{"include":[{"gt":"abc"}]}}, + "groups":[{"group":"g2","columns":["a","b"]}]}})").value(); + FitReport report; + StateDocument const fitted = fitState(stored, d, report); + CHECK_FALSE(fitted.layout->order.empty()); + CHECK(fitted.layout->hidden.size() < 3); // one column at least is visible + CHECK(*fitted.sort == SortChain{{.column = "name", .dir = SortDirection::Descending}}); + CHECK(fitted.filters->columns.empty()); + CHECK(fitted.filters->groups.empty()); + CHECK_FALSE(fitted.layout->units.contains("weight")); // unknown unit "zz" dropped + using W = FitDrop::What; + CHECK(report.dropped == std::vector{ + {W::Column, "gone", true}, {W::SortKey, "gone", true}, {W::Filter, "gone", true}, + {W::Filter, "weight", false}, {W::GroupFilter, "g2", true}, {W::Unit, "weight/mass", false}}); + CHECK(fitted.carried.columns == std::vector{{.id = "gone", .after = "id", .width = 9}}); + StateDocument const again = readState(writeState(fitted)).value(); + CHECK(again.layout->order == std::vector{"id", "gone", "name", "weight"}); // its place kept + CHECK(again.sort->at(0).column == "gone"); // carried key back at index 0 + CHECK(again.filters->columns.contains("gone")); + CHECK_FALSE(again.filters->columns.contains("weight")); // did not compile: gone at the next save +} +// Review Focus 3. Mutation: readState accepting an unknown version. +TEST_CASE("table state: a document that is not JSON or of an unknown version is refused", "[table][state]") { + CHECK_FALSE(readState("{not json").has_value()); + CHECK_FALSE(readState(R"({"version":2,"sort":[]})").has_value()); + CHECK(readState(R"({"version":1})").has_value()); +} +// Mutation: save ignoring `expected`. +TEST_CASE("memory table state store: a save with a stale revision is a conflict", "[table][state]") { + // save "a" → Saved rev "1"; save "b" expected "1" → Saved "2"; save "c" expected "1" → Conflict, revision "2" +} +``` +T1 benchmark: `"table state benchmark: parse, fit and apply 300 columns, 8 keys, 32 filters"` tagged `[.benchmark][table][state]`; it asserts first that every one of the 8 keys and 32 filters survives fitting. + +- [ ] **Step 2: Run to verify they fail** — `cmake --build --preset cl-debug --target morph_table_tests` → FAIL: `state_store.hpp` not found. +- [ ] **Step 3: Implement** `state_store.hpp` as above. +- [ ] **Step 4: Run to verify they pass** — `ctest --test-dir build/cl-debug -R "table state|memory table state" --output-on-failure`; `morph_table_tests` green. +- [ ] **Step 5: Mutations**, as named. +- [ ] **Step 6: Commit** `feat(table): the table state store and the state document`. + +#### 9c — The preferences-backed store and the session's kept states + +**Files:** +- Create: `include/morph/ui/table_state.hpp`, `include/morph/ui/table_state_model.hpp`, `tests/test_ui_table_state.cpp` +- Modify: `include/morph/ui/interpreter.hpp` (`SessionOptions` members listed in 9a; `Session::tableStates() -> TableStates*`, null without a store), `tests/net/test_push_tables.cpp` ("The person is the store's") + +**Interfaces:** +- Consumes: `IPreferencesStore`, `PrefKey` (existing), `session::current()` as `PreferencesModel` reads it, `morph::session::detail::sha256`, `Dispatcher`/`ModelChannel` (existing), 9b's interface. +- Produces: + +```cpp +// table_state_model.hpp, namespace morph::ui +inline constexpr std::string_view kTableStateModel = "morph.tableState"; +struct TableStateOptions { + std::size_t segmentBytes = std::size_t{12} * 1024; + std::size_t partBytes = std::size_t{1} << 20; // a save part and a load head + std::optional maxBytesPerPerson; // unset: no quota +}; +struct TableStateLoad { std::vector keys; }; +struct TableStateHead { bool found = false; std::string revision; std::uint64_t size = 0; std::string text; }; +struct TableStateLoaded { std::vector entries; }; // one per key, in order +struct TableStateRead { table::TableStateKey key; std::string revision; std::uint64_t offset = 0; }; +struct TableStateChunk { std::string revision; std::string text; }; // another revision: start again +struct TableStateSave { table::TableStateKey key; std::string expected; std::string upload; + std::uint64_t offset = 0; std::string text; bool last = true; }; +struct TableStateSaved { table::SaveResult result; }; +struct TableStateRemove { table::TableStateKey key; }; +struct TableStateRemoved{ bool removed = false; }; +class TableStateModel; // execute(TableStateLoad), execute(TableStateRead), execute(TableStateSave), execute(TableStateRemove) +void registerTableState(std::shared_ptr store, model::detail::ActionDispatcher& dispatcher, + model::detail::ModelRegistryFactory& registry, TableStateOptions options = {}); + +// table_state.hpp +class PreferencesTableStateStore final : public table::ITableStateStore { +public: + PreferencesTableStateStore(Dispatcher& dispatcher, exec::IExecutor& owner); + [[nodiscard]] bool shares() const override { return true; } +}; +enum class StateParts : std::uint8_t { Layout = 1, Sort = 2, Filters = 4, View = 8 }; // a bit set +class TableStates { // one per Session with a store +public: + TableStates(std::shared_ptr store, reactive::Scheduler* scheduler, + std::chrono::milliseconds debounce); + [[nodiscard]] std::optional kept(table::TableStateKey const& key) const; + async::Completion> load(table::TableStateKey const& key); + /// Marks @p parts of the key's state changed to those of @p document; a debounced save, one in flight per key. + void changed(table::TableStateKey const& key, table::StateDocument const& document, StateParts parts); + void flush(table::TableStateKey const& key); // unmount: save now + void onSaveFailed(std::function handler); +}; +``` + +Decisions: +- Server layout per table.md "The framework's stores": person from the verified session (none: `load` answers `found = false` for every key, `save` and `remove` are refused); `app` = the key's `application`; scope `morph.table.` + the first 32 lower-case hex digits of `sha256(model + "\x1F" + table + "\x1F" + qualifier)`; point `` holds `{"key": {…in clear…}, "revision": r, "document": …}` for a document of at most `segmentBytes`, else `{"key": …, "revision": r, "segments": n, "bytes": b}` with segments at `..`, each a JSON string of at most `segmentBytes`. A head whose clear key differs from the asked key reads as not found. A missing segment fails the load. +- Write order, with an intent entry because a preferences store cannot list its entries: (1) read `.pending`; if present, remove the segments it names (`..<0…segments−1>`) unless the head names that revision, then remove it; (2) for a segmented document write `.pending = {"revision": r, "segments": n}`; (3) the segments; (4) the head; (5) remove the previous revision's segments; (6) remove `.pending`. A save stopped anywhere before (4) leaves the old head and its segments readable, and the next save of the key removes the stray segments in step (1). +- Key bounds: `application`, `table`, `qualifier` and `part` are 1 to 256 bytes; `model` is 0 to 256 bytes, because a table over a document's rows has an empty `model` (table.md "Persisted state", the key's comment). This corrects table.md's "Bounds" bullet ("Each key member is 1 to 256 bytes"); this commit changes that bullet to say so. +- Revision: the first 16 hex digits of `sha256(previousRevision + document)`. +- Quota: with `maxBytesPerPerson`, the model keeps the person's total at `PrefKey{person, app = key.application, scope = "morph.table", point = "usage"}` as `{"bytes": n}`, `n` counting document bytes (not entries). A save that replaces a document of `b0` bytes (0 when none, read from the old head's `bytes` or its document's length) with one of `b1` bytes would make it `n − b0 + b1`; over the quota it answers `Refused`, reason `"over the table-state quota"`, and writes nothing; otherwise it writes the document, then `usage`. A remove makes it `max(0, n − b0)`. Without the option no `usage` entry is read or written. +- `PreferencesTableStateStore::load` calls made in one turn of `owner` go out as one `TableStateLoad`; a head shorter than `size` is completed with `TableStateRead` calls by offset, restarting when the revision changes. A `save` over `partBytes` is sent as parts under one upload id (`upload` = a random 16-hex id), committed by the part with `last`. +- `TableStates` keeps every document it loaded or saved, by key, for the session's life; `load` of a kept key resolves at once with no call. +- A change before the key's load settles is held, not saved: when the load completes (or fails, or is given up on at the table's timeout and later completes) the save writes the loaded document with only the changed parts replaced (`writeState` over the loaded `StateDocument` with `layout`, `sort`, `filters` or `view` taken from the change per `parts`). A load that failed saves the change over an empty document. So a sort changed while a slow load is held never overwrites the stored filters. + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_table_state.cpp` (over a local `RemoteServer`-free loop: `registerTableState` on a `MemoryPreferencesStore`, a principal stamped as `PreferencesModel`'s tests do, and a `Dispatcher` over that dispatcher): + +```cpp +// Mutation: the store calling the preferences model, which refuses the value over maxValueBytes. +TEST_CASE("table state: state of any size", "[ui][state]") { + std::string const big = stateWithOneOfValues(20'000); // a 20,000-value oneOf, well over 1 MiB + REQUIRE(save(store, key, big).outcome == SaveOutcome::Saved); + CHECK(load(store, key)->json == big); + CHECK(prefs.largestValue() <= 12 * 1024 + kSegmentEnvelopeBytes); // every entry within segmentBytes + prefs.failSetsAfter(prefs.setCount() + segmentsOf(big2)); // stop after the segments of a second save + CHECK_FALSE(trySave(store, key, big2)); + CHECK(load(store, key)->json == big); // the previous state still reads + CHECK(prefs.has(pendingPoint(key))); // the stopped save left its intent + prefs.healSets(); + REQUIRE(save(store, key, big3).outcome == SaveOutcome::Saved); // the next save cleans up first + CHECK_FALSE(prefs.has(pendingPoint(key))); + CHECK(prefs.pointsUnder(scopeOf(key)) == headAndSegmentsOf(store, key)); // no stray segment of big2 or big + prefs.erase(segmentPoint(key, 3)); + CHECK_THROWS(load(store, key)); // a missing segment is a failed load +} +// Mutation: the quota checked against the new size alone (a replace of a large state is refused). +TEST_CASE("table state: a save over the quota is refused and the previous state still reads", "[ui][state]") { + // registerTableState with maxBytesPerPerson = 10'000 + REQUIRE(save(store, keyA, docOf(6'000)).outcome == SaveOutcome::Saved); + REQUIRE(save(store, keyA, docOf(9'000)).outcome == SaveOutcome::Saved); // a replace: 9,000 in use + auto const over = save(store, keyB, docOf(2'000)); + CHECK(over.outcome == SaveOutcome::Refused); CHECK(over.reason == "over the table-state quota"); + CHECK_FALSE(load(store, keyB)); + CHECK(load(store, keyA)->json == docOf(9'000)); + CHECK(remove(store, keyA)); + CHECK(save(store, keyB, docOf(2'000)).outcome == SaveOutcome::Saved); // 9,000 freed +} +// Review: a save during a held load. Mutation: saving the change as a whole new document at once. +TEST_CASE("table state: a change during a held load keeps the stored parts", "[ui][state]") { + // gated store holding {"version":1,"filters":{"columns":{"name":{"include":[{"contains":"a"}]}}}} + states.changed(key, docWithSort("name", Descending), StateParts::Sort); h.advance(600ms); + CHECK(gated.saves() == 0); // held until the load settles + gated.release(); h.advance(600ms); + REQUIRE(gated.saves() == 1); + CHECK(gated.lastSaved().contains(R"("contains":"a")")); // stored filter carried + CHECK(gated.lastSaved().contains(R"("column":"name")")); +} +// Mutation: each table's load sent on its own (five calls). Gate T5. +TEST_CASE("table state: five tables of a screen load in one call, and a second mount in none", "[ui][state]") { + CHECK(dispatcher.count("load") == 1); + remountScreen(); + CHECK(dispatcher.count("load") == 1); +} +// Mutation: no debounce (fifty saves). Gate T4. +TEST_CASE("table state: fifty changes within a second are one save of the last", "[ui][state]") { + REQUIRE(loaded(states, key)); + for (int i = 0; i < 50; ++i) { states.changed(key, docWithSortOn(i), StateParts::Sort); h.advance(10ms); } + h.advance(500ms); + CHECK(store.saves() == 1); CHECK(readState(store.lastSaved())->sort == docWithSortOn(49).sort); +} +``` +In `tests/net/test_push_tables.cpp`: `"table state: the person is the store's"` — two principals over one socket server save different `state` documents under one `TableStateKey` and each loads its own; a `TableStateLoad` body carrying a `"person"` member is read with that member ignored (glaze `error_on_unknown_keys = false`), so no request names another person. Mutation: the model keying by a body member. + +- [ ] **Step 2: Run to verify they fail** — `cmake --build --preset cl-debug --target morph_tests` → FAIL: `table_state.hpp` not found. `morph_net_tests` is POSIX-only (cross-part decision 18): build and run its case on `linux-everything` (`cmake --preset linux-everything && cmake --build --preset linux-everything --target morph_net_tests`). +- [ ] **Step 3: Implement** the interfaces above. +- [ ] **Step 4: Run to verify they pass** — `ctest --test-dir build/cl-debug -R "table state" --output-on-failure` and `morph_tests` green; `ctest --test-dir build/linux-everything -R "table state" --output-on-failure` for the `morph_net_tests` case. +- [ ] **Step 5: Mutations**, as named. +- [ ] **Step 6: Commit** `feat(ui): table state in the application's preferences store`. + +#### 9d — The model-backed `table` node on the engine + +**Files:** +- Modify: `include/morph/ui/view.hpp` (`TableSource`, `TableStatus`, `Table::source`, `Table::status`), `include/morph/ui/mount.hpp` (`mountKind(Table)` reads `Table::source` when set), `include/morph/ui/backend.hpp` (`TableWidget::setStatus`), `include/morph/ui/testing/recording_backend.hpp` (records `status`), `include/morph/ui/interpreter.hpp` and `src/ui/interpreter.cpp` (the lowering; `RtScope::onMissingRowField`; element values), `tests/test_ui_table_model.cpp`, `tests/qt_quick/test_qt_quick_table.cpp` (F6, F7, F1) +- Docs: `docs/spec/ui/interpreter.md` ("Not yet lowered" loses the model-backed `table` entry; "Lowering" describes the lowering onto `TableRows`), `docs/spec/ui/document.md` (the `table` kind: `model` with `table` or `action`, `body`, `status`, `persist`, `empty` on a model-backed table, element values `viewCount`, `viewCountKind`, `undecided`, `awaiting`, `failed`, `pending`), `docs/spec/ui/preferences.md` (table state has its own model, `morph.tableState`, not bounded by `maxValueBytes`; the `columns` and `saved` points read and write the table's state), `docs/spec/table/engine.md` ("Not here yet": the model-backed table entry goes). + +**Interfaces:** +- Consumes: 9a, 9b, 9c; Task 7's query-cache rules (the description is an entry of `Table..describe`). +- Produces: + +```cpp +// view.hpp +class TableRows; +struct TableSource { + std::shared_ptr rows; + /// A realized row's cells, built once per realized row from a signal the mount keeps equal to its record. + std::function const& row)> rowView; +}; +struct TableStatus { + std::string counts; // the leading part + std::string state; // a failed level or fetch; empty when none + bool retry = false; // the state offers "Retry" + std::string message; // the latest refusal or notice; empty when none + bool shown = false; // whether the line takes space + [[nodiscard]] bool operator==(TableStatus const&) const = default; +}; +// struct Table gains: +std::optional source; // when set, `rows` is ignored and rows come by view index +Prop status; +std::function onRetry; + +// backend.hpp, TableWidget gains (default: does nothing) +virtual void setStatus(TableStatus const& status); +virtual void setOnRetry(std::function onRetry); + +// interpreter.hpp, RtScope gains +/// Called with a field a row expression read that the row's record does not hold; set on a server-mode row scope. +std::function const* onMissingRowField = nullptr; +``` + +Decisions: +- The mount over a `TableSource`: rows are always windowed (a server-mode table is); `setRealizedWindow(first, count, total)` with `total = rows->total().rows`; a realized index whose `row()` is empty mounts a placeholder row of the estimated height and becomes a keyed row when it arrives; `onChange` applies the `ViewChange` to the keyed realized rows, so a row keeps its widget, native state and selection when it moves; the visible range and the columns from `TableSource`'s owner go to `rows->visible`. +- The lowering: `model` with `table` binds the model's channel as any query does, makes a document query on `Table.
.describe` (one per mounted table, W6) and a `WindowRows` with `restoring` set when the session has a store and `persist` is not `false`. Columns: the description's columns with `shown`, overridden per id by the node's `columns` parts; a node with `columns` lays out those. The columns named to `visible`: the laid-out columns, plus the columns `cell`, `tone`, `tooltip`, `rowTone`, `rowLabel` and `onActivate` read as `row.` (found when the document loads), plus every column when any of them reads the row otherwise (`row[name]`, the row passed whole), reported once as a warning naming the node. +- `RtScope::onMissingRowField` on a server-mode row scope adds the field to the set named to `visible` (one fetch), and the read is `null` meanwhile, as the spec says it reads as absent. +- `pageSize`/`page` on a model-backed table are reported once and ignored; `body` is the query's fixed part; header sorts and `filters` go to `setSort`/`setFilter`. +- State: key `{application = SessionOptions::application, model = , table =
, qualifier = persist string or the screen's id, part = "state"}`. The load starts at mount; when it completes within `tableStateTimeout` the fitted sort and filters are applied (bound `sort` or `filters` are the document's and are neither applied nor saved; a literal `sort` is the default a stored one replaces) and `release()` is called; past the timeout `release()` is called with the defaults and a later state is applied (one more query) unless the user scrolled, selected or changed the arrangement meanwhile. A failed load is reported once through the session and changes still save. The stored layout part is carried unchanged in this task: nothing here changes a layout. +- `status`: a model-backed table's `shown` is always true; the counts are `countsText(counts(), selectedCount, formatting)`; a fetch error fills `state`; a failed level fills `state` with `" data could not be loaded."` and `retry = true`, whose handler calls `WindowRows::retry`. `status: false` routes `message` to the session's notifications. +- `empty` on a model-backed table shows only when `total().known`, `total().rows == 0`, `kind == Exact` and `counts().undecided == 0`. + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_table_model.cpp` (document: `{"vocab":"ui/1","id":"samples","view":{"kind":"table","id":"list","model":"Lims_SampleModel","table":"samples"}}`; `Harness(std::shared_ptr)` is a new constructor that sets `SessionOptions::tableState` and `application = "lims"`; `GatedTableStateStore` holds each load until `release()` or `fail()`): + +```cpp +// Mutation: a lowering that needs the node's `columns`. +TEST_CASE("model table: a client given only the catalog draws its columns, sorts, filters and pages it", "[ui][table]") { + // describe reply: 3 columns; first page of 200 of total 9093 + CHECK(h.headerLabels() == std::vector{"Id", "Name", "C3"}); + CHECK(h.dispatcher.count("Table.samples.describe") == 1); // W6: one request + h.clickHeader("Name"); CHECK(lastQuery(h).sort.at(0).column == "name"); +} +// Mutation: a table that ignores its store sends the default query. Gate T3. +TEST_CASE("model table: persisted state before the first query", "[ui][table][state]") { + auto store = std::make_shared(h.owner, R"({"version":1,"sort":[{"column":"name","direction":"desc"}], + "filters":{"columns":{"name":{"include":[{"contains":"a"}]}}}})"); + auto screen = h.mount(kModelTable); h.replyDescribe(); + CHECK(h.dispatcher.count("Table.samples") == 0); + store->release(); h.settle(); + REQUIRE(h.dispatcher.count("Table.samples") == 1); + CHECK(lastQuery(h).sort == table::SortChain{{.column = "name", .dir = table::SortDirection::Descending}}); + CHECK(lastQuery(h).filters.columns.contains("name")); +} +// Mutation: no debounce; unmount not flushing. +TEST_CASE("model table: saves are coalesced, and unmounting saves at once", "[ui][table][state]") { + for (int i = 0; i < 50; ++i) { h.clickHeader(i % 2 ? "Name" : "Id"); h.advance(5ms); } + h.advance(500ms); CHECK(store->saves() == 1); + h.clickHeader("Name"); screen.reset(); h.settle(); CHECK(store->saves() == 2); // at once, no debounce + // a change during a save in flight is one more save after it +} +// Mutation: applying a late state after the user scrolled. +TEST_CASE("model table: a slow or failed load", "[ui][table][state]") { + h.advance(300ms); CHECK(h.dispatcher.count("Table.samples") == 1); // defaults after tableStateTimeout + store->release(); h.settle(); CHECK(h.dispatcher.count("Table.samples") == 2); // applied: one more query + // second case: scrolled before the late state → not applied, no second query + // third case: fail() → one diagnostic, defaults, a later sort still saves +} +// Mutation: applying a stored sort over a bound one. +TEST_CASE("model table: bound values win", "[ui][table][state]") { + // `sort` bound to state: stored sort not applied, not saved; stored filters applied; `persist: false` → no load, no save +} +// Mutation: the lowering refetching every held row's cells after a page refetch. Gate K5. +TEST_CASE("model table: a refetch that changed 20 of 200 rows evaluates 20 rows again", "[ui][table]") { + CHECK(rowEvaluations(h) == 20); +} +// Review Focus 1. Mutation: setDescription not dropping keys and filters on a removed column. +TEST_CASE("table rows: a new description revision drops what names a removed column", "[ui][table]") { + // description rev A with columns id, name, c; sort on c, filter on c, visible columns {id, name, c} + // a page reply names revision B → exactly one more describe call + // describe reply B lacks c + CHECK(h.dispatcher.count("Table.samples.describe") == 2); + auto const q = lastQuery(h); + CHECK(std::ranges::none_of(q.sort, [](auto const& k) { return k.column == "c"; })); + CHECK_FALSE(q.filters.columns.contains("c")); + CHECK(std::ranges::find(*q.columns, "c") == q.columns->end()); + CHECK(diagnosticsMentioning(h, "c") == 2); // the key and the filter, once each +} +// Review Focus 2. Mutation: the load completion capturing the table by reference with no liveness check. +TEST_CASE("model table: unmounted while its state load is held", "[ui][table][state]") { + h.clickHeader("Name"); // nothing loaded yet: a change marks the state unsaved + screen.reset(); store->release(); h.settle(); + CHECK(h.dispatcher.count("Table.samples") == 0); + CHECK(store->saves() == 1); // flushed at unmount +} +// Review Focus 4. Mutation: `empty` shown while `total().known` is false. +TEST_CASE("model table: empty waits for the first reply and for undecided rows", "[ui][table]") { + CHECK_FALSE(h.backend.find("Text", "text", "No samples")); // before any reply + h.replyPage(pageJson(0, 0, 0, cols, TotalKind::AtLeast, /*undecided*/ 40)); + CHECK_FALSE(h.backend.find("Text", "text", "No samples")); + CHECK(statusOf(h).counts == "0 so far, 40 loading"); + h.replyPage(pageJson(0, 0, 0, cols)); + CHECK(h.backend.find("Text", "text", "No samples")); + CHECK(statusOf(h).counts == "0 rows"); +} +// Mutation: a status line built without `failed`. +TEST_CASE("model table: a failed level offers Retry, which sends `retry`", "[ui][table]") { + CHECK(statusOf(h).retry); h.backend.retryTable(tableId); CHECK(lastQuery(h).retry); +} +// Mutation: a refetch per event. Gate E3 on a screen. +TEST_CASE("model table: an arrival is one keyed request", "[ui][table]") { + // after the first page, a RecordChange with 20 keys → exactly one more Table.samples call, naming `keys` +} +``` + +In `tests/qt_quick/test_qt_quick_table.cpp`, over the fixture's `wide` model through a scripted dispatcher (the description and pages written from `bench::WideTableSource` at 100,000 × 300, the node declaring twenty columns): + +```cpp +// Mutation: rebind every cell of a changed row (F6 counts 50 × 20 × rows' columns). Gate F6. +TEST_CASE("qt_quick model table: an arrival for 50 rows re-evaluates their 1,000 cells and no other", "[qt_quick]") { + CHECK(cellEvaluations == 1000); +} +// Mutation: widen a column on every cell report. Gate F7. +TEST_CASE("qt_quick model table: each column grows at most once on an arrival", "[qt_quick]") { + for (int column = 0; column < 20; ++column) CHECK(widthChanges(column) <= 1); +} +``` +`cellEvaluations` reads a `cellBinds` counter the mount exposes through `MountStats` (a new `[[nodiscard]] MountStats const& Mounted::stats() const`, `MountStats{std::size_t cellBinds; std::size_t cellsCreated; std::size_t cellsRebound;}`, created here, used again by Task 10). F1 is a `[.benchmark]` case in the same file: from the page reply to the first rendered frame, recorded. + +- [ ] **Step 2: Run to verify they fail** — `cmake --build --preset cl-debug --target morph_tests` → FAIL: no `Table::source`; the model table cases report the node as not lowered. +- [ ] **Step 3: Implement** the interfaces and decisions above; remove the "Not yet lowered" entry for a model-backed `table` from the interpreter. +- [ ] **Step 4: Run to verify they pass** — `ctest --test-dir build/cl-debug -R "model table|table rows" --output-on-failure`; `morph_tests` green; then `cmake --build --preset cl-qt-debug --target morph_qt_quick_tests` and `ctest --test-dir build/cl-qt-debug -L qt_quick -R "model table" --output-on-failure` green. +- [ ] **Step 5: Mutations**, as named. Record F1 (and U6's F1–F7/E1–E3 figures, measured with `morph_qt_quick_tests "[.benchmark]"` on `cl-qt-release`) in table.md's Performance tables. +- [ ] **Step 6: Neighbouring specs and docs build.** Update the four specs listed under Docs. Run the roadmap's docs build: green. +- [ ] **Step 7: Commit** `feat(ui): the model-backed table on a screen, with stored sort and filters`. + +--- + +### Task 10: The column window (table.md step 5) + +Two commits and an exit criterion. + +#### 10a — The column window in the mount + +**Files:** +- Modify: `include/morph/ui/view.hpp` (`Virtualization` additions; `TableColumn::id`; `Table::arrangement`), `include/morph/ui/mount.hpp`, `include/morph/ui/backend.hpp`, `include/morph/ui/testing/recording_backend.hpp`, `src/ui/interpreter.cpp` (range selection through `keysOf`; `maxRangeRows`), `include/morph/ui/interpreter.hpp` (`SessionOptions::maxRangeRows = 10'000`) +- Test: `tests/test_ui_mount.cpp`, `tests/test_ui_table_model.cpp` + +**Interfaces:** + +```cpp +// view.hpp +struct Virtualization { // existing members kept; added, verbatim from the spec: + std::size_t columnThreshold = 40; + std::size_t cellThreshold = 20'000; + std::size_t columnOverscan = 4; + std::size_t realizePerFrame = 240; +}; +// TableColumn gains: std::string id; // the column's id; empty for a table built from C++ with no ids +struct TableArrangement { // which columns show, in which order, how many pinned, user widths + std::vector visible; // indices into Table::columns, in layout order; empty: every column + std::size_t pinned = 0; + std::map widths; + [[nodiscard]] bool operator==(TableArrangement const&) const = default; +}; +// Table gains: Prop arrangement; + +// mount.hpp +struct Windowing { bool rows = false; bool columns = false; }; +[[nodiscard]] Windowing windowingOf(Virtualization const& v, std::size_t rows, std::size_t visibleColumns, bool serverMode); +class ColumnExtents { +public: + void assign(std::span widths); // visible columns, layout order + bool grow(std::size_t column, int width); // only grows; false when unchanged + void set(std::size_t column, int width); // a width the user set + [[nodiscard]] std::pair columnsAt(int offset, int viewport) const; // [first, last) + [[nodiscard]] int offsetOf(std::size_t column) const; + [[nodiscard]] int widthOf(std::size_t column) const; + [[nodiscard]] int totalWidth() const noexcept; + [[nodiscard]] std::size_t columnReads() const noexcept; // F9 counter +}; +struct ColumnWindow { std::vector columns; int leading = 0; int trailing = 0; }; // pinned, then the range +[[nodiscard]] ColumnWindow columnWindow(ColumnExtents const& extents, std::size_t pinned, int scrollX, int viewport, + std::size_t overscan); +struct MountStats { std::size_t cellBinds = 0; std::size_t cellsCreated = 0; std::size_t cellsRebound = 0; + std::size_t placeholders = 0; }; // from 9d, gains the last member + +// backend.hpp, TableWidget gains (defaults do nothing; a backend that windows columns overrides them) +virtual void setColumnWindow(std::span columns, std::span offsets, + std::span widths, int totalWidth); +virtual void setOnHorizontalRange(std::function handler); +virtual void setHorizontalOffset(int scrollX); // anchoring +virtual void setOnSelectRange(std::function handler); +virtual void setFocusCell(std::size_t viewRow, std::size_t column); // column: its place in the layout +virtual void setOnFocusMove(std::function handler); +virtual void setCounts(std::optional rows, std::size_t columns); // nullopt: unknown (AtLeast) +enum class FocusMove : std::uint8_t { Up, Down, Left, Right, Home, End, First, Last, PageUp, PageDown }; +``` + +Decisions: +- `windowingOf`: rows when `serverMode` or `rows >= threshold` or `rows × visibleColumns > cellThreshold`; columns when `visibleColumns > columnThreshold` or `rows × visibleColumns > cellThreshold`. +- `ColumnExtents` keeps widths in a Fenwick tree (prefix sums in O(log n), `columnsAt` by binary descent on the tree), so a width change and a lookup read O(log n) nodes; `columnReads` counts per-column width reads only. +- Pools: cells of the default cell of one `TextRole` share a pool keyed by role; a column with its own `cell` template has a pool keyed by its column index. A cell leaving either window is detached and pooled; one entering takes a pooled widget and rebinds it (`cellsRebound`) or is created (`cellsCreated`). +- The frame: one pass of the windowing Effect realizes or rebinds at most `realizePerFrame` cells, nearest the viewport centre first; the rest are placeholders of their column's width (`placeholders`) and are realized in later turns posted with `_rt->owner().post`, one batch of `realizePerFrame` a turn. +- Anchoring: a growth of a column left of the first fully visible unpinned column calls `setHorizontalOffset(scrollX + growth)` in the same pass. The vertical rule is the same for rows above the first visible row. +- Pinned: the first `arrangement.pinned` visible columns are always in the window; while their widths exceed half the viewport, the last pinned columns scroll with the rest (display only; `pinned` is not changed). +- Focus is `(row key, column id)`, held by the mount; that row and column stay realized; a focused row that leaves the view moves focus to the row now at its view index, or the last row; a move onto an unfetched index focuses the placeholder (announced as loading) and the row takes focus when it arrives; `Last` with an `AtLeast` total goes to the last row known. +- `setCounts(rows, columns)`: `rows` the view's total when `Exact`, `nullopt` under `AtLeast`; `columns` the visible column count. +- Range selection on a `TableSource`: the backend reports two view rows; the lowering calls `keysOf(min, count)`; more than `maxRangeRows` rows is refused with the message `"A range selects at most 10,000 rows"` (number grouped by the locale) and selects nothing. `WindowRows::keysOf` fetches pages naming no columns (`columns: []`) for what it does not hold. +- A hidden column has no header, width or cells; hiding returns its realized cells to their pool; moving a column inside the window creates nothing (D8). + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_mount.cpp` (RecordingBackend gains `scrollTableColumns(id, scrollX, viewport)`, `focusMove(id, FocusMove)`, `selectTableRange(id, anchor, focus)`, and records `setColumnWindow`, `setCounts`, `setFocusCell`): + +```cpp +TEST_CASE("windowing: the three thresholds", "[ui][mount]") { + Virtualization const v{}; + CHECK(windowingOf(v, 1999, 300, false).rows); CHECK(windowingOf(v, 1999, 300, false).columns); + CHECK_FALSE(windowingOf(v, 1999, 10, false).rows); CHECK_FALSE(windowingOf(v, 1999, 10, false).columns); + CHECK(windowingOf(v, 2000, 10, false).rows); CHECK_FALSE(windowingOf(v, 2000, 10, false).columns); + CHECK(windowingOf(v, 100, 41, false).columns); CHECK_FALSE(windowingOf(v, 100, 41, false).rows); + CHECK(windowingOf(v, 10, 10, true).rows); +} +// Mutation: a column window that ignores cellThreshold. Gate F2 on the recording backend. +TEST_CASE("table window: 1,999 × 300 realizes only the windows", "[ui][mount]") { + // 30 rows and 20 columns in view, 2 pinned + CHECK(cellsAlive(h) <= realizedRows(h) * (2 + 20 + 2 * 4)); + CHECK(headerCellsAlive(h) == 2 + windowColumns(h)); +} +// Mutation: release a leaving cell without pooling it. Gate F8. +TEST_CASE("table window: a fling creates no cell once the pools are warm", "[ui][mount]") { + flingRows(h, 2000); auto const created = h.stats().cellsCreated; + flingRows(h, 2000); CHECK(h.stats().cellsCreated == created); + CHECK(maxRebindsPerTurn(h) <= 240); +} +// Mutation: no anchoring. +TEST_CASE("table window: a column growing left of the view keeps the columns in view where they are", "[ui][mount]") {} +// Mutation: unpin by changing `pinned`. +TEST_CASE("table window: pinned columns stay realized, and a narrow table unpins for display only", "[ui][mount]") {} +// Mutation: reading every column's width per pass. Gate F9. +TEST_CASE("table window: a pass reads no width outside the column window", "[ui][mount]") {} +// Mutation: rebuild rows on a layout change. Gate D8. +TEST_CASE("table window: hiding and moving a column create no cell widgets", "[ui][mount]") { + CHECK(h.stats().cellsCreated == before); CHECK(pooledAfterHide == realizedRows(h)); +} +// Mutation: focus held by widget, which recycling drops. Gate F10. +TEST_CASE("table focus: the focused cell survives a scroll of 10,000 rows and 200 columns", "[ui][mount]") { + CHECK(focusedCellRealized(h)); CHECK(h.backend.lastCounts(table) == std::pair{std::optional{100'000}, 300uz}); +} +TEST_CASE("table focus: Ctrl+End over rows not fetched lands on the last row when it arrives", "[ui][mount]") {} +TEST_CASE("table focus: the reported row count is the total, or unknown for a lower bound", "[ui][mount]") {} +``` +In `tests/test_ui_table_model.cpp`: `"model table: a range over rows not held resolves through pages that name no columns"` (fetches carry `"columns":[]`; the selection is the keys of view rows 500 to 1,499); `"model table: a range over maxRangeRows is refused and selects nothing"` (status message, selection unchanged). Mutation: `keysOf` fetching the visible columns. + +- [ ] **Step 2: Run to verify they fail** — `cmake --build --preset cl-debug --target morph_tests` → FAIL: `windowingOf`, `ColumnExtents` undeclared. +- [ ] **Step 3: Implement** the interfaces and decisions above. +- [ ] **Step 4: Run to verify they pass** — `ctest --test-dir build/cl-debug -R "windowing|table window|table focus|model table" --output-on-failure`; `morph_tests` green. +- [ ] **Step 5: Mutations**, as named. +- [ ] **Step 6: Commit** `feat(ui): the column window in the mount`. + +#### 10b — The column window in Qt Quick and the terminal + +**Files:** +- Modify: `src/qt_quick/table.cpp`, `src/qt_quick/qml/MorphTable.qml` (the column window, pinned columns, the header for windowed columns only, horizontal anchoring, focus cell, counts for assistive technology, the status line from `setStatus` with its "Retry" button), `src/tui/table_widget.cpp`, `src/tui/table_widget.hpp` (the column range, the status line) +- Test: `tests/qt_quick/test_qt_quick_table.cpp`, `tests/tui/test_tui_table.cpp` +- Docs: `docs/spec/qt_quick/backend.md` ("Containers and lists": the column window, cell pools, pinned columns, the status line; "Tests": the new cases). + +**Interfaces:** consumes 10a's `TableWidget` members; produces nothing new. + +Decisions: +- MorphTable exposes, for tests, the QML properties `realizedColumnFirst`, `realizedColumnCount`, `pinnedShown`, `headerCellCount`, `statusText`, and keeps `rowScans`, `realizedFirst`, `realizedCount`, `totalRows`. +- The status line is one `Text` row under the rows, an accessible polite live region (`Accessible.role: Accessible.StaticText`, announced on change), with a "Retry" button while `TableStatus::retry`; it takes no space while `shown` is false. +- The terminal table draws the cells of the columns in the window only, and the status line as its last line. + +- [ ] **Step 1: Write the failing tests** + +```cpp +// tests/qt_quick/test_qt_quick_table.cpp, a model-backed table of the fixture's `wide` at 100,000 × 300, 1,280 × 800. +// Mutation: MorphTable instantiating a cell per column of a realized row. Gate F2. +TEST_CASE("qt_quick table window: cells and header cells alive are the windows'", "[qt_quick]") { + CHECK(cellItems(scene) <= realizedRows * (pinned + inView + 2 * 4)); + CHECK(tableItem.property("headerCellCount").toInt() == pinned + windowColumns); +} +// Mutation: a pool keyed by column for default cells. Gate F8. +TEST_CASE("qt_quick table window: flings create no cell once the pools are warm", "[qt_quick]") {} +// Gate F10. +TEST_CASE("qt_quick table window: focus and counts after scrolling far away", "[qt_quick]") {} +// Gate D8, Qt Quick half. +TEST_CASE("qt_quick table window: hiding and moving a column create no cell items", "[qt_quick]") {} +TEST_CASE("qt_quick table: the status line shows counts and Retry", "[qt_quick]") { + CHECK(tableItem.property("statusText").toString() == QStringLiteral("100,000 rows")); +} +// tests/tui/test_tui_table.cpp. Mutation: the terminal table ignoring the column range. Gate U7. +TEST_CASE("tui table: cell widgets alive are rows in view times columns in view", "[tui]") {} +``` +Benchmarks, `[.benchmark][qt_quick]`, in the same file: F3 (vertical at 20 rows/s, a fling at 2,000 rows/s, a thumb drag), F4 (horizontal at 10 and 100 columns/s), F5 (memory per realized row), each first asserting the table is windowed both ways, timed per step through `QQuickRenderControl` (`polishItems`, `sync`, `render`) under `offscreen` and the software scene graph. + +- [ ] **Step 2: Run to verify they fail** — `cmake --build --preset cl-qt-debug --target morph_qt_quick_tests morph_tui_tests` then `ctest --test-dir build/cl-qt-debug -L qt_quick -R "table window" --output-on-failure` → FAIL on `headerCellCount` being invalid. +- [ ] **Step 3: Implement** the backends' halves. +- [ ] **Step 4: Run to verify they pass** — `ctest --test-dir build/cl-qt-debug -L qt_quick --output-on-failure` and `ctest --test-dir build/cl-qt-debug -R "tui table" --output-on-failure`: all green, the existing virtualized-table cases included. +- [ ] **Step 5: Mutations**, as named. Record F1 to F5 from `build/cl-qt-release` (`cmake --preset cl-qt-release -DMORPH_BUILD_QT_QUICK=ON`, then `morph_qt_quick_tests "[.benchmark]"` with `QT_QPA_PLATFORM=offscreen`) into table.md's "The frontend at 300 columns" table, against step 0's baseline. +- [ ] **Step 6: Neighbouring spec** `docs/spec/qt_quick/backend.md` as listed; docs build green. +- [ ] **Step 7: Commit** `feat(qt_quick): the column window, cell pools and the status line`. + +#### 10c — Exit criterion U8 + +U8 is "U1 and U2 at 20,000 rows, and U6's scrolls, in the WebAssembly build in a browser … the frame budgets of F3 and F4 hold". What can be done without a person at a browser: + +- [ ] **Build**: the Qt WebAssembly build of `morph_qt_quick` already builds in `.github/workflows/wasm-demo.yml` (Qt `wasm_singlethread`, emsdk 4.0.7, `qt-cmake … -DMORPH_BUILD_QT=ON -DMORPH_BUILD_QT_QUICK=ON`). Confirm this branch still builds there (`cmake --build build-wasm --target morph_examples_client`) and that the count gates of the single-threaded path still run under Node in the WebAssembly job Task 0 adds. +- [ ] **Measure with Task 0's B5 harness.** Part A adds `tests/qt_quick/wasm/` (built only when `EMSCRIPTEN AND MORPH_BUILD_QT_QUICK`), run in a headless browser with `emrun --browser chrome --browser_args=--headless=new`, printing figures to the console. Add `tests/qt_quick/wasm/u8_scrolls.cpp` beside `wide_rows_baseline.cpp`: it mounts a 20,000-row `rows` table (U1: a header click on a text column; U2: one row changed) and a model-backed table over an in-process `bench::WideTableModel` at 100,000 × 300, scrolls both as F3 and F4 do, and prints U1, U2 and each scroll's p50/p99 frame time. Build it in the WebAssembly tree configured as `wasm-demo.yml` does (`qt-cmake … -DMORPH_BUILD_QT=ON -DMORPH_BUILD_QT_QUICK=ON -DMORPH_BUILD_TESTS=ON`), run it with the same `emrun` line, and record the figures beside the native ones. The exit criterion holds when every scroll's p50 ≤ 8 ms and p99 ≤ 16.6 ms. +- [ ] **What stays open.** The run is local: no CI job drives a browser, and on a machine without Chrome it cannot run. If it cannot be run, Task 10 lands with its native gates and the executor writes "U8 not measured" in the commit body and tells the user. No other command is invented for it. + +--- + +### Task 11: Tables over `rows` on the engine (table.md step 6) + +Two commits. + +#### 11a — `ValueRowsSource` + +**Files:** +- Modify: `include/morph/ui/table_rows.hpp` +- Test: `tests/test_ui_table_model.cpp`, `tests/bench_interpreter.cpp` (U1 to U4) + +**Interfaces:** + +```cpp +/// The expression language's text order: the default collator's fold, then the text itself. +class ValueRowsCollator final : public table::TextCollator { +public: + [[nodiscard]] std::string sortKey(std::string_view text) const override; // fold(text) + '\0' + text + [[nodiscard]] std::string fold(std::string_view text) const override; // CodePointCollator's + [[nodiscard]] std::size_t folds() const noexcept; // U3 counter +}; +struct ValueColumn { + std::string id; + std::optional kind; // the column part's literal `kind`; empty: inferred + std::function sortBy; // the column's `sortBy` in the row's scope, tracked; empty: the field +}; +class ValueRowsSource final : public table::DataSource { +public: + ValueRowsSource(reactive::Runtime& runtime, std::vector columns, + std::function keyOf); + void assign(ValueList rows); // a new list: match by key, then by cell + void use(std::string_view column); // a sort, a filter or a values list began + void unuse(std::string_view column); // its last use ended: its cells go + [[nodiscard]] RecordOf recordOf() const; // for EngineRows + [[nodiscard]] std::size_t cellsHeld() const noexcept; // U4 counter + [[nodiscard]] std::size_t cellsDerived() const noexcept; // U2 counter +}; +``` + +Decisions: +- A cell of a column in use is a `reactive::Computed` over the row's signal and what `sortBy` reads; when one changes, the source notifies `Updated` for that row alone; a change every row's cell reads derives the column again and notifies `Reset`. +- `assign`: a row whose `Value` shares storage with the held one is kept unread; another object derives the in-use cells and compares them with the held cells, `Updated` only on a difference; new keys `Inserted`, missing keys `Removed`. A later duplicate of a key is dropped and reported through `site::kDuplicateKey`, once per duplicate, as the mount reports it today. +- Kind inference over non-null cells: all `Int` → `Integer`; `Int` and `Decimal` → `Decimal`; any `double`-sourced number → `Number`; all `Bool` → `Bool`; all `String` → `Text`; any other mix takes the first non-null cell's kind, and a cell not of it is invalid (sorts last, reported once at the `sort` path); a change of the inferred kind is a `Reset` of that column. `null` is an empty cell. +- The source keeps each row's `Value` beside its cells; `recordOf` returns its key and that value. + +- [ ] **Step 1: Write the failing tests** + +```cpp +// Mutation: a diff by row value (every field compared). +TEST_CASE("value rows source: a document's rows as a source", "[ui][table]") { + src.use("name"); src.assign(rows()); listener.clear(); + src.assign(freshDecodeOf(rows())); CHECK(listener.changes.empty()); + src.assign(withName(rows(), 7, "zeta")); CHECK(listener.changes == std::vector{updated({7})}); + sortByReadsChange(src, 3); CHECK(listener.changes.back() == updated({3})); + CHECK(src.cellsHeld() == rows().size()); // one column in use; "notes" holds no cells + // integers with decimals are one Decimal column; a text among them is one invalid cell, reported once +} +// Review Focus 5. Mutation: keep the last duplicate. +TEST_CASE("value rows source: a duplicate key keeps its first row", "[ui][table]") { + src.assign(list({row(1, "a"), row(2, "b"), row(1, "c")})); + CHECK(src.snapshot()->rowCount() == 2); + CHECK(duplicateReports(h) == 1); + engine.setSort({{.column = "name", .dir = table::SortDirection::Descending}}); + CHECK(namesInView(engine) == std::vector{"b", "a"}); +} +// Mutation: fold inside the comparison. Gate U3. +TEST_CASE("value rows source: a sort of 100,000 rows folds each text once", "[ui][table]") { + CHECK(collator->folds() == 100'000); +} +``` +Benchmarks `[.benchmark][interpreter]` in `bench_interpreter.cpp`: U1 (a header click on a text column of 100,000 rows), U2 (one row changed, shared and fresh decode; gates inside: `EngineStats::repairs == 1`, `fullSorts` unchanged, `cellsDerived` equal to one column's rows for the fresh decode), U4 (memory of 10,000 × 300 with one column sorted; gate `cellsHeld() == 10'000`). + +- [ ] **Step 2: Run to verify they fail** — `cmake --build --preset cl-debug --target morph_tests` → FAIL: `ValueRowsSource` undeclared. +- [ ] **Step 3: Implement** as above. +- [ ] **Step 4: Run to verify they pass** — `ctest --test-dir build/cl-debug -R "value rows source" --output-on-failure`. +- [ ] **Step 5: Mutations**, as named. +- [ ] **Step 6: Commit** `feat(ui): a document's rows as a table source`. + +#### 11b — The interpreter's tables on the engine + +**Files:** +- Modify: `include/morph/ui/interpreter.hpp` (`TableView` stays as the shape element values read; `ElementState::tableRows` (`std::shared_ptr`) replaces `sortBound`'s use for ordering), `src/ui/interpreter.cpp` (`sortedRows` and the ordering half of `tableView` removed; the lowering of a `rows` table onto `ValueRowsSource` → `Engine` → `EngineRows`; `pageSize`/`page` as a slice of the engine's view; `group` as a hidden leading sort key `ValueRowsSource` computes, headings placed after paging; `rowSpans` on the page's rows, drawn only unsorted and below the windowing thresholds; select-all through `table::Selection::selectAll`) +- Test: `tests/test_ui_table_model.cpp`, `tests/test_ui_table.cpp`, `tests/bench_interpreter.cpp` (B1) +- Docs: `docs/spec/ui/interpreter.md` ("Lowering", **The rows' pipeline** and **Groups** describe the engine's order and what the lowering does around it), `docs/spec/ui/document.md` (a column's `kind`; the client-mode bound warning), `docs/spec/table/engine.md` ("Not here yet": the single-threaded steps entry). + +**Interfaces:** +- Consumes: 11a; `table::Engine` with `EngineOptions{owner = &runtime.owner(), worker = nullptr, frameBudget = 4 ms, services.collator = ValueRowsCollator}`. +- Produces: nothing new in the API; every existing table element value keeps its name and shape. + +Decisions: +- A plain list over 20,000 rows or 1,000,000 cells (rows × the node's columns) is reported once as a warning naming the table: `"table holds rows; a table over rows is ordered on the client up to 20,000 rows or 1,000,000 cells — serve it as a model's table"`. +- A table with no `id` gets no state key: a `persist` other than `false` on it is refused when the document loads. + +- [ ] **Step 1: Write the failing tests** + +```cpp +// Mutation: the interpreter's pipeline left in place for `rows` tables. +TEST_CASE("one order on every surface: a rows table and a model table order alike", "[ui][table]") { + // the same 200 rows (texts with accents and mixed case, decimals, nulls) as `rows` and as pages of `samples` + // the model-table side's pages are written by `table::apply` with ValueRowsCollator as the served collator + for (auto const& chain : {byName, byNameDesc, byWeightThenName}) + CHECK(rowsTableOrder(h, chain) == modelTableOrder(h, chain)); +} +// Mutation: sorting the whole list on a row change. Gate U2 counts. +TEST_CASE("rows table: one changed row is one repair", "[ui][table]") { + CHECK(engineOf(h).stats().repairs == 1); CHECK(engineOf(h).stats().fullSorts == sortsBefore); +} +// Mutation: no bound check. +TEST_CASE("rows table: a list over 20,000 rows is reported once", "[ui][table]") {} +// Invariant 17: the existing `rows` table suites (test_ui_table.cpp, test_ui_interpreter_table.cpp, +// test_ui_table_row_span.cpp) stay green with no expectation changed. +``` +The collection half of "One order on every surface" is Task 13's (step 8): this case covers the `rows` and model-backed surfaces and Task 13 adds the collection. B1 at 20,000 rows is a `[.benchmark]` case in `bench_interpreter.cpp` reading `EngineStats::maxOwnerStepMicros` after a three-key sort and a text filter, asserting first that the engine ran in more than one owner step. + +- [ ] **Step 2: Run to verify they fail** — `ctest --test-dir build/cl-debug -R "one order on every surface|rows table" --output-on-failure` → FAIL: orders differ (the interpreter's `Evaluator::sortRows` is still in use). +- [ ] **Step 3: Implement** as above; delete `ScreenCore::sortedRows` and the ordering part of `tableView`. +- [ ] **Step 4: Run to verify they pass** — the new cases and the whole `morph_tests` target green, then `morph_qt_quick_tests` and `morph_tui_tests` on `cl-qt-debug` green (their table cases mount through the interpreter). +- [ ] **Step 5: Mutations**, as named. Record U1 to U4 and B1 (10,000, 20,000 and 100,000 rows) from `cl-release` into table.md's Performance tables. +- [ ] **Step 6: Neighbouring specs** as listed; docs build green. +- [ ] **Step 7: Commit** `feat(ui): tables over rows on the engine`. + +--- + +## Self-review notes + +- Spec coverage: "The pipeline" and "Sources" — 9a (`EngineRows`, `WindowRows`), 11a (`ValueRowsSource`); `VectorSource` is Task 13's. "The client table model" — 9a, 9d. "The renderer" — 10a, 10b; its "Range selection in server mode" lands in 10a (cross-part decision 13), although Delivery order lists range selection under step 7. "The document `table` node" — model-backed rows in 9d, `rows` rows in 11b; `cellMenu`, `headerMenu`, `layout`, `configurable`, `locked`, display parts and `maxSortKeys` are Tasks 12, 15, 16, 17. "Where sort and filter run" — the client-mode bound warning in 11b; the refusal of a client-mode sort on a column with a level needs a `rows` column with a level, which a document cannot declare yet, so it has no task here. "Persisted state" (sort and filters, both stores, load before rows) — 9b to 9d; saved and shared views are Task 14's. +- Gates: U1–U4 (11a/11b), U6 and F1, F6, F7, K5, K8, T1, T3–T5, W6 (9), F1–F5, F8–F10, U7, D8 (10), U8 (10c, measured locally in a headless browser), B1 (11b). diff --git a/docs/superpowers/plans/2026-11-17-one-table-e-configure.md b/docs/superpowers/plans/2026-11-17-one-table-e-configure.md new file mode 100644 index 000000000..ca0dff098 --- /dev/null +++ b/docs/superpowers/plans/2026-11-17-one-table-e-configure.md @@ -0,0 +1,1102 @@ +# The One Table, Part E: Configuring, Collections, Saved Views, the Sort Bound — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let a user arrange any table from its header and its table menu (column order, visibility, pinning, width, sort chains, filters by values and conditions, column groups, display units, copy, range selection in server mode, a status line that shows every refusal), put collections on the same lowering, keep saved and shared views, and raise the sort bound to 32 keys. + +**Architecture:** The gestures are pure functions in a new UI header (`include/morph/ui/table_config.hpp`) over `table::ColumnLayout`, `ui::SortChain` and `table::FilterSpec`, tested without a backend. The backend contract grows header menus, sort chains, column moves and resizes, filter buttons, group headers, the status line and the clipboard; both backends and `RecordingBackend` implement them and one set of conformance cases holds all three to it. The interpreter's table lowering (built by Tasks 9 and 11) reads the new document keys and wires gestures to `TableRows`; the filter menu, the column chooser and the sort dialog are view trees the lowering mounts in dialogs, so both backends draw them with widgets they already have. Collections lower through the same table surface; saved views are the `views` part of persisted state, with an operation-and-conflict rule and a `publish` action for shared views. + +**Tech Stack:** C++23, header-only morph plus `src/ui`, `src/qt_quick`, `src/tui`; glaze; Catch2; Qt 6 Quick (offscreen `morph_qt_quick_tests` only); the terminal backend (`morph_tui_tests`). + +**Spec:** `docs/spec/table/table.md` — "Configuring a table" (all subsections: "Columns: order, visibility, pinning, width", "Sort chains", "Filters from the header", "Column groups" (the menu), "The status line", "Copy", "Persisted state" with "The framework's stores", "Saved views"), "Keyboard, focus and assistive technology" (range selection in server mode), "Collections and master-detail", "Values with units" ("The values list and the menu", "Display"), the matching rows of "Tests", invariants 21, 26, 31, gates D7, D9, B3, B4, U5, T2, and the step 7, 8, 9 and 9c rows of "Neighbouring specs" and "Delivery order". Roadmap: `docs/superpowers/plans/2026-11-17-one-table.md` (Tasks 12, 13, 14, 16). + +Part E of 7; index: 2026-11-17-one-table.md + +## Global Constraints + +Everything in the roadmap's Global Constraints and its standard task loop applies to every task here. In addition: + +- Limits, verbatim from the spec: a chain holds at most `maxSortKeys` keys, 8 by default and up to 32 (`TableCapabilities::maxSortKeys`, a `rows` table's literal `maxSortKeys`); `SessionOptions::maxRangeRows = 10'000`, `maxCopyRows = 10'000`; `QueryLimits::maxKeys = 1000` (rows per keyed copy request), `maxSetValues = 10'000`, `maxGroupColumns = 16`; `DistinctRequest::limit = 200`; a text filter and a values search are debounced 250 ms; a status message holds for six seconds or until the user's next gesture on the table; `tableStateDebounce = 500 ms`; `segmentBytes = 12 KiB`. +- Status line and menu copy is exact, English, with counts grouped by `ui::groupedCount` with the session's numeric grouping separator (`FormatContext`; "10,000" in English): + - `A table sorts by at most {n} columns` · `A range selects at most {n} rows` · `A copy holds at most {n} rows` + - `Copied {n} rows` (`Copied 1 row`), and with unloaded cells `Copied {n} rows, {m} cells were not loaded` (`1 cell was not loaded`) + - `{label} is locked` · `At least one column stays visible` · `This table's columns cannot be moved` (`hidden`, `pinned`, `resized`) + - `Too many values to list; filter with a condition` · `The list may grow while data loads` · `{n} cells could not be read` (`1 cell could not be read`) + - header accessible description `Sorted ascending` / `Sorted descending`, and with two or more keys `Sorted ascending, key 2 of 3`; header mark ` ▲` / ` ▼`, and with two or more keys ` ▲2`; a filtered header's accessible name `{label}, filtered` + - menus and dialogs: `Sort ascending`, `Sort descending`, `Add to sort`, `Remove from sort`, `Filter…`, `Filter {group}, {k} columns…`, `Move left`, `Move right`, `Pin column`, `Unpin`, `Hide column`, `Display unit`; `Columns…`, `Sort…`, `Clear all filters`, `Reset columns`, `Copy`, `Copy with headers`; `(Select all)`, `(Blanks)`, `Group columns…`, `Add key`; views (Task 14): `Views`, `Save view…`, `Rename view…`, `Remove view`, `Set as default`, `Start from the default view`, `Shared views`, `Publish view…`, `Save as my view`, `{name} (modified)`, `Saved view {name}`, `A view named {name} exists`, `Views could not be saved: another window changed them at the same time`, `Published view {name}`, `This view could not be published: {reason}` +- Test trees. `morph_tests` and `morph_table_tests` build in `cl-debug`. `morph_qt_quick_tests` and `morph_tui_tests` need Qt Quick and the terminal backend, which no checked-in Windows preset turns on: configure once with `cmake --preset cl-qt-debug -DMORPH_BUILD_QT_QUICK=ON -DMORPH_BUILD_TUI=ON`, build with `cmake --build build/cl-qt-debug --target `, run with `ctest --test-dir build/cl-qt-debug -R "" --output-on-failure` (Qt Quick cases run with `QT_QPA_PLATFORM=offscreen`, set by their `catch_discover_tests`). `morph_net_tests` is POSIX-only (`MORPH_BUILD_NET` is ignored on Windows): its cases build and run on the `linux-everything` preset, `ctest --test-dir build/linux-everything -R ""`. Every case that can run in `morph_tests` is placed there, so a Windows tree runs it. +- Timings marked (P) or (M) in the spec are recorded into `docs/spec/table/table.md` "Performance" with the date and the machine, from a `cl-release` (or `linux-everything` for `tests/net`) build; a gate is an ordinary test. +- No GUI application is launched. The one step that grabs screens (Task 12f) runs the example client offscreen through the repository's grab scripts and is run by the user, not by an agent. + +## Review Focus + +1. **Hiding a column the view is sorted or filtered by.** Arranging is not ordering (invariant 21): the sort and the filter keep applying, the next query still sorts and filters by the hidden column while no longer naming it in `columns`, the header mark and the filtered state reappear when it is shown, and the chooser lists it unticked. Test goes to Task 12c ("hiding a sorted and filtered column changes no row"). +2. **A chain at its bound.** With `maxSortKeys` keys held, `Shift`+click on a column already in the chain flips its direction and is not refused; on any other column it is refused and the chain is unchanged; "Add to sort" is disabled and "Remove from sort" frees a place. Tests go to Task 12a (the gesture) and Task 12c (the message). +3. **Exactly at a row bound.** A range of exactly `maxRangeRows` (10,000) rows selects them all; 10,001 is refused and selects nothing (Part D's Task 10a rule, held here at its edge). A copy of exactly 10,000 rows copies; 10,001 is refused. Test goes to Task 12f ("range and copy at their bounds"). +4. **Cell text that breaks a line or a column.** A cell holding `a\tb\r\nc`, a lone `\r` and a label with a tab copy as one line of tab-separated fields: each tab, `\r\n`, `\r` and `\n` is one space, headers included. Test goes to Task 12a ("copy text: tabs and line breaks become one space"). +5. **A view name already taken.** Saving a view under a name the user already has replaces that view (one `replace` operation, still one view of that name); renaming a view onto a taken name is refused with `A view named {name} exists` and changes nothing; the same holds when the taken name arrived from another window between load and save (the operation is applied to what is stored now). Test goes to Task 14b ("views: a taken name is replaced on save and refused on rename"). + +--- + +## Interfaces this part consumes from Parts A–D + +Names the roadmap pins are used as it gives them. Where the roadmap leaves a name open, this plan uses the one below; the executor of each consuming task checks it against the landed code first and, if it differs, uses the landed name and corrects this plan. + +| From | Pinned by the roadmap | Assumed here (not pinned) | +|---|---|---| +| Task 1 | `table::TableColumn{info, label, group, sortable, filterable, shown}`, `ColumnGroupInfo{id, label, columns}`, `TableCapabilities{keys, distinct, maxSortKeys = 8}`, `TableDescription`, `GenericRow`, `GenericPage` | — | +| Task 2 | `ServedTable{description, source, levels, mayRead, scope, services, limits}`, `BRIDGE_REGISTER_TABLE` with `Table.` and `Table..describe` | the index's answer path `table::detail::answerQuery(ServedTable const&, TableQuery const&, AnswerContext const& = {})` in `description.hpp`; Task 16 makes it and the `describe` handler take their description from `table::servedDescription` | +| Tasks 3–5 | `struct Unloaded`, `TableQuery::{columns, keys}`, `Page::indices` | `TableQuery::retry` (bool), from the level cache's rule 11 | +| Task 8a | `FilterEntry::oneOf`, `DistinctRequest`, `DistinctValue`, `DistinctValues`, `DateBucket`, `Engine::distinctValues`, `TableQuery::distinct`, `Page::distinct` | — | +| Task 8b | `GroupMode`, `ColumnCount{Kind, n}`, `GroupFilter{group, columns, mode, match, combine, include, exclude}` | — | +| Task 8c | `ColumnKind::Measure`, `UnitInfo`, `DistinctValue::{dimension, unit}` | — | +| Task 8d | `ColumnLayout::setWidth(id, std::optional)`, `widths()`, `setPinned(n)`, `pinned()`, `value()`, `apply(value)` | as Part C writes it: `table::ColumnLayoutValue{order, hidden, pinned, widths, groups, units}` (`groups` a `std::vector` of user-formed groups, `units` column → dimension → unit id), `std::size_t pinned()` counting the first visible columns, `widths()` a `std::map const&`; `apply` and `rebuild` fit groups and units | +| Task 9 (Part D) | `ui::SortChain`, `TableCountsView`, `TableRows` (`total`, `row`, `unloadedLevels`, `visible`, `keysOf`, `setSort`, `setFilter`, `counts`, `onChange`), `EngineRows`, `WindowRows`, `TableStateKey`, `StoredTableState`, `SaveOutcome`, `SaveResult`, `ITableStateStore`, `MemoryTableStateStore`, `PreferencesTableStateStore`, `TableStateModel` (`morph.tableState`), `SessionOptions::tableState` | as Part D writes them: `EngineRows(std::shared_ptr, RecordOf, table::EngineOptions)`; `WindowRows::retry()`, `WindowRows::description()`; `ui::countsText(TableCountsView const&, std::int64_t selected, FormatContext const&)`; `ui::detail::standInCellText(Cell const&, table::TableColumn const&)`; `ui::TableStatus{counts, state, retry, message, shown}`, `Table::status`, `Table::onRetry`, `TableWidget::setStatus`, `setOnRetry` (the status line, its counts, Retry and the `status` key are Part D's); `SessionOptions::application`; `table::StateDocument{version, layout, sort, filters, view, carried}`, `readState`, `writeState`, `fitState(StateDocument, TableDescription const&, FitReport&)`; `MemoryTableStateStore(exec::IExecutor&)` with revisions `"1"`, `"2"`, …; `ui::TableStateLoad{keys}`, `TableStateSave{key, expected, upload, offset, text, last}` → `TableStateSaved{result}`, `registerTableState(std::shared_ptr, dispatcher, registry, TableStateOptions)`; `PreferencesTableStateStore(Dispatcher&, exec::IExecutor& owner)`; test support `tests/ui_table_support.hpp`: `tableCatalog()` (the corpus catalog plus `Table.samples` and `.describe` on `Lims_SampleModel`), `samplesDescription(int columns)` (`id` Integer key, `name` Text, then `c3…` Decimal), `describeJson`, `pageJson(offset, count, total, columns, kind, undecided, awaiting)`, `GatedTableStateStore`, `Harness(std::shared_ptr)` (sets `tableState` and `application = "lims"`), `h.replyDescribe()`, `h.replyPage(json)`, `lastQuery(h)`, `statusOf(h)` | +| Task 10 (Part D) | the column window; `TableWidget`'s range report by view rows | as Part D writes them: `ui::TableColumn::id`, `ui::TableArrangement{visible, pinned, widths}`, `Table::arrangement` (applied without rebuilding rows, gate D8), `TableWidget::setOnSelectRange(std::function)`, `RecordingBackend::selectTableRange(id, anchor, focus)`, `SessionOptions::maxRangeRows`; range selection through `keysOf` and its refusal are Task 10a's | +| Task 11 (Part D) | `ValueRowsSource`, rows tables on `EngineRows`; the internal lowering of a table node onto `TableRows` | a column part's `kind` (`ValueColumn::kind`) and the refusal of `persist` on a table with no `id` are Task 11's | + +--- + +## Task 12: The configuration interface (table.md step 7) + +**Spec:** "Configuring a table" (all), "Values with units" ("The values list and the menu", "Display"), "Keyboard, focus and assistive technology" (range selection in server mode), "The document `table` node on the engine" (rows for `layout`, `status`, `persist`, `filters`, a column's `kind`, `locked`, `filterable`). Gates D7, D9, B3, B4. Six commits, each its own reviewer gate. + +### Task 12a: Gestures, menus, value filters and copy text, without a backend + +**Files:** +- Create: `include/morph/ui/table_config.hpp`, `tests/test_ui_table_config.cpp` (add to `morph_tests` in `tests/CMakeLists.txt`, beside `test_ui_table.cpp`) +- Modify: `include/morph/table/engine.hpp` (`ColumnLayout`, as Task 8d leaves it: the mutators for user-formed groups and display units), `tests/table/test_table_columns.cpp`, `CMakeLists.txt` (list the new header in the `morph` header set) +- Docs: `docs/spec/table/engine.md` ("`ColumnLayout`": the group and unit mutators) + +**Interfaces:** +- Consumes: Task 8d's `ColumnLayout` and `ColumnLayoutValue`; Task 8a's `FilterEntry::oneOf`; Task 8b's `GroupFilter`, `GroupMode`, `ColumnCount`; `table::SortKey{column, dir}`, `table::SortDirection`, `table::ColumnKind`. +- Produces (namespace `morph::table`, `engine.hpp`; the stored shape and its fitting are Task 8d's): + - `ColumnLayout::formGroup(std::string label, std::vector columns) -> std::expected` (returns the new id `g1`, `g2`, … the first unused; `InvalidSpec` for fewer than 2 or more than 16 columns, an unknown column, or a column already in a user-formed group), `removeGroup(std::string const& id) -> bool`, `groups() -> std::span`, `setUnit(std::string const& column, std::string const& dimension, std::optional unit) -> bool` (`nullopt` removes it), `units() -> std::map> const&`; all of them read and write the same members `value()` and `apply` carry. +- Produces (namespace `morph::ui`, `table_config.hpp`): + - `enum class SortGesture : std::uint8_t { Click, Extend, Ascending, Descending, Add, Remove }`; `struct SortChange { SortChain chain; std::optional refusal; }`; `[[nodiscard]] SortChange applySortGesture(SortChain const& chain, std::string const& column, SortGesture gesture, std::size_t maxSortKeys)` + - `enum class Configure : std::uint8_t { None = 0, Reorder = 1, Hide = 2, Pin = 4, Resize = 8, All = 15 }` with `operator|`, `operator&` and `[[nodiscard]] constexpr bool allows(Configure set, Configure what)` + - `struct LayoutColumn { std::string id; std::string label; bool locked = false; table::ColumnKind kind = table::ColumnKind::Text; }` + - gestures `MoveColumn{id, int by}`, `MoveColumnTo{id, std::size_t position}`, `HideColumn{id}`, `ShowColumn{id}`, `PinColumn{id}`, `UnpinColumn{id}`, `ResizeColumn{id, std::optional units}`, `ResetColumns{}`, `SetDisplayUnit{id, dimension, std::optional unit}`, `FormGroup{label, columns}`, `RemoveGroup{id}`, as `using LayoutGesture = std::variant<…>`; `struct LayoutResult { bool changed = false; std::optional refusal; }`; `[[nodiscard]] LayoutResult applyLayoutGesture(table::ColumnLayout& layout, std::span columns, Configure allowed, LayoutGesture const& gesture)` + - The tick model. Under a search, "(Select all)" means the values the search shows (spec 1766-1767), which the menu does not hold when they pass a page, so a tick is recorded as an operation and resolved only at Apply: + - `struct TickOp { std::string search; std::optional operand; bool ticked = true; }` — `operand` set: one value's box; unset: "(Select all)" under `search` (empty `search`: every value) + - `struct ValueTicks { bool ticked = true; std::vector ops; std::optional blanks; }` — `ticked` is every value's state before the first operation (true with no filter on the column); later operations win + - `struct ResolvedTicks { bool ticked = true; std::map covered; std::optional blanks; }` — `covered`: the final state of every value an operation reached, by operand + - `[[nodiscard]] ResolvedTicks resolveTicks(ValueTicks const& ticks, std::function(std::string const& search)> const& valuesUnder)` — `valuesUnder` returns the operands a search shows; the menu answers it by paging `distinctValues` with `limit = QueryLimits::maxLimit` at Apply; an op with an empty search and no operand clears `covered` and sets `ticked` + - `struct ValueFilterPlan { enum class Side : std::uint8_t { All, Include, Exclude, Nothing, Disabled } side = Side::All; std::size_t members = 0; bool needsComplement = false; }`; `[[nodiscard]] ValueFilterPlan planValueFilter(std::int64_t distinct, ResolvedTicks const& ticks, std::size_t maxSetValues)` + - `enum class ValueForm : std::uint8_t { Set, Ranges }` (`Ranges` for date buckets, whose operand is `a..b`); `[[nodiscard]] std::optional valueFilter(ValueFilterPlan::Side side, std::vector members, std::optional blanks, ValueForm form = ValueForm::Set)` + - Values and conditions are exclusive per column: one `ColumnFilter` has one `combine` (`filter.hpp:86-93`), so `[oneOf, isEmpty, condition]` under `All` would match nothing. `enum class FilterPart : std::uint8_t { None, Values, Conditions }`; `[[nodiscard]] FilterPart classifyFilter(table::ColumnFilter const& filter, table::ColumnKind kind)` — `Values` when the filter is a values shape: `combine = Any` and `include` is `{oneOf}`, `{oneOf, isEmpty}`, one or more `between` entries (a date column's buckets) with or without `isEmpty`, or a lone `{isEmpty}` on a column that is not `Custom`, and `exclude` the same shapes with `include` empty; a lone `isEmpty`/`notEmpty` on a `Custom` column is `Conditions` (it has no values list); anything else not empty is `Conditions` + - `[[nodiscard]] std::optional canonicalOperand(std::string_view typed, table::ColumnKind kind, LocaleFacts const& locale, std::span units = {}, std::string_view chosenUnit = {})` — the wire form of a typed operand: a number in the locale's form (`"1.234,5"` under a comma locale) becomes a decimal with a point and no grouping (`"1234.5"`); a `Measure` operand goes through Part C's `ui::parseMeasure(typed, units, locale, chosenUnit)` and `ui::measureOperand`; a date is the picker's ISO text; text is unchanged; `nullopt` when it does not read + - `enum class GroupMatch : std::uint8_t { AnyColumn, EveryColumn, AtLeast, Exactly, Together }`; `[[nodiscard]] std::expected groupValueFilter(std::string group, std::size_t groupColumns, std::vector ticked, std::optional blanks, GroupMatch match, std::int64_t n)` + - `enum class ConditionOp : std::uint8_t { Equals, NotEquals, Contains, BeginsWith, EndsWith, Less, LessOrEqual, Greater, GreaterOrEqual, Between, IsEmpty, IsNotEmpty }`; `[[nodiscard]] std::vector conditionOps(table::ColumnKind kind)`; `[[nodiscard]] std::string_view conditionLabel(ConditionOp op, table::ColumnKind kind)`; `[[nodiscard]] table::FilterEntry conditionEntry(ConditionOp op, std::string operand, std::string second, bool matchCase)` + - `enum class MenuCommand : std::uint8_t { SortAscending, SortDescending, AddToSort, RemoveFromSort, Filter, FilterGroup, MoveLeft, MoveRight, PinColumn, UnpinColumn, HideColumn, DisplayUnit, Columns, SortDialog, ClearAllFilters, ResetColumns, Copy, CopyWithHeaders }` (Task 14 appends its view commands); `struct TableMenuEntry { MenuEntry entry; MenuCommand command; std::string argument; std::vector items; }`; `struct MenuGroup { std::string id; std::string label; std::size_t columns = 0; }`; `struct HeaderMenuInput { LayoutColumn column; bool sortable, filterable; std::optional sortPlace; std::size_t chainSize, maxSortKeys, visibleIndex, visibleCount, pinned; std::vector>> dimensions; std::vector groups; Configure allowed; }`; `struct TableMenuInput { bool anyFilter; std::vector groups; Configure allowed; }`; `[[nodiscard]] std::vector headerMenu(HeaderMenuInput const&)`; `[[nodiscard]] std::vector tableMenu(TableMenuInput const&)`; `[[nodiscard]] std::vector menuEntries(std::span)`; `[[nodiscard]] TableMenuEntry const* chosen(std::span, std::span path)` + - `struct CopiedText { std::string text; std::size_t unloaded = 0; }`; `[[nodiscard]] CopiedText copyText(std::span labels, std::span> const> rows)` (empty `labels`: no header line; `nullopt`: an unloaded cell) + - `[[nodiscard]] std::string groupedCount(std::int64_t n, std::string_view separator = ",")` + - `namespace tableText`: `sortBound(std::size_t)`, `rangeBound(std::size_t, std::string_view separator = ",")`, `copyBound(std::size_t, std::string_view separator = ",")`, `copied(std::size_t rows, std::size_t unloaded, std::string_view separator = ",")`, `locked(std::string_view label)`, `notConfigurable(Configure what)`, `lastVisible()`, `tooManyValues()`, `mayGrow()`, `unreadable(std::size_t)`, each `-> std::string`, the exact copy of Global Constraints + - `inline constexpr std::chrono::seconds kStatusMessageHold{6};` + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_table_config.cpp` (tag `[ui][table][config]`), each preceded by its mutation comment: + +```cpp +// Mutation: Extend treated as Click (Shift ignored). +TEST_CASE("sort gesture: a click replaces the chain; Shift adds a key or flips one in place", "[ui][table][config]") { + using table::SortDirection::Ascending; using table::SortDirection::Descending; + ui::SortChain const two{{"name", Ascending}, {"mass", Descending}}; + CHECK(ui::applySortGesture(two, "name", ui::SortGesture::Click, 8).chain == ui::SortChain{{"name", Descending}}); + CHECK(ui::applySortGesture(two, "mass", ui::SortGesture::Click, 8).chain == ui::SortChain{{"mass", Ascending}}); + CHECK(ui::applySortGesture({{"name", Descending}}, "name", ui::SortGesture::Click, 8).chain == + ui::SortChain{{"name", Ascending}}); + CHECK(ui::applySortGesture(two, "site", ui::SortGesture::Extend, 8).chain == + ui::SortChain{{"name", Ascending}, {"mass", Descending}, {"site", Ascending}}); + CHECK(ui::applySortGesture(two, "name", ui::SortGesture::Extend, 8).chain == + ui::SortChain{{"name", Descending}, {"mass", Descending}}); + CHECK(ui::applySortGesture(two, "mass", ui::SortGesture::Remove, 8).chain == ui::SortChain{{"name", Ascending}}); + CHECK(ui::applySortGesture(two, "site", ui::SortGesture::Descending, 8).chain == ui::SortChain{{"site", Descending}}); +} + +// Mutation: the bound checked before the "already in the chain" case (the flip at the bound is refused). +TEST_CASE("sort gesture: at the bound a new key is refused and a key in the chain still flips", "[ui][table][config]") { + using table::SortDirection::Ascending; using table::SortDirection::Descending; + ui::SortChain const full{{"a", Ascending}, {"b", Ascending}}; + ui::SortChange const added = ui::applySortGesture(full, "c", ui::SortGesture::Extend, 2); + CHECK(added.chain == full); + CHECK(added.refusal == "A table sorts by at most 2 columns"); + CHECK(ui::applySortGesture(full, "c", ui::SortGesture::Add, 2).refusal == "A table sorts by at most 2 columns"); + ui::SortChange const flipped = ui::applySortGesture(full, "b", ui::SortGesture::Extend, 2); + CHECK(flipped.chain == ui::SortChain{{"a", Ascending}, {"b", Descending}}); + CHECK_FALSE(flipped.refusal); +} + +// Mutation: applyLayoutGesture ignoring `locked`. +TEST_CASE("layout gesture: moves, hides, pins and resets, refusing a locked column and the last visible one", + "[ui][table][config]") { + table::ColumnLayout layout{{"id", "name", "mass", "site"}}; + std::vector const columns{{"id", "ID", true}, {"name", "Name"}, {"mass", "Mass"}, {"site", "Site"}}; + CHECK(ui::applyLayoutGesture(layout, columns, ui::Configure::All, ui::MoveColumn{"mass", -1}).changed); + CHECK(std::ranges::equal(layout.order(), std::vector{"id", "mass", "name", "site"})); + CHECK(ui::applyLayoutGesture(layout, columns, ui::Configure::All, ui::MoveColumn{"id", 1}).refusal == "ID is locked"); + CHECK(ui::applyLayoutGesture(layout, columns, ui::Configure::All, ui::PinColumn{"name"}).changed); + CHECK(layout.pinned() == 1); + CHECK(layout.visible().front() == "name"); + CHECK(ui::applyLayoutGesture(layout, columns, ui::Configure::All, ui::ResizeColumn{"mass", 14}).changed); + CHECK(layout.widths().at("mass") == 14); + for (std::string const id : {"name", "mass", "site"}) { + ui::applyLayoutGesture(layout, columns, ui::Configure::All, ui::HideColumn{id}); + } + CHECK(layout.visible() == std::vector{"id"}); + CHECK(ui::applyLayoutGesture(layout, columns, ui::Configure::All, ui::HideColumn{"id"}).refusal == + "ID is locked"); + table::ColumnLayout lone{{"a", "b"}}; + std::vector const ab{{"a", "A"}, {"b", "B"}}; + ui::applyLayoutGesture(lone, ab, ui::Configure::All, ui::HideColumn{"a"}); + CHECK(ui::applyLayoutGesture(lone, ab, ui::Configure::All, ui::HideColumn{"b"}).refusal == + "At least one column stays visible"); + CHECK(ui::applyLayoutGesture(lone, ab, ui::Configure::Hide, ui::MoveColumn{"b", -1}).refusal == + "This table's columns cannot be moved"); + CHECK(ui::applyLayoutGesture(layout, columns, ui::Configure::All, ui::ResetColumns{}).changed); + CHECK(std::ranges::equal(layout.order(), std::vector{"id", "name", "mass", "site"})); + CHECK(layout.pinned() == 0); + CHECK(layout.widths().empty()); +} + +// Mutation: formGroup accepting a column already in a user-formed group. +TEST_CASE("column layout: the user forms and removes groups and sets display units", "[table][columns]") { + table::ColumnLayout layout{{"companyA", "companyB", "companyC", "weight"}}; + auto const id = layout.formGroup("Company", {"companyA", "companyB"}); + REQUIRE(id == "g1"); + CHECK_FALSE(layout.formGroup("Again", {"companyB", "companyC"})); + CHECK_FALSE(layout.formGroup("One", {"companyC"})); + CHECK_FALSE(layout.formGroup("Ghost", {"companyC", "ghost"})); + CHECK(layout.formGroup("Pair", {"companyC", "weight"}) == "g2"); + CHECK(layout.removeGroup("g2")); + CHECK(layout.setUnit("weight", "mass", "g")); + CHECK_FALSE(layout.setUnit("ghost", "mass", "g")); + table::ColumnLayoutValue const value = layout.value(); + REQUIRE(value.groups.size() == 1); + CHECK(value.groups.at(0).columns == std::vector{"companyA", "companyB"}); + CHECK(value.units == std::map>{{"weight", {{"mass", "g"}}}}); + CHECK(layout.setUnit("weight", "mass", std::nullopt)); + CHECK(layout.units().empty()); +} + +// Mutation: planValueFilter always sending the side the operations reached. +TEST_CASE("value filter: the smaller side is sent, and both sides over the bound disable the list", "[ui][table][config]") { + using Side = ui::ValueFilterPlan::Side; + auto const covering = [](int count, bool ticked) { + std::map covered; + for (int i = 0; i < count; ++i) { covered.emplace("v" + std::to_string(i), ticked); } + return covered; + }; + auto const one = ui::planValueFilter(50'000, ui::ResolvedTicks{.ticked = true, .covered = {{"v00007", false}}}, 10'000); + CHECK(one.side == Side::Exclude); + CHECK(one.members == 1); + CHECK_FALSE(one.needsComplement); + CHECK(ui::planValueFilter(25'000, ui::ResolvedTicks{.ticked = true}, 10'000).side == Side::All); + CHECK(ui::planValueFilter(25'000, ui::ResolvedTicks{.ticked = false, .covered = covering(12'000, true)}, 10'000).side == + Side::Disabled); + auto const complement = ui::planValueFilter(25'000, ui::ResolvedTicks{.ticked = true, .covered = covering(20'000, false)}, 10'000); + CHECK(complement.side == Side::Include); + CHECK(complement.members == 5'000); + CHECK(complement.needsComplement); + CHECK(ui::planValueFilter(3, ui::ResolvedTicks{.ticked = false, .blanks = false}, 10'000).side == Side::Nothing); + auto const include = ui::valueFilter(Side::Include, {"F", "S"}, true); + REQUIRE(include); + CHECK(include->combine == table::Combine::Any); + REQUIRE(include->include.size() == 2); + CHECK(include->include.at(0).oneOf == std::vector{"F", "S"}); + CHECK(include->include.at(1).isEmpty == true); + auto const exclude = ui::valueFilter(Side::Exclude, {"X"}, false); + REQUIRE(exclude); + REQUIRE(exclude->exclude.size() == 2); + CHECK(exclude->exclude.at(1).isEmpty == true); + auto const ranges = ui::valueFilter(Side::Include, {"2026-03-01..2026-03-31", "2026-05-14..2026-05-14"}, std::nullopt, + ui::ValueForm::Ranges); + REQUIRE(ranges); + REQUIRE(ranges->include.size() == 2); + CHECK(ranges->include.at(0).between == std::array{"2026-03-01", "2026-03-31"}); +} + +// Mutation: resolveTicks applying the operations in reverse order. +TEST_CASE("value ticks: operations under searches resolve in order, the last one winning", "[ui][table][config]") { + std::map> const shows{{"ab", {"ab1", "xab"}}, {"x", {"xab", "xy"}}}; + auto const under = [&](std::string const& search) { return shows.at(search); }; + ui::ValueTicks const ticks{.ticked = true, + .ops = {{.search = "", .operand = std::nullopt, .ticked = false}, + {.search = "ab", .operand = std::nullopt, .ticked = true}, + {.search = "x", .operand = std::nullopt, .ticked = false}, + {.search = "", .operand = "xab", .ticked = true}}}; + ui::ResolvedTicks const resolved = ui::resolveTicks(ticks, under); + CHECK_FALSE(resolved.ticked); + CHECK(resolved.covered == std::map{{"ab1", true}, {"xab", true}, {"xy", false}}); +} + +// Mutation: classifyFilter reading a Custom column's lone isEmpty as "(Blanks)". +TEST_CASE("filter parts: a column's filter is values or conditions, never both", "[ui][table][config]") { + using table::ColumnKind; + auto const entry = [](auto set) { table::FilterEntry e; set(e); return e; }; + table::ColumnFilter values{.combine = table::Combine::Any, + .include = {entry([](auto& e) { e.oneOf = std::vector{"F"}; }), + entry([](auto& e) { e.isEmpty = true; })}}; + CHECK(ui::classifyFilter(values, ColumnKind::Text) == ui::FilterPart::Values); + table::ColumnFilter blanks{.combine = table::Combine::Any, .include = {entry([](auto& e) { e.isEmpty = true; })}}; + CHECK(ui::classifyFilter(blanks, ColumnKind::Text) == ui::FilterPart::Values); + CHECK(ui::classifyFilter(blanks, ColumnKind::Custom) == ui::FilterPart::Conditions); + table::ColumnFilter conditions{.combine = table::Combine::All, + .include = {entry([](auto& e) { e.gt = "2"; }), entry([](auto& e) { e.lt = "5"; })}}; + CHECK(ui::classifyFilter(conditions, ColumnKind::Integer) == ui::FilterPart::Conditions); + CHECK(ui::classifyFilter({}, ColumnKind::Integer) == ui::FilterPart::None); +} + +// Mutation: canonicalOperand passing typed text through unchanged. +TEST_CASE("typed operands: numbers and measures reach the wire with a point and a unit id", "[ui][table][config]") { + ui::LocaleFacts const de = ui::localeFacts("de"); + CHECK(ui::canonicalOperand("1.234,5", table::ColumnKind::Decimal, de) == "1234.5"); + CHECK(ui::canonicalOperand("12,5", table::ColumnKind::Integer, de) == std::nullopt); + CHECK(ui::canonicalOperand("0,3 kg", table::ColumnKind::Measure, de, massUnits()) == "0.3 kg"); + CHECK(ui::canonicalOperand("250", table::ColumnKind::Measure, de, massUnits(), "g") == "250 g"); + CHECK(ui::canonicalOperand("abc", table::ColumnKind::Text, de) == "abc"); +} + +// Mutation: Together built as one oneOf entry (the row F, F, X would pass). +TEST_CASE("group filter from ticks: the match line chooses the quantifier, together takes one eq per value", + "[ui][table][config]") { + auto const atLeast = ui::groupValueFilter("company", 3, {"F", "S"}, std::nullopt, ui::GroupMatch::AtLeast, 2); + REQUIRE(atLeast); + CHECK(atLeast->match.kind == table::ColumnCount::Kind::AtLeast); + CHECK(atLeast->match.n == 2); + CHECK(atLeast->include.at(0).oneOf == std::vector{"F", "S"}); + auto const together = ui::groupValueFilter("company", 3, {"F", "S"}, std::nullopt, ui::GroupMatch::Together, 0); + REQUIRE(together); + CHECK(together->mode == table::GroupMode::Together); + REQUIRE(together->include.size() == 2); + CHECK(together->include.at(0).eq == "F"); + CHECK(together->include.at(1).eq == "S"); + CHECK_FALSE(ui::groupValueFilter("company", 3, {"F"}, std::nullopt, ui::GroupMatch::Together, 0)); + CHECK_FALSE(ui::groupValueFilter("company", 3, {"F", "S", "T", "U"}, std::nullopt, ui::GroupMatch::Together, 0)); + auto const blanks = ui::groupValueFilter("company", 3, {"F"}, true, ui::GroupMatch::AnyColumn, 0); + REQUIRE(blanks); + CHECK(blanks->include.at(1).isEmpty == true); +} + +// Mutation: conditionOps returning the numeric list for every kind. +TEST_CASE("conditions follow the column's kind", "[ui][table][config]") { + using enum ui::ConditionOp; + CHECK(ui::conditionOps(table::ColumnKind::Text) == + std::vector{Equals, NotEquals, Contains, BeginsWith, EndsWith}); + auto const numeric = std::vector{Equals, NotEquals, Less, LessOrEqual, Greater, GreaterOrEqual, Between}; + for (auto const kind : {table::ColumnKind::Integer, table::ColumnKind::Key, table::ColumnKind::Decimal, + table::ColumnKind::Quantity, table::ColumnKind::Number, table::ColumnKind::Measure, + table::ColumnKind::Date, table::ColumnKind::DateTime}) { + CHECK(ui::conditionOps(kind) == numeric); + } + CHECK(ui::conditionOps(table::ColumnKind::Bool).empty()); + CHECK(ui::conditionOps(table::ColumnKind::Custom) == std::vector{IsEmpty, IsNotEmpty}); + CHECK(ui::conditionLabel(NotEquals, table::ColumnKind::Text) == "does not equal"); + CHECK(ui::conditionLabel(NotEquals, table::ColumnKind::Integer) == "≠"); + table::FilterEntry const between = ui::conditionEntry(Between, "1", "5", false); + CHECK(between.between == std::array{"1", "5"}); + CHECK(ui::conditionEntry(Contains, "ab", {}, true).caseSensitive == true); +} + +// Mutation: headerMenu leaving "Add to sort" enabled for a column already in the chain. +TEST_CASE("header menu: entries, order and enablement", "[ui][table][config]") { + ui::HeaderMenuInput const input{ + .column = {"name", "Name"}, .sortable = true, .filterable = true, .sortPlace = 0, .chainSize = 1, + .maxSortKeys = 8, .visibleIndex = 1, .visibleCount = 3, .pinned = 0, .dimensions = {}, + .groups = {{"company", "Company", 3}}, .allowed = ui::Configure::All}; + std::vector const entries = ui::menuEntries(ui::headerMenu(input)); + std::vector labels; + for (ui::MenuEntry const& entry : entries) { labels.push_back(entry.separator ? "-" : entry.label); } + CHECK(labels == std::vector{"Sort ascending", "Sort descending", "Add to sort", "Remove from sort", "-", + "Filter…", "Filter Company, 3 columns…", "-", "Move left", "Move right", + "Pin column", "Hide column"}); + CHECK_FALSE(entries.at(2).enabled); + CHECK(entries.at(3).enabled); + CHECK(entries.at(8).keys == "Ctrl+Shift+Left"); + ui::HeaderMenuInput locked = input; + locked.column.locked = true; + std::vector const lockedEntries = ui::menuEntries(ui::headerMenu(locked)); + CHECK_FALSE(lockedEntries.at(8).enabled); + CHECK_FALSE(lockedEntries.at(11).enabled); + ui::HeaderMenuInput hideOnly = input; + hideOnly.allowed = ui::Configure::Hide; + CHECK(ui::menuEntries(ui::headerMenu(hideOnly)).back().label == "Hide column"); + CHECK(ui::menuEntries(ui::headerMenu(hideOnly)).size() == 9); + std::vector const table = ui::menuEntries(ui::tableMenu({.anyFilter = false, .groups = {}, .allowed = ui::Configure::All})); + std::vector tableLabels; + for (ui::MenuEntry const& entry : table) { tableLabels.push_back(entry.separator ? "-" : entry.label); } + CHECK(tableLabels == std::vector{"Columns…", "Sort…", "Clear all filters", "Reset columns", "-", "Copy", + "Copy with headers"}); + CHECK_FALSE(table.at(2).enabled); + CHECK(table.at(5).keys == "Ctrl+C"); +} + +// Mutation: copyText replacing "\r\n" with two spaces. +TEST_CASE("copy text: tabs and line breaks become one space; unloaded cells are empty and counted", "[ui][table][config]") { + std::vector const labels{"Na\tme", "N"}; + std::vector>> const rows{{"a\tb\r\nc", "1"}, {std::nullopt, "x\ry\nz"}}; + ui::CopiedText const plain = ui::copyText({}, rows); + CHECK(plain.text == "a b c\t1\n\tx y z"); + CHECK(plain.unloaded == 1); + CHECK(ui::copyText(labels, rows).text == "Na me\tN\na b c\t1\n\tx y z"); +} + +TEST_CASE("status copy: counts are grouped and messages are exact", "[ui][table][config]") { + CHECK(ui::groupedCount(999) == "999"); + CHECK(ui::groupedCount(10'000) == "10,000"); + CHECK(ui::groupedCount(-1'234'567) == "-1,234,567"); + CHECK(ui::groupedCount(10'000, ".") == "10.000"); + CHECK(ui::tableText::sortBound(32) == "A table sorts by at most 32 columns"); + CHECK(ui::tableText::rangeBound(10'000) == "A range selects at most 10,000 rows"); + CHECK(ui::tableText::copyBound(10'000) == "A copy holds at most 10,000 rows"); + CHECK(ui::tableText::copied(200, 12) == "Copied 200 rows, 12 cells were not loaded"); + CHECK(ui::tableText::copied(1, 1) == "Copied 1 row, 1 cell was not loaded"); + CHECK(ui::tableText::copied(3, 0) == "Copied 3 rows"); + CHECK(ui::tableText::unreadable(1) == "1 cell could not be read"); +} +``` + +(`massUnits()` is a file-local helper returning the `UnitInfo`s `kg` and `g` of dimension mass, as Part C's measure tests build them; `ui::LocaleFacts` and `ui::localeFacts` are the existing locale table's.) + +- [ ] **Step 2: Run them to see them fail** + +Run: `cmake --build --preset cl-debug --target morph_tests morph_table_tests` +Expected: compile errors naming `morph/ui/table_config.hpp` (missing) and `ColumnLayout::formGroup`. + +- [ ] **Step 3: Implement** the interfaces above in `include/morph/ui/table_config.hpp` and `include/morph/table/engine.hpp`. + - `groupedCount` groups by three with `separator`; the lowering passes the session's numeric grouping separator, as Part D's `countsText` does; the status copy is English, as the framework's other defaults (`Select all`) are. + - `applySortGesture`: `Click` makes `{column, Descending}` when `chain[0]` is `{column, Ascending}`, else `{column, Ascending}`; `Ascending`/`Descending` make that one key; `Extend` flips a key already in the chain in place, else appends ascending; `Add` appends only a column not in the chain; `Remove` erases it. The "in the chain" case is decided before the bound; a new key past `maxSortKeys` returns the chain unchanged with `tableText::sortBound(maxSortKeys)`. + - The rule that pinned columns take at most half the table's width is the renderer's (Task 10), for display only; no gesture here changes the stored count for it. + - `applyLayoutGesture`: a gesture on a locked column refuses with `tableText::locked(label)` before anything else, a gesture the `allowed` set lacks with `notConfigurable` (`Reset` is allowed while any capability is); `MoveColumn` moves among visible columns, `PinColumn` places the column right after the last pinned visible column and adds one, `UnpinColumn` places it right after the pinned ones less one; `ResetColumns` restores the declared order and `shown`, `pinned = 0`, no widths; a hide that would leave none visible refuses with `lastVisible()`. + - `planValueFilter`: with D = `distinct`, C = `covered`, T = `(ticked ? D − |C| : 0) + |{c in C : c ticked}|`, U = D − T: `All` when T = D and `blanks` is not `false`; `Nothing` when T = 0 and `blanks` is not `true`; otherwise `Include` when T ≤ U, else `Exclude`, with `members = min(T, U)`, `Disabled` when that exceeds `maxSetValues`, and `needsComplement` when the side's members include values outside C (then the menu pages every value at `maxLimit` to name them). + - `valueFilter`: `Set` → `Include`: `include = {oneOf members} + {isEmpty} if blanks == true`, `combine = Any`; `Exclude`: `exclude = {oneOf members} (when not empty) + {isEmpty} if blanks == false`. `Ranges` → the same with one `between` entry per member, its two ends the member split at `..`. `All`, `Nothing`, `Disabled` → `nullopt`. + - `canonicalOperand` reads the locale's decimal and grouping separators from `LocaleFacts` (`ui::localeFacts`), never assuming a point. + - `groupValueFilter`: `AnyColumn`, `EveryColumn`, `AtLeast n`, `Exactly n` → one `oneOf` entry (plus `isEmpty` under `AnyColumn` when `blanks == true`) with `match` `Any`, `All`, `AtLeast`, `Exactly`; `Together` → one `eq` per ticked value, `mode = Together`, refused unless 2 ≤ ticked ≤ `groupColumns`. + - Every public symbol gets complete Doxygen. + +- [ ] **Step 4: Run them to see them pass** + +Run: `cmake --build --preset cl-debug --target morph_tests morph_table_tests && ctest --test-dir build/cl-debug -R "sort gesture|layout gesture|column layout|value filter|group filter from ticks|conditions follow|header menu|copy text|status copy" --output-on-failure` +Expected: all pass; then the whole of `morph_table_tests` and the `[ui]` cases of `morph_tests` pass. + +- [ ] **Step 5: Mutations, docs, commit.** Apply each mutation named above, see its case fail, revert; paste the failing lines into the commit body. Update `docs/spec/table/engine.md` ("`ColumnLayout`": user-formed groups and display units, and that `apply` and `rebuild` fit them). Run the docs build. Commit: `feat(ui): table gestures, menus and value filters as functions`. + +### Task 12b: The backend contract for configuring, in every backend + +**Files:** +- Modify: `include/morph/ui/view.hpp:364-374` (`TableSort`), `:1217-1283` (`TableColumn`, `Table`), `:2026-2049` (`TableOptions`); `include/morph/ui/backend.hpp:764-879` (`TableWidget`), `:1293-1510` (`IViewBackend::copyText`); `include/morph/ui/mount.hpp:1622-1645` (binding the new members); `include/morph/ui/testing/recording_backend.hpp:290-298` (`formatSort`), `:1986-2240` (`FakeTable`), `:3015-3040` (drivers); `include/morph/ui/testing/backend_conformance.hpp:60-225` (probe operations) and its cases +- Modify: `src/qt_quick/kinds.hpp:614-695`, `src/qt_quick/table.cpp`, `src/qt_quick/qml/MorphTable.qml`, `src/qt_quick/qml/MorphMenu.qml` (as the header and table menus' popup), `src/qt_quick/backend.cpp` (`copyText` through `QGuiApplication::clipboard()`); `src/tui/table_widget.hpp`, `src/tui/table_widget.cpp` (and the TUI backend's `copyText`: OSC 52) +- Modify: `include/morph/forms/engine/collection_view.hpp:143-227` (the `sort` and `onSort` members it leaves unset keep compiling) +- Test: `tests/test_ui_conformance.cpp`, `tests/qt_quick/test_qt_quick_conformance.cpp`, `tests/qt_quick/test_qt_quick_table.cpp`, `tests/tui/test_tui_conformance.cpp`, `tests/tui/test_tui_table.cpp` +- Docs: `docs/spec/ui/backend_contract.md` ("Widgets" table entry, "The probe", "The cases", "RecordingBackend"), `docs/spec/ui/view_tree.md` ("ForEach and Table rows"), `docs/spec/qt_quick/backend.md` (header menus, the table menu button, header drag and resize, the group header), the TUI table's header comment in `src/tui/table_widget.hpp`. + +**Interfaces:** +- Consumes: Part D's `ui::TableColumn::id`, `Table::arrangement` (`TableArrangement`, whose visible places `onColumnMove` positions and `TableColumnGroup::first` count in) and `Table::status`. +- Produces (namespace `morph::ui`): + - `enum class HeaderClick : std::uint8_t { Plain, Shift }` + - `[[nodiscard]] std::string sortMarkText(SortDirection direction, std::size_t place, std::size_t count)` and `[[nodiscard]] std::string sortDescription(SortDirection direction, std::size_t place, std::size_t count)` in `view.hpp` (`place` from 1) + - `struct TableColumnGroup { std::string label; std::size_t first = 0; std::size_t count = 0; bool filtered = false; }` + - `ui::TableColumn::filterable` (`bool`, default `false`) + - `ui::Table` and `TableOptions`: `Prop> sort` (replacing `Prop>`, chain order), `std::function onSort` (replacing the one-argument handler), `std::function(std::optional)> menu` and `std::function, std::vector)> onMenu` (no column: the table's menu), `std::function onColumnMove`, `std::function units)> onColumnResize`, `std::function onFilter`, `Prop> filtered`, `Prop> columnGroups`, `std::function onGroupFilter`, `Action onCopy` + - `TableWidget`: `virtual void setSort(std::vector const& chain) = 0;` and `virtual void setOnSort(std::function onSort) = 0;` (replacing both), and with default empty bodies `setMenu(std::function(std::optional)> entries, std::function, std::vector)> onChoose)`, `setOnColumnMove(…)`, `setOnColumnResize(…)`, `setOnFilter(…)`, `setFiltered(std::vector const&)`, `setColumnGroups(std::vector const&)`, `setOnGroupFilter(…)`, `setOnCopy(Action)` (the status line's `setStatus` and `setOnRetry` are Part D's) + - `IViewBackend::copyText(std::string_view text) -> bool` (default `false`: no clipboard) + - `RecordingBackend` drivers: `clickHeader(int table, std::size_t column, HeaderClick click)`, `menuOf(int table, std::optional column) -> std::vector`, `chooseMenu(int table, std::optional column, std::vector const& labels)` (labels from the top, resolved to the entry path), `moveColumn(int table, std::size_t column, std::size_t position)`, `resizeColumn(int table, std::size_t column, std::optional units)`, `clickFilter(int table, std::size_t column)`, `clickGroupFilter(int table, std::size_t group)`, `pressCopy(int table)`, `clipboard() -> std::vector const&`; properties `sort` (`"0:Ascending,2:Descending"`, `none` when empty), `filtered` (`[0,2]`), `columnGroups` (`[Company:0+3]`, label, first visible place, count) + - `ConformanceProbe`: `clickHeader(Widget& table, std::size_t column, HeaderClick click)`, `headerMenu(Widget& table, std::optional column) -> std::vector`, `chooseHeaderMenuEntry(Widget& table, std::optional column, std::vector const& path)`, `moveColumn(Widget& table, std::size_t column, std::size_t position)`, `headerText(Widget const& table, std::size_t column) -> std::string`, `headerName(Widget const& table, std::size_t column) -> std::string`, `headerDescription(Widget const& table, std::size_t column) -> std::string`, `clipboardText() -> std::string` +- Backend behaviour both backends implement: a header's menu opens on a right click on the header, on `Shift+F10` or the context-menu key with the header focused, and from a `⋯` button at the header row's trailing end for the table's menu (accessible name `Table options`); `Shift`+click, `Shift+Enter` and `Shift+Space` report `HeaderClick::Shift`; `Ctrl+Shift+Left` and `Ctrl+Shift+Right` on a focused header report `onColumnMove(column, position ∓ 1)` among the arrangement's visible columns; on Qt Quick dragging a header onto another reports the move, dragging a header's trailing edge reports `onColumnResize` in units, a double click on the edge reports `nullopt`; a filterable header draws a filter button, filled while `setFiltered` names it; a group header row is drawn over side-by-side group columns with its own filter button; `Ctrl+C` on a focused table reports `onCopy`. The terminal header becomes focusable: `Up` on the first row focuses the header at the sorted column (else the first), `Left`/`Right` move along it, `Enter`/`Space` sort, `Shift+Enter` extends, `Ctrl+Shift+Left/Right` move, `Shift+F10` opens its menu, `Down` returns to the rows; the terminal draws no drag or resize handle. + +- [ ] **Step 1: Write the failing conformance cases** in `include/morph/ui/testing/backend_conformance.hpp`, appended to `conformanceCases()` in this order, each with its control comment: + +```cpp +// Control: report every header click as Plain. +{"table: a Shift-click on a sortable header reports Shift, a plain click Plain", /* mount a Table of columns + {Name sortable, Mass sortable}, onSort recording (column, click); probe.clickHeader(table, 1, HeaderClick::Shift); + probe.clickHeader(table, 0, HeaderClick::Plain); expect {(1, Shift), (0, Plain)} */}, +// Control: draw only the chain's first key. +{"table: a chain of two keys marks both headers with their places", /* sort = {{1, Descending}, {0, Ascending}}; + headerText(table, 0) == "Name ▲2", headerText(table, 1) == "Mass ▼1", + headerDescription(table, 0) == "Sorted ascending, key 2 of 2"; with sort = {{0, Descending}}: + headerText(table, 0) == "Name ▼", headerDescription(table, 0) == "Sorted descending" */}, +// Control: pass the clicked header's index plus one to onMenu. +{"table: a header's menu shows its entries and choosing one reports the column and the path", /* menu returns + {Sort ascending, separator, Hide column} for column 1; headerMenu(table, 1) labels == those three; + chooseHeaderMenuEntry(table, 1, {2}) reports (1, {2}) */}, +{"table: the table's menu is the menu with no column", /* chooseHeaderMenuEntry(table, nullopt, {0}) reports (nullopt, {0}) */}, +// Control: report the move's target as the source. +{"table: a column moved by the user reports its column and new place", /* moveColumn(table, 0, 1) reports (0, 1) */}, +// Control: name a filtered header as an unfiltered one. +{"table: a filtered column's header is named so", /* columns filterable; filtered = {0}: headerName(table, 0) == + "Name, filtered", headerName(table, 1) == "Mass" */}, +{"table: Ctrl+C on a focused table reports a copy", /* press(table, "Ctrl+C") calls onCopy once */}, +// Control: copyText returning true without writing. +{"backend: copyText puts its text on the clipboard", /* backend().copyText("a\tb") == true; clipboardText() == "a\tb" */}, +``` + +Write each case's body with the assertions in its comment. Add to `tests/qt_quick/test_qt_quick_table.cpp`: + +```cpp +// Control: report the edge drag's width in pixels instead of units. +TEST_CASE("qt_quick table: dragging a header's trailing edge reports units, a double click on it none", "[qt_quick][table]") +// header 0 fixed(12); drag its trailing edge right by 4 units (theme unit width × 4): onColumnResize == (0, 16); +// double-click the edge: onColumnResize == (0, nullopt). +// Control: ignore a drag that ends on another header. +TEST_CASE("qt_quick table: dragging a header onto another reports the move", "[qt_quick][table]") +// drag header 0 onto header 2 of three: onColumnMove == (0, 2). +``` + +and to `tests/tui/test_tui_table.cpp`: + +```cpp +// Control: leave the header unfocusable. +TEST_CASE("tui table: Up on the first row focuses the header; Left, Right, Shift+Enter and Ctrl+Shift+Right act on it", "[tui]") +// three sortable columns, sort {{1, Ascending}}: Up on row 0 focuses header 1; Right focuses header 2; +// Shift+Enter reports (2, Shift); Ctrl+Shift+Left reports onColumnMove(2, 1); Down returns to row 0. +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `cmake --build --preset cl-debug --target morph_tests` then `cmake --build build/cl-qt-debug --target morph_qt_quick_tests morph_tui_tests` +Expected: compile errors for `HeaderClick`, `TableColumnGroup` and the probe operations. + +- [ ] **Step 3: Implement** the interfaces and behaviour above in every listed file. `mount.hpp` binds `sort`, `filtered` and `columnGroups` as it binds `sort` today, wires each handler with `event(...)`, calls `setMenu` only when `menu` is set and `setOnColumnResize` only when `onColumnResize` is set, and keeps calling `setSort` only for a table with a sortable column. `copyText` writes `QGuiApplication::clipboard()` on Qt Quick, an OSC 52 sequence on the terminal, and a record on `RecordingBackend`; each backend's conformance probe reads back what its `copyText` wrote. + +- [ ] **Step 4: Run them to see them pass** + +Run: `ctest --test-dir build/cl-debug -R "conformance" --output-on-failure`; `ctest --test-dir build/cl-qt-debug -R "qt_quick table|tui table|conformance" --output-on-failure` +Expected: all pass, the existing table cases included. + +- [ ] **Step 5: Mutations, docs, commit.** Run each control; paste the failures. Update the docs listed. Run the docs build. Commit: `feat(ui): header menus, sort chains, column moves and the status line in every backend`. + +### Task 12c: Lowering sort chains, the layout, the menus and the status line's messages + +**Files:** +- Modify: `src/ui/document.cpp:292-334` (keys), `:3209-3290` (load checks); `src/ui/interpreter.cpp` (the table lowering Part D built: its `sort`, `onSort`, columns, element values and `TableStatus::message`), `include/morph/ui/interpreter.hpp:371-395` (`SessionOptions::maxCopyRows`) +- Test: `tests/test_ui_table_model.cpp`, `tests/test_ui_document.cpp`, `tests/bench_interpreter.cpp` +- Docs: `docs/spec/ui/document.md` (the `table` row: `layout`, `onLayoutChange`, `configurable`, `onFiltersChange`, a column's `locked` and `filterable`; element values `layout`, `visibleColumns`, `hiddenCount`), `docs/spec/ui/interpreter.md` (the lowering's menus and messages), `docs/spec/table/table.md` (D7 figure) + +**Interfaces:** +- Consumes: 12a, 12b; Part D's `TableRows`, `TableStatus`, the `status` key's routing of `message`, the state store's load and save, `Table::arrangement`, the lowering of both table kinds. +- Produces: + - document keys on `table`: `layout` (Expr; bound, it is controlled), `onLayoutChange` (Commands, `event` the new layout), `configurable` (a literal `true`, `false` or a list of `"reorder"`, `"hide"`, `"pin"`, `"resize"`; anything else is refused at load), `onFiltersChange` (Commands, `event` the new `filters`); on a column part: `locked` (Flag), `filterable` (a literal boolean, default `true`; a column without `id` is never filterable) + - element values `layout` (the stored shape), `visibleColumns` (ids in order), `hiddenCount` + - `SessionOptions::maxCopyRows = 10'000` (used by 12f) + - the status line's message channel, for every later task that refuses a gesture on a table (Parts F and G use it): `void ScreenCore::tableMessage(ElementState& table, std::string text)` in `src/ui/interpreter.cpp` — sets `TableStatus::message`, restarts the `kStatusMessageHold` timer on the session's scheduler, and under `status: false` sends `text` once to `InterpreterHost::notify(text, "warning")` instead; the user's next gesture on the table (a header click, a menu choice, a selection, a scroll, a key) clears it +- The lowering's rules: `onSort(column, click)` maps `Plain` to `SortGesture::Click` and `Shift` to `Extend`, applies `applySortGesture` with the table's bound (8, or Task 16's), and on success calls `TableRows::setSort`, sets the held `sort` (when not bound) and runs `onSortChange`; a refusal goes to `tableMessage`. Menu commands map to the same gestures and to 12d, 12e and 12f's openers; `menu` and `onMenu` are set unless `configurable` is `false`. A layout change updates `Table::arrangement`, calls `TableRows::visible` with the new visible columns, sets the held `layout` (when not bound), runs `onLayoutChange` and marks the state unsaved; the stored state's `layout` member is applied before the first rows, as `sort` and `filters` are, and a bound `layout` is neither applied nor saved. A `rows` table's `TableStatus::shown` holds while its `state` or `message` is not empty (a model-backed table's is always shown, Part D); a literal `status: true` shows it always. Every refusal is shown (invariant 31). + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_table_model.cpp` (tag `[ui][table][config]`), over a `rows` table document `kNine` with `"id": "t"`, nine sortable columns `c1`…`c9` and three rows, unless a test says otherwise: + +```cpp +// Mutation: the lowering mapping HeaderClick::Shift to Click. +TEST_CASE("sort chains: a plain click replaces, Shift adds and flips, a key past the bound is refused", "[ui][table][config]") { + Harness h; + auto screen = h.mount(kNine); + int const table = h.backend.all("Table").front(); + h.backend.clickHeader(table, 0, ui::HeaderClick::Plain); + h.backend.clickHeader(table, 1, ui::HeaderClick::Shift); + h.backend.clickHeader(table, 0, ui::HeaderClick::Shift); + h.settle(); + CHECK(readJson(*screen, R"({"ref":"t.sort"})") == + R"([{"column":"c1","direction":"desc"},{"column":"c2","direction":"asc"}])"); + CHECK(h.backend.prop(table, "sort") == "0:Descending,1:Ascending"); + for (std::size_t column = 2; column < 9; ++column) { h.backend.clickHeader(table, column, ui::HeaderClick::Shift); } + h.settle(); + CHECK(h.backend.prop(table, "sort") == + "0:Descending,1:Ascending,2:Ascending,3:Ascending,4:Ascending,5:Ascending,6:Ascending,7:Ascending"); + CHECK(statusOf(h).message == "A table sorts by at most 8 columns"); + CHECK(statusOf(h).shown); + h.backend.clickHeader(table, 4, ui::HeaderClick::Plain); + h.settle(); + CHECK(readJson(*screen, R"({"ref":"t.sort"})") == R"([{"column":"c5","direction":"asc"}])"); + CHECK(statusOf(h).message.empty()); + CHECK_FALSE(statusOf(h).shown); +} + +// Mutation: "Add to sort" mapped to SortGesture::Click. +TEST_CASE("sort chains: the header menu builds a chain without a modifier", "[ui][table][config]") +// chooseMenu(table, 0, {"Sort descending"}); chooseMenu(table, 2, {"Add to sort"}); chooseMenu(table, 0, {"Remove from sort"}): +// sort == [{"column":"c3","direction":"asc"}]; onSortChange ran three times. + +// Mutation: the lowering applying a menu gesture without updating `visibleColumns`. +TEST_CASE("column layout: hiding, moving, pinning and resetting from the menus change the layout and the visible columns", + "[ui][table][config]") +// chooseMenu(table, 2, {"Hide column"}); chooseMenu(table, 1, {"Move left"}); chooseMenu(table, 1, {"Pin column"}); +// t.visibleColumns == ["c2","c1","c4",…,"c9"]; t.hiddenCount == 1; +// t.layout == {"order":["c2","c1","c3",…],"hidden":["c3"],"pinned":1,"widths":{}}; +// resizeColumn(table, 0, 20): t.layout.widths == {"c2":20}; resizeColumn(table, 0, nullopt): widths == {}; +// chooseMenu(table, nullopt, {"Reset columns"}): t.layout == the declared order, nothing hidden, pinned 0; +// onLayoutChange ran six times with `event` the new layout. + +// Mutation: `configurable` ignored for menus (the hide entry is present). +TEST_CASE("column layout: configurable and locked limit what the user can do, and each refusal is shown", "[ui][table][config]") +// "configurable": ["hide"], c1 "locked": true: menuOf(table, 1) has no "Move left"; moveColumn(table, 1, 0) leaves +// visibleColumns unchanged and status message "This table's columns cannot be moved"; menuOf(table, 0)'s "Hide column" +// is disabled and moveColumn(table, 0, 1) shows "C1 is locked"; "configurable": false: menuOf(table, 1) is empty and no +// "⋯" table menu is offered (menuOf(table, nullopt) empty). + +// Mutation: a status message never cleared by a gesture. +TEST_CASE("status line: a refusal is shown once, until the next gesture or six seconds", "[ui][table][config]") +// refusal at the bound shows the message; h.advance(5'999ms): still shown; h.advance(1ms): message empty and +// statusOf(h).shown false (a rows table with nothing else to say takes no room); refuse again then +// clickHeader(table, 0, Plain): message gone. "status": false: the refusal reaches h.host.log as +// "notify warning A table sorts by at most 8 columns" exactly once, and statusOf(h).shown stays false. + +// Mutation: the lowering applying a stored layout after the first query. +TEST_CASE("persisted layout: a stored layout applies before the first query, a change saves once after the debounce", + "[ui][table][config]") +// Harness(store) over a MemoryTableStateStore holding, under the table's key, part "state", +// {"version":1,"layout":{"order":["c2","c1"],"hidden":["c3"],"pinned":0,"widths":{"c2":20}}}: before any rows, +// t.visibleColumns starts with "c2","c1" and lacks "c3"; chooseMenu(table, 0, {"Hide column"}) three times inside 500 ms +// then h.advance(500ms): the store received exactly one save whose layout hides those columns. With `layout` bound to +// state `lay`, the stored layout is neither applied nor saved and the sort part still is. + +// Review Focus 1. Mutation: hiding a column removing it from the query's sort. +TEST_CASE("hiding a sorted and filtered column changes no row", "[ui][table][config]") +// kModelTable over samplesDescription(5), sorted by `name`, filter `{"columns":{"name":{"include":[{"contains":"a"}]}}}`: +// h.replyPage(pageJson(0, 50, 50, cols)) holds rows R; chooseMenu(table, 1, {"Hide column"}): no new Table.samples call +// whose sort or filters differ (any call sent is a column fetch of held rows: lastQuery(h)'s `sort` and `filters` equal +// the previous ones and its `columns` lacks "name"); +// the view's row keys equal R; showing it again restores the header mark ("sort" prop names its index) and `filtered`. + +// Review Focus 2 (message half). Mutation: refusing Shift on a column already in the chain at the bound. +TEST_CASE("sort chains: at the bound a key in the chain still flips, with no message", "[ui][table][config]") +// eight keys held; clickHeader(table, 3, Shift): sort's 4th key flips to desc; status message empty. +``` + +In `tests/test_ui_document.cpp`: + +```cpp +// Mutation: the loader accepting any `configurable` literal. +TEST_CASE("table: configurable, locked and filterable are literals the loader checks", "[ui][document]") +// "configurable": ["reorder","sort"] is refused naming `sort`; "configurable": {"ref":"x"} refused; a column +// "filterable": {"ref":"x"} refused; "locked": true and "filterable": false accepted; "onLayoutChange" and +// "onFiltersChange" take commands. +``` + +In `tests/bench_interpreter.cpp`: + +```cpp +TEST_CASE("interpreter bench: a third sort key on 100,000 rows", "[.benchmark][interpreter]") +// a rows table of 100,000 rows sorted by two keys; BENCHMARK the Shift-click that adds a third key to the settled +// view (D7, budget ≤ 50 ms). +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `cmake --build --preset cl-debug --target morph_tests && ctest --test-dir build/cl-debug -R "sort chains|column layout|status line|persisted layout|hiding a sorted|configurable, locked" --output-on-failure` +Expected: FAIL — unknown keys `configurable` and `locked` refused at load, and no `layout` element value. + +- [ ] **Step 3: Implement** the keys, the checks, the element values, `maxCopyRows`, `tableMessage` and the lowering's rules above. + +- [ ] **Step 4: Run them to see them pass** + +Run: the Step 2 command, then `ctest --test-dir build/cl-debug -R "ui" --output-on-failure` +Expected: all pass, the existing `test_ui_table.cpp` and `test_ui_interpreter_table.cpp` cases included. + +- [ ] **Step 5: Record D7.** Run: `cmake --preset cl-release && cmake --build --preset cl-release --target morph_tests && build/cl-release/tests/morph_tests.exe "interpreter bench: a third sort key on 100,000 rows"`. Expected: a mean ≤ 50 ms; record it, dated, in table.md's D7 row. + +- [ ] **Step 6: Mutations, docs, commit.** Commit: `feat(ui): configure a table from its header`. + +### Task 12d: The filter menu + +**Files:** +- Create: `include/morph/ui/table_filter_menu.hpp` (the menu's view tree and its state, built by the lowering) +- Modify: `include/morph/ui/table_rows.hpp` (`TableRows::distinctValues`), `src/ui/interpreter.cpp`, `src/ui/document.cpp` (a `rows` table's `groups` parts `{id, label, columns}`), `include/morph/ui/interpreter.hpp` (`SessionOptions::tableFilterDebounce`) +- Test: `tests/test_ui_table_model.cpp`, `tests/net/test_push_tables.cpp` +- Docs: `docs/spec/ui/document.md` (a `rows` table's `groups`), `docs/spec/qt_quick/backend.md` (the filter menu), table.md (B3 and D9 figures) + +**Interfaces:** +- Consumes: 12a's tick model, `planValueFilter`, `valueFilter`, `classifyFilter`, `canonicalOperand`, `groupValueFilter`, `conditionOps`, `conditionEntry`; 12b's `onFilter`, `onGroupFilter`, `filtered`, `columnGroups`; Task 8a's `DistinctRequest`/`DistinctValues` and `DateBucket` (a bucket's `text` is its ISO form, `2026`, `2026-03`, `2026-03-14`, and its operand `a..b`, as Part C's tests pin them); Task 8c's `DistinctValue::{dimension, unit}`, `ui::parseMeasure`, `ui::measureOperand`; Part D's `ScreenCore::tableMessage`. +- Produces: + - `TableRows::distinctValues(table::DistinctRequest request) -> async::Completion>` (pure virtual; `EngineRows` answers at once through `Engine::distinctValues`; `WindowRows` sends a query carrying `distinct` through the session's query cache, so reopening a menu sends nothing; `TableCapabilities::distinct == false` → `Unavailable`, and the menu shows conditions only) + - `SessionOptions::tableFilterDebounce = std::chrono::milliseconds{250}` + - the menu, a `ui::Dialog` (test id `table_filter`, title the column's or group's label) the lowering mounts beside the table, with test ids: `table_filter_search` (TextInput), `table_filter_all` (`(Select all)` Checkbox), `table_filter_blanks` (`(Blanks)`, only while `blanks > 0`), `table_filter_value_` (Checkbox, label `{text} ({count})`, in the reply's order, `i` from 0), `table_filter_heading_` (Text, a `Measure` column's dimension heading before its first value), `table_filter_open_` (Button `▸`, a date bucket that opens to its months or days), `table_filter_more` (Button, the next 200), `table_filter_note` (Text: `mayGrow()` while `partial`, `unreadable(invalid)` while `invalid > 0`, `tooManyValues()` after an Apply whose plan is `Disabled`), `table_filter_op_1` / `_2` (Select of `conditionLabel`s), `table_filter_operand_1` / `_2` and `table_filter_second_1` / `_2` (a TextInput, or on a `Date`/`DateTime` column a `DateTimeInput` in `Date`/`DateTime` mode — the date picker; the second only for `between`), `table_filter_case` (Checkbox `match case`, Text columns), `table_filter_combine` (Select `and`/`or`), `table_filter_unit_1` / `_2` (Select of the column's units, `Measure` only), `table_filter_match` and `table_filter_count` (group menus: `in any column`, `in every column`, `in at least N columns`, `in exactly N columns`, `all of them, each in a column`; the last only while 2 to k values are ticked), `table_filter_apply` (Button, disabled while nothing is ticked), `table_filter_clear` (Button `Clear filter`) + - the `Display unit` header submenu (Measure columns) lists, per dimension the column holds, its units (from the description's `units`), checked on the chosen one; choosing sets `layout.units[column][dimension]` +- Rules: + - **Values or conditions.** A column's filter is either its values part or its conditions part (12a's `classifyFilter`). Reopening the menu shows the part the stored filter is, the other part empty; editing one part clears the other's draft. A user who wants both writes conditions only. + - **When it applies.** Ticks and "(Select all)" are recorded as `TickOp`s and applied only by Apply, which resolves them (paging each search's values at `maxLimit`), plans the side, and sets the filter; a `Disabled` plan applies nothing and puts `tooManyValues()` in the note. Conditions apply live: an edit of an operator, an operand, the unit, `match case` or `and`/`or` is sent `tableFilterDebounce` after the last such edit, provided every operand canonicalises (`canonicalOperand`) and a `between` has both ends; otherwise nothing is sent and the operand shows its error. Apply on the conditions part sends a pending edit at once. A sent filter calls `TableRows::setFilter`, sets the held `filters` and runs `onFiltersChange`. + - **The values list** asks `DistinctRequest{column, search, offset, limit = 200}`, the search sent `tableFilterDebounce` after the last keystroke; a group menu asks with `group`. A `Measure` column's list draws `table_filter_heading_` before each dimension's first value. + - **Dates.** A `Date`/`DateTime` column's list is a tree: it asks `bucket = Year`; opening a year asks `bucket = Month, search = "-"`; opening a month asks `bucket = Day, search = "--"`, each paged at 200 like any list. A ticked bucket is a member of the `Ranges` form: one `between` per bucket. The conditions take dates from the picker. + - **Groups.** A declared group's menu pools its values (`DistinctRequest::group`). A user-formed group (a layout group, index decision #26) has no values list: its menu shows the conditions and the match line only, and its filter is held under its layout id and sent as `GroupFilter{columns}`. + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_table_model.cpp`: + +```cpp +// Mutation: the menu counting over the view (the column's own filter applied). +TEST_CASE("filter menu: values are counted over what the other filters leave, and ticks build the smaller side", + "[ui][table][config]") +// rows table `t` over 7 rows, status: Open×3, Closed×2, Held×1, empty×1; site: A×5, B×2; filter on site eq "A" leaves +// Open×2, Closed×2, empty×1 → clickFilter(table, ): table_filter_value_0 label "Closed (2)", _1 "Open (2)", +// no "Held"; table_filter_blanks label "(Blanks)"; untick "Open", Apply: t.filters.columns.status == +// {"combine":"any","exclude":[{"oneOf":["Open"]}],"include":[]} — then reopen: "Open (2)" is listed unticked; +// onFiltersChange ran once; the header's `filtered` prop names the status column. chooseMenu(table, nullopt, +// {"Clear all filters"}): t.filters == {"columns":{},"groups":[]}, `filtered` is empty, onFiltersChange ran twice. + +// Mutation: Apply writing the ticks and the conditions into one entry list. +TEST_CASE("filter menu: values and conditions are exclusive, and reopening shows the stored part", "[ui][table][config]") +// status (Text): tick only "(Blanks)", Apply → include [{"isEmpty":true}]; reopen: table_filter_blanks checked, every +// value unticked, table_filter_op_1 empty. Then choose "contains", operand "pe": after 250 ms t.filters.columns.status == +// {"combine":"any","include":[{"contains":"pe"}],"exclude":[]} and the ticks are gone; reopen shows the condition and +// (Select all) ticked. A Custom column `code` filtered {"include":[{"isEmpty":true}]} reopens with table_filter_op_1 +// "is empty" and no values list. + +// Mutation: the search sent on every keystroke (no debounce). +TEST_CASE("filter menu: a search typed quickly is one distinct request after the pause", "[ui][table][config]") +// kModelTable over samplesDescription(5), first page replied: open the menu on `name` (one Table.samples call with +// `distinct`, replied by hand); type "a","ab","abc" 50 ms apart in table_filter_search; +// h.advance(249ms): no new call; h.advance(1ms): exactly one more, whose body's distinct.search == "abc". (D9 gate) + +// Mutation: the condition sent on every keystroke, or a half-typed between sent. +TEST_CASE("filter menu: a condition applies once after typing pauses, and an incomplete between is not sent", + "[ui][table][config]") +// kModelTable, `name` (Text), op "contains": type ten letters 150 ms apart into table_filter_operand_1: +// h.advance(249ms) after the last: no new page request; h.advance(1ms): exactly one, filtering contains the ten +// letters (B4 gate). c3 (Decimal), op "between", operand_1 "2", no second: h.advance(1s): no request; second "5": one, +// include [{"between":["2","5"]}]; edit operand_1 to "3" and press Apply at once: the request goes without waiting. + +// Mutation: the note line ignoring `partial`. +TEST_CASE("filter menu: a partial list says it may grow and invalid cells are counted under it", "[ui][table][config]") +// kModelTable; reply the distinct request with {"values":[{"text":"x","operand":"x","count":4}],"distinct":1, +// "blanks":0,"invalid":3,"partial":true}: table_filter_note text == "The list may grow while data loads\n3 cells could not +// be read"; no checkbox for the invalid cells. + +// Mutation: "(Select all)" under a search ticking only the values the page shows. +TEST_CASE("filter menu: Select all under a search covers every match, and both sides over the bound refuse", "[ui][table][config]") +// rows table of 25,000 rows, `code` "c00000".."c24999": untick (Select all), search "c0" (10,000 matches, 200 shown), +// tick (Select all), Apply: include [{"oneOf": the 10,000 operands "c00000".."c09999"}], not the 200 shown — resolved by +// ten distinct requests of limit 1000. Untick (Select all), tick (Select all) under "c1" (10,000) and under "c2" (5,000), +// Apply: T = 15,000, U = 10,000 → exclude [{"oneOf": "c00000".."c09999"}], named by paging every value. Untick +// (Select all), tick it under "c1", "c00" and "c01" (T = 12,000, U = 13,000), Apply: table_filter_note == "Too many +// values to list; filter with a condition" and t.filters is unchanged. + +// Mutation: conditions offered by Text for every kind; an operand sent as typed. +TEST_CASE("filter menu: conditions follow the column's kind, and typed operands are canonical", "[ui][table][config]") +// Integer column: table_filter_op_1 options "=","≠","<","≤",">","≥","between"; Bool column: no table_filter_op_1; +// Custom column: "is empty","is not empty"; a Date column: table_filter_operand_1 is a DateTimeInput (Date mode). +// Under a session locale "de": a Decimal column, ">" and operand "1.234,5" → include [{"gt":"1234.5"}]; a Measure column +// `weight` (mass: g, kg): table_filter_unit_1 offers "g","kg"; ">" with "0,3 kg" → [{"gt":"0.3 kg"}]; ">" with "250" and +// unit "g" → [{"gt":"250 g"}]; "12,5 xx" shows an error on the operand and sends nothing. + +// Mutation: the Measure list drawn without dimension headings. +TEST_CASE("filter menu: a Measure column's values are grouped under a heading per dimension", "[ui][table][config]") +// kModelTable; distinct reply values 0.3 kg (mass), 300 g counted as one with it, 0.5 % (ratio): table_filter_heading_mass +// precedes table_filter_value_0, table_filter_heading_ratio precedes the ratio value. + +// Mutation: the group menu sending `together` as one oneOf. +TEST_CASE("filter menu: a group's values are pooled and the match line builds the quantifier", "[ui][table][config]") +// rows table with "groups": [{"id":"company","label":"Company","columns":["companyA","companyB","companyC"]}] and the five +// rows of the spec's truth table; the table's `columnGroups` prop is "[Company:0+3]" (the three stand side by side) and +// clickGroupFilter(table, 0) opens the group's menu; after moving companyB to the end it is "[]" and the menu is reached +// from the header and table menus only; menuOf(table, ) has "Filter Company, 3 columns…"; its values list counts +// F 3, S 2, X 3, Y 1, Z 1 (rows having it in any column); tick F and S, match "all of them, each in a column", Apply: +// filters.groups == [{"group":"company","mode":"together","include":[{"eq":"F"},{"eq":"S"}]}] and the view holds rows 1 +// and 3 only; match "in at least N columns", N 2: rows 1, 2, 3. + +// Critical review item. Mutation: a user-formed group's menu sending DistinctRequest::group with its layout id. +TEST_CASE("filter menu: a user-formed group offers conditions and the match line, no values list", "[ui][table][config]") +// columns a, b (Text) formed into "Pair" (12e's chooser, id g1): chooseMenu(table, , {"Filter Pair, 2 columns…"}): +// no table_filter_search, table_filter_all or table_filter_value_* widget, and no distinct request is sent; +// table_filter_op_1 offers the Text conditions and table_filter_match the four quantifiers (not "all of them, each in a +// column"); "equals" "F" with "in every column", after 250 ms: the request's filters.groups == +// [{"columns":["a","b"],"match":"all","include":[{"eq":"F"}]}]; t.filters keeps it under "g1". + +// Mutation: the date tree filtering months on the client (an opened year past 200 months shows nothing). +TEST_CASE("filter menu: a date column lists years, then a year's months, then a month's days", "[ui][table][config]") +// kModelTable over a description whose c3 is a Date column: opening its menu sends distinct {"bucket":"year"}; the reply +// lists 2025, 2026; table_filter_open_1 (2026) sends {"bucket":"month","search":"2026-"}; opening March sends +// {"bucket":"day","search":"2026-03-"}; tick March and 2025, Apply: include [{"between":["2025-01-01","2025-12-31"]}, +// {"between":["2026-03-01","2026-03-31"]}] with combine "any". + +// Mutation: the unit choice written to the column's own unit instead of the layout. +TEST_CASE("display unit: the header menu writes the layout's units", "[ui][table][config]") +// Measure column `weight` (dimension mass, units g and kg): chooseMenu(table, , {"Display unit", "mass: g"}): +// t.layout.units == {"weight":{"mass":"g"}}; no new page request. +``` + +In `tests/net/test_push_tables.cpp` (Linux, recorded, not gates): + +```cpp +TEST_CASE("table benchmark: a sort change and a text filter over a socket with 50 ms each way", "[.benchmark][table]") +// B3: the server's handler waits 50 ms before running each request and 50 ms before sending each reply; a header sort on +// `wide` (100,000 × 300) to the accepted page (≤ 150 ms), and the last key of a text filter to the accepted page +// (≤ 250 ms + 150 ms). +TEST_CASE("table benchmark: a server-mode values list from open to shown", "[.benchmark][table]") // D9, ≤ 150 ms +``` + +- [ ] **Step 2: Run them to see them fail.** Run: `cmake --build --preset cl-debug --target morph_tests && ctest --test-dir build/cl-debug -R "filter menu|display unit" --output-on-failure`. Expected: FAIL — no `table_filter` dialog is mounted. + +- [ ] **Step 3: Implement** `TableRows::distinctValues` in both implementations, `tableFilterDebounce`, the `groups` key, the menu and the `Display unit` submenu per the rules above. + +- [ ] **Step 4: Run them to see them pass.** The Step 2 command, then all `[ui]` cases. Expected: all pass. + +- [ ] **Step 5: Record B3 and D9** on Linux: `cmake --preset linux-everything && cmake --build build/linux-everything --target morph_net_tests && build/linux-everything/tests/net/morph_net_tests "table benchmark: a sort change*" "table benchmark: a server-mode values list*"`; write the figures into table.md's B3 and D9 rows. + +- [ ] **Step 6: Mutations, docs, commit.** Commit: `feat(ui): the filter menu with values, conditions, groups and units`. + +### Task 12e: The column chooser and the sort dialog + +**Files:** +- Create: `include/morph/ui/table_dialogs.hpp` (both dialogs' view trees) +- Modify: `src/ui/interpreter.cpp` +- Test: `tests/test_ui_table_model.cpp` +- Docs: `docs/spec/qt_quick/backend.md` (the dialogs) + +**Interfaces:** +- Consumes: 12a's gestures and `FormGroup`; 12c's lowering. +- Produces: `Columns…` opens a `ui::Dialog` `table_columns` listing every column of the description in layout order (a companion column is not listed once Task 15 marks companions): per column a Checkbox `table_columns_show_` (its label; disabled for a locked column and for the only visible one), Buttons `table_columns_up_` and `table_columns_down_`; Buttons `table_columns_reset` (`Reset columns`) and `table_columns_group` (`Group columns…`, offered while `hide` or `reorder` is allowed), which shows `table_group_label` (TextInput), `table_group_column_` Checkboxes (after the first is ticked, only columns of its kind stay enabled), `table_group_create` (Button, enabled at 2 to 16 ticked and a label); `Sort…` opens `table_sort` listing the chain: per key `i` from 0 a Select `table_sort_column_` (sortable columns), Select `table_sort_direction_` (`ascending`, `descending`), Buttons `table_sort_up_`, `table_sort_down_`, `table_sort_remove_`; Button `table_sort_add` (`Add key`, adds the first sortable column not in the chain; at the bound it refuses with the message). Both dialogs apply each change as it is made, through the same gesture functions; a `Done` button and a dismissal close them. + +- [ ] **Step 1: Write the failing tests:** + +```cpp +// Mutation: the chooser listing only the visible columns. +TEST_CASE("column chooser: every column in order, with show, move and reset", "[ui][table][config]") +// kNine with c3 hidden: table_columns_show_c3 is listed unchecked in third place; check it: t.hiddenCount == 0; +// table_columns_down_c1: t.visibleColumns starts "c2","c1"; table_columns_reset: declared order. + +// Mutation: group creation accepting columns of two kinds. +TEST_CASE("column chooser: Group columns forms a group of one kind, which the layout keeps", "[ui][table][config]") +// columns a, b (Text) and n (Integer): tick table_group_column_a → table_group_column_n disabled; tick b, label +// "Pair", create: t.layout.groups == [{"id":"g1","label":"Pair","columns":["a","b"]}]; menuOf(table, ) holds +// "Filter Pair, 2 columns…". + +// Mutation: the sort dialog's move applying to the wrong neighbour. +TEST_CASE("sort dialog: keys in order with direction, move and remove; Add key refuses at the bound", "[ui][table][config]") +// sort c1 asc, c2 desc: table_sort_up_1 → [c2 desc, c1 asc]; table_sort_direction_0 "ascending" → c2 asc; +// table_sort_remove_1 → [c2 asc]; with eight keys table_sort_add leaves eight and shows "A table sorts by at most 8 columns". +``` + +- [ ] **Step 2: Run to see them fail.** `ctest --test-dir build/cl-debug -R "column chooser|sort dialog" --output-on-failure` after building `morph_tests`. Expected: FAIL — no `table_columns` dialog. +- [ ] **Step 3: Implement** the dialogs and their openers. +- [ ] **Step 4: Run to see them pass**, then all `[ui]` cases. +- [ ] **Step 5: Mutations, docs, commit.** Commit: `feat(ui): the column chooser and the sort dialog`. + +### Task 12f: Copy, and the screens + +**Files:** +- Modify: `include/morph/ui/table_rows.hpp` (`TableRows::fetchRows`), `src/ui/interpreter.cpp` +- Test: `tests/test_ui_table_model.cpp` +- Docs: `docs/spec/ui/interpreter.md` (copy), `docs/spec/ui/document.md` (`maxCopyRows` named in the copy rule) + +**Interfaces:** +- Consumes: 12a's `copyText`, `tableText`; 12b's `onCopy`, `copyText`; 12c's `tableMessage`; Part D's `ui::detail::standInCellText(table::Cell const&, table::TableColumn const&)` (replaced by Task 15's `ui::formatCell(table::Cell const&, table::TableColumn const&, ui::FormatContext const&, DisplayUnits const& = {})`, which copy calls with the session's `FormatContext` and the layout's display units) and its range selection (Task 10a), whose bound this task's Review Focus case holds at its edge. +- Produces: + - `struct FetchedRow { table::RowId key; std::int64_t viewRow = 0; std::vector cells; }`; `TableRows::fetchRows(std::vector keys, std::vector columns) -> async::Completion, table::TableError>>` (pure virtual; `EngineRows` answers at once; `WindowRows` sends keyed queries naming `columns`, at most `QueryLimits::maxKeys` keys each, through the read that makes no cache entry, and returns the rows the replies hold with their view rows; a key no reply holds is not returned) +- Rules: `Ctrl+C` and `Copy` copy the selected rows in view order by the visible columns in layout order, each cell's display text; with no row selected the focused cell's text; `Copy with headers` adds the labels' line. Over `maxCopyRows` selected keys it refuses with `copyBound` and copies nothing; otherwise it fetches the rows the window does not hold, writes `copyText`, calls `IViewBackend::copyText`, and shows `copied(rows, unloaded)`; every message goes through `tableMessage`. + +- [ ] **Step 1: Write the failing tests:** + +```cpp +// Mutation: copy writing rows in selection order instead of view order. +TEST_CASE("copy: selected rows as tab-separated display text in layout order, rows not held fetched", "[ui][table][config]") +// kModelTable over samplesDescription(5) sorted by name, c3 hidden, `name` moved first; the page holds view rows 0-29; +// select keys 9 and 2 (view rows 40 and 3); pressCopy(table): one keyed Table.samples request naming only the visible +// columns and keys [9] (the held row is not +// fetched); h.backend.clipboard().back() == "\n" with "\t" between cells in the arrangement's +// order; statusOf(h).message "Copied 2 rows". chooseMenu(table, nullopt, {"Copy with headers"}) prefixes the labels' line. +// No row selected: the focused cell's text alone. A rows table whose `name` column has its own `cell` (a badge reading +// `row.name`) copies the value's text, not the template's. + +// Mutation: unloaded cells copied as their placeholder text. +TEST_CASE("copy: unloaded cells copy as empty and are counted", "[ui][table][config]") +// kModelTable whose page reply names both rows in `unloadedRows` for level `recipe` (hand-written reply; c4 of that +// level): select the two rows, copy: their c4 cells are empty fields; +// message "Copied 2 rows, 2 cells were not loaded"; no request waits for the level. + +// Review Focus 3. Mutation: the bounds compared with `>=`. +TEST_CASE("range and copy at their bounds: 10,000 pass, 10,001 are refused", "[ui][table][config]") +// kModelTable, pageJson total 100,000, nothing held past the first page: selectTableRange(table, 0, 9'999): keysOf +// fetches pages naming no columns (each body's "columns" is []), t.selectedCount == 10000; pressCopy: copies 10,000 lines; +// selectTableRange(table, 0, 10'000): statusOf(h).message "A range selects at most 10,000 rows", selection unchanged; +// with 10,001 keys selected (a 10,000-row range, then a Ctrl-click), pressCopy: "A copy holds at most 10,000 rows", +// clipboard unchanged. +``` + +(`selectTableRange` is the `RecordingBackend` driver Part D's Task 10a adds for its range report. Replies the helpers of `tests/ui_table_support.hpp` do not build (distinct values, keyed replies, `unloadedRows`) are written as JSON in the test.) + +- [ ] **Step 2: Run to see them fail.** `ctest --test-dir build/cl-debug -R "copy:|range and copy" --output-on-failure` after building `morph_tests`. Expected: FAIL — `fetchRows` does not exist; the clipboard stays empty (the range half of the bounds case passes already: it holds Part D's rule at its edge, and is here so the two bounds are tested side by side). +- [ ] **Step 3: Implement** `fetchRows` in both `TableRows` and the copy rule above. +- [ ] **Step 4: Run to see them pass**, then the whole of `morph_tests`, `morph_table_tests`, and in `build/cl-qt-debug` `morph_qt_quick_tests` and `morph_tui_tests`: all pass. +- [ ] **Step 5: The screens.** Existing tables now draw a `⋯` table-menu button, filter buttons on filterable headers and, on model-backed tables, a status line. This repository checks in no pixel baselines (`git ls-files '*.png'` lists none); screens are grabbed by `scripts/shoot_screens.sh` and `scripts/shoot_lims.sh`, which run the example clients with `QT_QPA_PLATFORM=offscreen --grab`. The user runs, on a Linux tree with Qt Quick and examples (`cmake --preset linux-everything && cmake --build build/linux-everything --target morph_examples_client morph_lims_client`), once on `master` and once on this branch: + `scripts/shoot_screens.sh /tmp/grabs- build/linux-everything/examples/client/morph_examples_client` and `scripts/shoot_lims.sh /tmp/lims-grabs- build/linux-everything/examples/client/morph_lims_client`, and compares each pair: the only differences are the three intended ones. The authoring goldens (`tests/ui/golden/*.json`) do not change, since no builder writes the new keys; confirm with `ctest --test-dir build/cl-debug -R "golden" --output-on-failure` (a golden that does change is regenerated with `MORPH_UPDATE_GOLDENS=1 build/cl-debug/tests/morph_tests.exe "[golden]"` and its diff reviewed in this commit). +- [ ] **Step 6: Mutations, docs, commit.** Commit: `feat(ui): copy a table's selection`. + +--- + +## Task 13: Collections through the same lowering (table.md step 8) + +**Spec:** "Collections and master-detail"; "Every surface on the pipeline" (the collection row); gate U5 (recorded); the collection half of the Tests row "One order on every surface". + +**Files:** +- Modify: `include/morph/forms/engine/collection_view.hpp:31-46` (`CollectionChrome::rows`), `:177-227` (`collectionRows` calls it when set), `include/morph/forms/engine/collection.hpp:366-430` (nothing new in the session; its `source()` is the list's `DataSource`), `src/ui/interpreter.cpp:7033-7055` (`lowerComposite` sets `chrome.rows`) and the table lowering (factored as below) +- Test: `tests/test_ui_table_model.cpp`, `tests/bench_interpreter.cpp`; existing `tests/test_ui_interpreter_forms.cpp` and `tests/test_forms_engine_collection_view.cpp` stay green unchanged +- Docs: `docs/spec/forms/engine.md` ("Collections": the rows through the table lowering), `docs/spec/ui/interpreter.md` (a collection's table), table.md (U5 figure) + +**Interfaces:** +- Consumes: Part D's `EngineRows(std::shared_ptr, RecordOf, table::EngineOptions)` over `session.source()` (a `RecordOf` reading the collection row's key and its cells as a record), the state store and `SessionOptions::application`; 12c's lowering. +- Produces: + - `forms::CollectionChrome::rows : std::function const& buttons)>` — replaces the rows table; `buttons` is empty when the view has no Open or row action + - in `src/ui/interpreter.cpp`: `struct TableSurface { std::shared_ptr rows; std::vector columns; std::function trailingCell; std::string trailingLabel; std::function cellTestId; ui::TableOptions options; table::TableStateKey stateKey; std::size_t maxSortKeys = 8; }`, `struct SurfaceColumn { std::string id; std::string label; table::ColumnKind kind; bool shown; bool sortable; bool filterable; }`, and `Node ScreenCore::lowerTableSurface(TableSurface surface, LowerCtx const& ctx, ElementState* element)` — the part of the table lowering below reading the node; the document `table` node builds a `TableSurface` and calls it + - a collection's surface: columns from `CollectionModel::columns()` (`id = field`, `label`, `kind`, `shown = !hidden`, sortable and filterable), the trailing buttons column (label empty), cell test ids `cell__`, options from `forms::detail::engine::rowsTableOptions(session)` (test id `collection_rows`, master-detail selection), state key `{application = SessionOptions::application, model = the node's model type, table = the collection's action, qualifier = the screen's id, part}` + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_table_model.cpp`: + +```cpp +// Mutation: the collection keeping the reply's order (chrome.rows not set by the interpreter). +TEST_CASE("collection: a collection and a rows table over the same rows show the same order and selection", "[ui][table]") { + Harness h; + auto screen = h.mount(kCollectionBesideTable); // the Samples collection and a rows table `t` over state `rows` + std::string const rows = R"([{"id":1,"name":"beta"},{"id":2,"name":"Alpha"},{"id":3,"name":"álpha"},{"id":4,"name":"Gamma"}])"; + h.reply(h.dispatcher.last("Lims_ListSamples"), R"({"rows":)" + rows + "}"); + int const collection = h.byTestId("Table", "collection_rows"); + int const table = h.find("Table", "testId", "t"); + h.backend.clickHeader(collection, 0, ui::HeaderClick::Plain); // Name: ID is hidden + h.backend.clickHeader(table, 1, ui::HeaderClick::Plain); // Name + h.settle(); + CHECK(rowKeys(h, collection) == rowKeys(h, table)); + CHECK(rowKeys(h, collection) != std::vector{"1", "2", "3", "4"}); + CHECK(h.text("cell_name_2") == "Alpha"); +} + +// Mutation: the master-detail selection lost when the engine reorders. +TEST_CASE("collection: a master-detail selection and its open row hold through a sort", "[ui][table]") +// master-detail view over the same rows: selectByUser(collection_rows, {3}) opens row 3; clickHeader(collection, 0): +// the table's selection prop still marks 3; detailForm visible; the editor's sampleId field reads 3. + +// Mutation: the collection's state saved under the screen's rows key. +TEST_CASE("collection: header and table menus are present and its arrangement persists under its action", "[ui][table]") +// Harness(store) over a MemoryTableStateStore: menuOf(collection_rows, 0) holds "Hide column"; hide it, h.advance(500ms): +// one save whose key is {application "lims", model "Lims_SampleModel", table "Lims_ListSamples", qualifier the screen's +// id, part "state"}. +``` + +Extend Task 11b's `"one order on every surface"` case with the collection over the same rows (the same sort gives the same key order on all three surfaces). In `tests/bench_interpreter.cpp`: + +```cpp +TEST_CASE("interpreter bench: a collection of 20,000 rows: mount, sort, one row edited", "[.benchmark][interpreter]") // U5 +``` + +- [ ] **Step 2: Run to see them fail.** `ctest --test-dir build/cl-debug -R "collection:" --output-on-failure` after building `morph_tests`. Expected: FAIL — the collection's order is the reply's. +- [ ] **Step 3: Implement** `CollectionChrome::rows`, `TableSurface`, `lowerTableSurface` and the collection's surface as above. +- [ ] **Step 4: Run to see them pass**, then `ctest --test-dir build/cl-debug -R "collection|forms engine collection|one order on every surface" --output-on-failure` and the whole of `morph_tests`. Expected: all pass, existing collection tests unchanged. +- [ ] **Step 5: Record U5** with `build/cl-release/tests/morph_tests.exe "interpreter bench: a collection of 20,000 rows*"` into table.md's U5 row. +- [ ] **Step 6: Mutations, docs, commit.** Commit: `feat(ui): collections on the table pipeline`. + +--- + +## Task 14: Saved and shared views (table.md step 9) + +**Spec:** "Saved views", "Persisted state" ("What is stored", "Two windows, two devices", "The framework's stores"), the Tests rows "Views are not lost between windows" and "Shared views", gate T2. Three commits. + +### Task 14a: The views document, its operations, and publishing on the server + +**Files:** +- Modify: `include/morph/table/state_store.hpp` (views document, operations, `ITableStateStore::publish`, `MemoryTableStateStore` sharing), `include/morph/ui/table_state_model.hpp` (`publish`; a user's `save` of `shared` refused), `include/morph/ui/table_state.hpp` (`PreferencesTableStateStore::shares()`, `publish`) +- Test: `tests/table/test_table_state.cpp`, `tests/test_ui_table_state.cpp` +- Docs: `docs/spec/ui/preferences.md` (the `shared` part kept with no person) + +**Interfaces:** +- Consumes: Part D's `TableStateKey`, `SaveOutcome`, `SaveResult`, `ITableStateStore`, `MemoryTableStateStore`, `TableStateModel`, `TableStateSaved`, `registerTableState`, `fitState`; Task 8d's `ColumnLayoutValue` (with `groups`, `units`). +- Produces (namespace `morph::table`): + - `struct SavedView { std::string name; ColumnLayoutValue layout; SortChain sort; FilterSpec filters; }` + - `enum class ViewsPart : std::uint8_t { Own, Shared }` + - `struct ViewsDocument { int version = 1; std::optional defaultView; bool defaultOnMount = false; std::vector views; }` (JSON `{"version", "default", "defaultOnMount", "views"}`; a `Shared` document writes neither `default` nor `defaultOnMount`) + - operations `AddView{SavedView}`, `ReplaceView{SavedView}`, `RenameView{std::string from; std::string to;}`, `RemoveView{std::string name}`, `SetDefaultView{std::optional name}`, `SetDefaultOnMount{bool on}`; `using ViewOperation = std::variant<…>`; JSON `{"add": view}`, `{"replace": view}`, `{"rename": {"from", "to"}}`, `{"remove": name}`, `{"default": name|null}`, `{"defaultOnMount": bool}` + - `enum class ViewError : std::uint8_t { NameTaken, NoSuchView, NotShared }`; `[[nodiscard]] std::expected applyViewOperation(ViewsDocument document, ViewOperation const& operation, ViewsPart part)` (`AddView` and `RenameView` onto a taken name → `NameTaken`; `ReplaceView`, `RenameView`, `RemoveView`, `SetDefaultView` naming no view → `NoSuchView`; `SetDefaultView`/`SetDefaultOnMount` on `Shared` → `NotShared`; removing the default view clears `defaultView`) + - `[[nodiscard]] std::expected readViews(std::string_view json)` (an unknown `version` → `InvalidSpec`), `[[nodiscard]] std::string writeViews(ViewsDocument const&, ViewsPart part)` + - `inline constexpr std::string_view kNoDocument = "-";` — an `expected` revision that stores only when nothing is stored under the key and answers `Conflict` otherwise; `MemoryTableStateStore` and `TableStateModel` honour it in `save` and `publish` (a `views` part saved by a table that loaded nothing sends it, so two windows that both start from no views cannot overwrite each other) + - `ITableStateStore::publish(TableStateKey const& key, ViewOperation const& operation, std::string expected) -> async::Completion` (virtual; default answers `Refused` with reason `"this store does not share views"`) + - `enum class Sharing : std::uint8_t { Private, Shared }`; `MemoryTableStateStore(exec::IExecutor& base, Sharing sharing)`; `shares()` is `sharing == Shared`; its `publish` applies the operation to the `shared` part under the `expected` rule +- Produces (namespace `morph::ui`): `struct TableStatePublish { table::TableStateKey key; std::string operation; std::string expected; }` (the key's `part` must be `"shared"`; `operation` the operation's JSON) → `TableStateSaved`, registered as `morph.tableState`'s action `publish` by `registerTableState`; the model applies it to the person-less `shared` entry (preferences `person` empty: no verified caller is ever empty, since the model refuses one with none), comparing `expected` with the head's revision (`Conflict` with the current revision when they differ), refuses `SetDefaultView`/`SetDefaultOnMount`, and refuses a caller with no verified person; its `save` with `part == "shared"` answers `Refused` (`"the shared views are written by publish"`); `PreferencesTableStateStore::shares()` returns `true` and its `publish` calls `morph.tableState.publish`. + +- [ ] **Step 1: Write the failing tests.** In `tests/table/test_table_state.cpp`: + +```cpp +// Mutation: AddView over a taken name appending a second view of that name. +TEST_CASE("views document: operations apply to what is stored, and names stay unique", "[table][state]") { + table::ViewsDocument doc; + auto const add = [](std::string name) { return table::AddView{table::SavedView{.name = std::move(name)}}; }; + doc = *table::applyViewOperation(doc, add("Open orders"), table::ViewsPart::Own); + doc = *table::applyViewOperation(doc, table::SetDefaultView{"Open orders"}, table::ViewsPart::Own); + CHECK(table::applyViewOperation(doc, add("Open orders"), table::ViewsPart::Own).error() == table::ViewError::NameTaken); + doc = *table::applyViewOperation(doc, add("Late"), table::ViewsPart::Own); + CHECK(table::applyViewOperation(doc, table::RenameView{"Late", "Open orders"}, table::ViewsPart::Own).error() == + table::ViewError::NameTaken); + doc = *table::applyViewOperation(doc, table::RemoveView{"Open orders"}, table::ViewsPart::Own); + CHECK_FALSE(doc.defaultView); + CHECK(table::applyViewOperation(doc, table::SetDefaultOnMount{true}, table::ViewsPart::Shared).error() == + table::ViewError::NotShared); + CHECK(table::writeViews(doc, table::ViewsPart::Shared) == + R"({"version":1,"views":[{"name":"Late","layout":{"order":[],"hidden":[],"pinned":0,"widths":{}},"sort":[],"filters":{"columns":{},"groups":[]}}]})"); + CHECK(table::readViews(R"({"version":2,"views":[]})").error().code == table::TableErrorCode::InvalidSpec); +} +``` + +(The expected `layout` text follows `ColumnLayoutValue`'s writer as Task 8d lands it; keep the rest exact.) + +In `tests/test_ui_table_state.cpp`, over the server harness Task 9 built for `morph.tableState` (`RemoteServer`, a recording `IPreferencesStore`, calls as a chosen token): + +```cpp +// Mutation: publish written under the caller's person (bob does not see it). +TEST_CASE("shared views: published by one principal, read by another", "[ui][table][state][remote]") +// alice publish {"add": {"name":"Mine",…}} with expected "-" → Saved "1"; bob load part "shared" → a document holding +// "Mine"; the store's calls show the write under person "" and the read under person "". + +// Mutation: the authoriser consulted with action "save" for a publish. +TEST_CASE("shared views: an authoriser that refuses publish refuses it by action id", "[ui][table][state][remote]") +// an IAuthorizer refusing (morph.tableState, "publish") for token "bob": bob's publish answers an error envelope and the +// store sees no write; alice's succeeds. + +// Mutation: save of part "shared" accepted. +TEST_CASE("shared views: a user's own save of the shared part is refused", "[ui][table][state][remote]") +// alice save {part "shared"} → {"result":{"outcome":"refused","reason":"the shared views are written by publish"}}; no write. + +// Mutation: publish ignoring `expected`. +TEST_CASE("shared views: a publish over a replaced revision is a conflict", "[ui][table][state][remote]") +// two publishes with expected "1"; the second → {"result":{"outcome":"conflict","revision":"2"}}. + +// Mutation: every state written as segments. (T2 gates) +TEST_CASE("table state: a full state is one entry and one call; a 20,000-value filter reads back equal through segments", + "[ui][table][state][remote]") +// a state of 300 columns, 8 sort keys and 32 filters: one store `set` and one entry; its byte count is logged (INFO) for +// table.md's T2 row; a state whose filter ticks 20,000 values: reads back byte-equal, every segment entry ≤ 12 KiB; the +// same two documents as the `views` part behave alike. +``` + +In `tests/net/test_push_tables.cpp` (Linux): `"shared views: a view published over a socket is read by a second client"` — the first two remote cases above, over a real socket. + +- [ ] **Step 2: Run to see them fail.** `cmake --build --preset cl-debug --target morph_tests morph_table_tests && ctest --test-dir build/cl-debug -R "views document|shared views|table state: a full state" --output-on-failure`. Expected: compile errors for `ViewsDocument`, `TableStatePublish`. +- [ ] **Step 3: Implement** the interfaces above. +- [ ] **Step 4: Run to see them pass**; on Linux `ctest --test-dir build/linux-everything -R "shared views" --output-on-failure`. +- [ ] **Step 5: Mutations, docs, commit.** Record T2's bytes and entry counts in table.md. Commit: `feat(table): saved views, their operations and publishing`. + +### Task 14b: Saved views on a table + +**Files:** +- Create: `include/morph/ui/table_views.hpp` +- Modify: `include/morph/ui/table_config.hpp` (view menu commands), `src/ui/interpreter.cpp` (the `Views` submenu, the name dialog, mount behaviour) +- Test: `tests/test_ui_table_state.cpp` +- Docs: `docs/spec/ui/interpreter.md` (saved views), `docs/spec/ui/document.md` (the table menu's views) + +**Interfaces:** +- Consumes: 14a; Task 9's state key, load at mount (Task 14 adds the `views` and `shared` keys to the one batched load of a screen's tables), save and fitting; 12c's arrangement. +- Produces: + - `MenuCommand::{ApplyView, SaveView, RenameView, RemoveView, SetDefaultView, DefaultOnMount, ApplySharedView, PublishView, SaveAsMyView}` + - `class TableViews` (`morph::ui`): `TableViews(std::shared_ptr store, table::TableStateKey key)`; `load() -> async::Completion>`; `own() -> table::ViewsDocument const&`; `shared() -> table::ViewsDocument const&`; `change(table::ViewOperation operation) -> async::Completion>` — saves with the loaded revision (`kNoDocument` when nothing was loaded); on `Conflict` loads again, applies the operation to what is stored now, and saves once more; a second `Conflict` answers the error `Views could not be saved: another window changed them at the same time`; an operation error answers `A view named {name} exists` or `No view named {name}`; `publish(table::ViewOperation)` likewise through `ITableStateStore::publish`; `shares() -> bool` +- Rules: the table menu gains `Views` (a submenu): each own view (checked when it is the state's `view`; labelled `{name} (modified)` while the arrangement differs from the fitted view), separator, `Save view…`, `Rename view…`, `Remove view`, `Set as default`, `Start from the default view` (checkable). `Save view…` and `Rename view…` open `table_view_dialog` with `table_view_name` (TextInput) and `table_view_save` (Button); saving under a name the user already has is a `ReplaceView`. Applying a view fits it as stored state is fitted, sets `layout`, `sort` and `filters` (through their handlers where bound) and writes its name to the state's `view`; the status line shows `Saved view {name}` after a save. At mount: with no stored `state`, the `default` view is applied; with `defaultOnMount`, the default view's `sort` and `filters` are applied over the stored state's, and the stored `layout` still applies. A mounted table does not follow saves made elsewhere. + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_table_state.cpp`: + +```cpp +// Mutation: a views save with no expected revision (the second window's save overwrites the first's view). +TEST_CASE("views: views are not lost between windows", "[ui][table][state]") +// two Harnesses (two sessions) over one MemoryTableStateStore, the same table: both mount with no views stored; window A +// saves view "A" (expected "-": Saved, revision "1"); window B saves view "B" (expected "-"): Conflict with "1", B reloads, +// applies its add and saves with "1": Saved "2"; the store's views part holds "A" and "B". Then both mount again from +// "2" and each adds a view: the second answers Conflict, reloads and saves once more; four views stored. Then A and B +// each change the sort: the state part holds B's whole arrangement (last save wins, no merge). + +// Mutation: defaultOnMount applying the default view's layout too. +TEST_CASE("views: the default view at mount, and Start from the default view", "[ui][table][state]") +// no state stored, views {"default":"Open orders", views:[Open orders: sort c2 desc, hidden c3]}: the first query sorts by +// c2 desc and c3 is hidden. Then state {sort c1 asc, layout hiding c4} and "defaultOnMount": true: the first query sorts +// by c2 desc and c4 (stored layout) is hidden, c3 is shown. + +// Mutation: the modified mark compared against the unfitted view. +TEST_CASE("views: applying a view records its name, and a change marks it modified", "[ui][table][state]") +// chooseMenu(table, nullopt, {"Views", "Open orders"}): t.sort is the view's; after h.advance(500ms) the saved state has +// "view":"Open orders"; clickHeader(table, 0): menuOf(table, nullopt)'s Views submenu labels it "Open orders (modified)". +// A view whose filter names a removed column applies without that filter. + +// Review Focus 5. Mutation: Save view… with a taken name sending AddView. +TEST_CASE("views: a taken name is replaced on save and refused on rename", "[ui][table][state]") +// views "A" and "B": Save view… named "A" after a sort change: one view "A" with the new sort, two views in all; +// Rename view… on "B" to "A": status "A view named A exists", views unchanged; the same rename when "C" was added by +// another window since load: refused the same way after the reload. +``` + +- [ ] **Step 2: Run to see them fail.** `ctest --test-dir build/cl-debug -R "views:" --output-on-failure` after building `morph_tests`. Expected: FAIL — no `Views` entry in the table menu. +- [ ] **Step 3: Implement** `TableViews`, the menu, the dialog and the mount rules. +- [ ] **Step 4: Run to see them pass**, then the whole of `morph_tests`. +- [ ] **Step 5: Mutations, docs, commit.** Commit: `feat(ui): saved table views`. + +### Task 14c: Shared views on a table + +**Files:** +- Modify: `include/morph/ui/table_views.hpp`, `src/ui/interpreter.cpp` +- Test: `tests/test_ui_table_state.cpp`, `tests/net/test_push_tables.cpp` +- Docs: `docs/spec/ui/interpreter.md` (shared views) + +**Interfaces:** +- Consumes: 14a's `publish`, 14b's `TableViews` and menu. +- Produces: the `Views` submenu gains, while `shared().views` is not empty, a heading `Shared views` and each shared view (applied as an own view is), and `Save as my view` while the current view is a shared one (an `AddView` of it into the user's views); while `shares()`, `Publish view…` (the name dialog, then a `publish` of `AddView`, or `ReplaceView` over a taken shared name); a refused publish shows `This view could not be published: {reason}`, a done one `Published view {name}`. Over a store whose `shares()` is false none of the publish entries is offered. + +- [ ] **Step 1: Write the failing tests:** + +```cpp +// Mutation: the shared part saved through `save` instead of `publish` (an authoriser cannot tell it from own state). +TEST_CASE("shared views on a table: publish, read by another person, Save as my view; none over a store that does not share", + "[ui][table][state][remote]") +// two Harnesses with PreferencesTableStateStore against one server harness, tokens alice and bob: alice's +// chooseMenu(table, nullopt, {"Views", "Publish view…"}) named "Team", dispatched as action "publish" (never "save"); +// bob mounts: his Views submenu has heading "Shared views" and "Team"; applying "Team" then "Save as my view" adds +// "Team" to bob's own views. An authoriser refusing bob's publish: status starts "This view could not be published:". +// A Harness over MemoryTableStateStore(Sharing::Private): no "Publish view…" entry. +``` + +In `tests/net/test_push_tables.cpp`: the publish-then-read flow over a real socket. Gate T2 is held by 14a. + +- [ ] **Step 2: Run to see them fail.** `ctest --test-dir build/cl-debug -R "shared views on a table" --output-on-failure`. Expected: FAIL — no `Publish view…` entry. +- [ ] **Step 3: Implement** the entries and the publish flow. +- [ ] **Step 4: Run to see them pass**, then `morph_tests`; on Linux `ctest --test-dir build/linux-everything -R "shared views" --output-on-failure`. +- [ ] **Step 5: Mutations, docs, commit.** Commit: `feat(ui): saved and shared table views`. + +--- + +## Task 16: A raised sort bound (table.md step 9c) + +**Spec:** "Sort chains" (the bound); Failure modes "A table whose `limits` set `maxSortKeys` above 32" and "A sort key past the table's `maxSortKeys`"; the Tests row "A raised sort bound". Depends only on Task 12. + +**Files:** +- Modify: `include/morph/table/description.hpp` (`servedDescription`; `table::detail::answerQuery` takes its description from it and answers its `SourceContract`), `include/morph/core/registry.hpp` (the `Table..describe` handler `BRIDGE_REGISTER_TABLE` registers obtains the description through it), `src/ui/document.cpp` (`maxSortKeys` on `table`), `src/ui/interpreter.cpp` (the bound the lowering passes to `applySortGesture`: the description's on a model-backed table, the literal on a `rows` table, else 8) +- Test: `tests/table/test_table_description.cpp`, `tests/test_ui_table_model.cpp`, `tests/test_ui_document.cpp` +- Docs: `docs/spec/ui/document.md` (`maxSortKeys`), `docs/spec/core/registry.md` (the bound carried from `limits`) + +**Interfaces:** +- Consumes: Task 2's `ServedTable::limits` (`QueryLimits`, whose `maxSortKeys` defaults to 8); Task 1's `TableCapabilities::maxSortKeys`; 12a's `applySortGesture`. +- Produces: `inline constexpr std::size_t kMaxSortKeysBound = 32;` and `[[nodiscard]] std::expected servedDescription(ServedTable const& served)` in `morph::table` — the description with `capabilities.maxSortKeys = served.limits.maxSortKeys`, or `SourceContract` (message ``"a table's `limits.maxSortKeys` is 1 to 32"``) when it is 0 or above `kMaxSortKeysBound`; the document key `maxSortKeys` on `table`: a literal integer 1 to 32, a `rows` table's only (on a model-backed table refused at load with ``"`maxSortKeys` is a `rows` table's; a model's table carries its bound in its description"``; out of range refused with ``"`maxSortKeys` is 1 to 32"``). + +- [ ] **Step 1: Write the failing tests.** In `tests/table/test_table_description.cpp`: + +```cpp +// Mutation: servedDescription leaving capabilities at the default 8. +TEST_CASE("served description: limits carry the sort bound, up to 32", "[table][description]") { + table::ServedTable served{.description = samplesDescription(3)}; + CHECK(table::servedDescription(served)->capabilities.maxSortKeys == 8); + served.limits.maxSortKeys = 32; + CHECK(table::servedDescription(served)->capabilities.maxSortKeys == 32); + served.limits.maxSortKeys = 33; + CHECK(table::servedDescription(served).error().code == table::TableErrorCode::SourceContract); + served.limits.maxSortKeys = 0; + CHECK(table::servedDescription(served).error().code == table::TableErrorCode::SourceContract); +} +``` + +In `tests/test_ui_table_model.cpp`: + +```cpp +// Mutation: the client holding 8 whatever the description says. +TEST_CASE("sort chains: a table whose limits set 32 accepts 32 keys from the header and refuses the 33rd", "[ui][table][config]") +// kModelTable over samplesDescription(40) whose capabilities.maxSortKeys is 32: Shift-click 33 sortable headers: t.sort +// has 32 keys, the last page request's sort has 32 keys, status message "A table sorts by at most 32 columns". +// A rows table with "maxSortKeys": 12 over 13 sortable columns: 12 keys, the 13th refused with "…at most 12 columns". +``` + +In `tests/test_ui_document.cpp`: `"table: maxSortKeys is a rows table's literal from 1 to 32"` — `33` refused, `{"ref":"n"}` refused, on a model-backed table refused with the reason above. + +- [ ] **Step 2: Run to see them fail.** `cmake --build --preset cl-debug --target morph_tests morph_table_tests && ctest --test-dir build/cl-debug -R "served description|accepts 32 keys|maxSortKeys is a rows" --output-on-failure`. Expected: compile error for `servedDescription`; the 32-key case stops at 8. +- [ ] **Step 3: Implement** `servedDescription`, call it from `table::detail::answerQuery` and from the `describe` handler (a `SourceContract` answers the request), the document key and the lowering's bound. +- [ ] **Step 4: Run to see them pass**, then `morph_table_tests` and `morph_tests`. +- [ ] **Step 5: Mutations, docs, commit.** Commit: `feat(table): a sort bound up to 32 keys`. + +--- + +## Order within this part + +12a → 12b → 12c → (12d, 12e, 12f in any order; each touches the lowering, so they merge one at a time) → 13 → 14a → 14b → 14c. Task 16 needs only 12c and may run beside 13 and 14. All of Task 12 needs Part D's Tasks 9 to 11 (the lowering, the status line, the arrangement); Task 13 needs Tasks 9 and 11; Task 14 needs Task 9's store and model. diff --git a/docs/superpowers/plans/2026-11-17-one-table-f-display-and-acting.md b/docs/superpowers/plans/2026-11-17-one-table-f-display-and-acting.md new file mode 100644 index 000000000..7da1e45d3 --- /dev/null +++ b/docs/superpowers/plans/2026-11-17-one-table-f-display-and-acting.md @@ -0,0 +1,1303 @@ +# The One Table, Part F — How a Column Shows Its Cells, and Acting on Rows — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Give every table one formatter of a cell's text (places, rounding, empty text, units, labels, tone, tooltip, locked rows), and let a user act on rows: select the whole view, have the server resolve it, keep changed rows in view, follow a changed key, and see busy, failed and locked rows, with the application's own menu entries and a header it can disable. + +**Architecture:** Task 15 adds `ColumnDisplay` and its companions to the table's description (`include/morph/table/description.hpp`) and one client function, `ui::formatCell`, which every backend, copy, the filter menu and the status line call. Task 17 adds `RowSelection`, `resolve` and the view's membership digest on the server (`include/morph/table/row_selection.hpp`, the view cache), retained rows in the query, the engine and the window, `ChangedRow::from` in the result walk, and the client's busy, failed and locked rows, menus and disabled header in `ui::TableRows` and the interpreter's lowering. + +**Tech Stack:** C++23, glaze, Catch2, the Qt Quick backend (offscreen `morph_qt_quick_tests`), the terminal backend (`morph_tui_tests`). + +**Spec:** `docs/spec/table/table.md` — "How a column shows its cells" (lines 369–455), "Values with units › Display" (2586–2593), "Acting on rows" and all its subsections (2258–2451), the matching rows of "Tests" (3032–3046), "Invariants" 31–35, "Failure modes" (3314–3322), Delivery order steps 9a and 9b (3378–3379), and the 9a/9b rows of "Neighbouring specs" (3345–3355). Roadmap: `docs/superpowers/plans/2026-11-17-one-table.md`, Tasks 15 and 17. + +Part F of 7; index: 2026-11-17-one-table.md + +## Global Constraints + +Every line of the roadmap's Global Constraints applies. Those this part leans on, verbatim: + +- AGENTS.md "Comments and documentation": a comment or spec states what the code does now and why; no history, no issue numbers, no commit hashes. +- Every public symbol has complete Doxygen `@param`/`@tparam`/`@return`; the docs build runs `WARN_AS_ERROR = FAIL_ON_WARNINGS` (`cmake -S . -B build/doc -G Ninja -DMORPH_BUILD_DOCUMENTATION=ON -DMORPH_BUILD_TESTS=OFF -DMORPH_BUILD_EXAMPLES=OFF && cmake --build build/doc --target doc`). +- morph is public: no company, product or customer names in code, tests, specs or commit messages; examples use the neutral `lims` domain. +- Commits are authored `yaraslau ` and end with the session's attribution lines. +- No GUI application is launched by hand or by an agent; Qt Quick behaviour is tested through `morph_qt_quick_tests` (offscreen). +- AGENTS.md "Verify rather than assert": every test names the mutation it was seen to fail on, and the task runs that mutation once and records the failing output in its commit message body. +- Limits, verbatim: `QueryLimits::maxSelectedRows = 1'000'000`, `maxRetained = 10'000`, `maxKeys = 1000`; `SessionOptions::maxCopyRows = 10'000`, `retainHold = 2 s`. +- A spec that describes code takes its part in the task whose step its row names: `docs/spec/ui/document.md` takes "a column's display parts, `tone` and `tooltip`" in Task 15 (step 9a) and "`cellMenu`, `headerMenu`, `retainChanged`, `headerEnabled`" and the element values `selectionSet`, `busyCount`, `selectedCountKind` in Task 17 (step 9b). A backend call this part adds is described in `docs/spec/ui/backend_contract.md` and `docs/spec/qt_quick/backend.md` in the task that adds it. + +**Build and run.** The standard task loop of the roadmap, with: + +```bash +cmake --preset cl-debug # once per tree +cmake --build --preset cl-debug --target morph_tests morph_table_tests +ctest --test-dir build/cl-debug -R "" --output-on-failure + +# Socket cases: morph_net_tests is POSIX-only (index decision 18); on Windows run them in WSL, or leave them to the +# Linux CI leg +cmake --preset linux-everything && cmake --build --preset linux-everything --target morph_net_tests +ctest --preset linux-everything -R "" --output-on-failure + +# Qt Quick and terminal cases (their own tree; offscreen, nothing is launched) +cmake --preset cl-qt-debug -DMORPH_BUILD_QT_QUICK=ON -DMORPH_BUILD_TUI=ON # once per tree +cmake --build build/cl-qt-debug --target morph_qt_quick_tests morph_tui_tests +ctest --test-dir build/cl-qt-debug -R "" --output-on-failure +``` + +**Status line messages.** A transient refusal or notice goes through Part E's `ScreenCore::tableMessage(ElementState& table, std::string text)` (Task 12c, index decision 11), and, when it offers a choice ("Run on these"), through the overload `tableMessage(ElementState&, std::string text, std::vector entries)` that Task 17a adds. `ui::StatusEntry{std::string id; std::string label;}` lives in `view.hpp` beside Part D's `TableStatus`, compares by value (so `TableStatus`'s `operator==` and its `Prop` hold), and `TableStatus` gains `std::vector entries` and `std::vector standingEntries`; the lowering keeps each entry's handler in a map keyed by `id`, and the backend reports a choice by `id` (`TableWidget::setOnStatusEntry(std::function)`). **Standing states** are not messages (index decision 19): "Saving n rows", "n rows could not be saved" with "Select them" / "Clear marks", and "n rows shown that no longer match" with "Show only matching rows" are a persistent part of `TableStatus` (`TableStatus::standing`, a `std::string`, with `standingEntries`), recomputed by the lowering from `RowStates::busyCount()`, `failedKeys()` and `counts().retained` whenever one changes; they stay until their source is zero, through the 6-second message hold and the user's gestures. When several hold, `standing` joins them with ". " in the order saving, failed, unmatched. + +**Shared test fixtures from other parts.** `tests/test_ui_table_model.cpp` and `tests/ui_table_support.hpp` are Part D's (Task 9): `Harness` (`h.mount(json)`, `h.replyDescribe()`, `h.replyPage(json)`, `h.settle()`, `h.advance(ms)`), `tableCatalog()` (`Table.samples` on `Lims_SampleModel`), `samplesDescription(n)`, `describeJson(description)`, `pageJson(offset, count, total, columns, kind, undecided, awaiting)`, `lastQuery(h)` (the last `TableQuery` sent) and `statusOf(h)` (the `TableStatus`; `.message` is the status line's message). Part E (Task 12) adds the `RecordingBackend` drivers `clickHeader`, `menuOf(table, column)`, `chooseMenu(table, column, labels)`, `moveColumn`, `clickFilter`, `pressCopy(table)` and `clipboardText()`; Part D's Task 10a adds `selectTableRange`. Replies those helpers do not build (keyed replies, `unmatched`, a description with companions) are written as JSON in the test, as Part E does. This part adds to `tests/ui_table_support.hpp`, in the task that first calls them: `element(h, id, name)` (an element value), `reportsOf(h)` (the interpreter's diagnostics), `fireEvent(h, record, keys)` (a record change event), `runMutation(h, name, body)`, `settleMutation(h, name, replyJson)`, `refuseMutation(h, name, code)`, `cellText(h, key, column)`, `cellAccent(h, key, column)`, `rowMarks(h, key)`, `columnOf(h, id)` (the `ui::TableColumn` given to `setColumns`), and the `RecordingBackend` drivers `selectAll(table)`, `clickRow(table, key)`, `ctrlClickRow(table, key)`, `rightClickCell(table, key, column)`. In the sketches below `screen` (or `h`) is a `Harness`, and `screen.x(...)` stands for this list's `x(screen, ...)`; `run`, `settle(mutation, …)`, `refuse`, `click`, `ctrlClick` in the comments are `runMutation`, `settleMutation`, `refuseMutation`, `clickRow`, `ctrlClickRow`. `tests/net/test_push_tables.cpp` is Part A's (Task 2), with its server-and-client harness; `morph_net_tests` runs on `linux-everything` only (index decision 18). + +## Review Focus + +1. **Negative and boundary values in the display.** `-0.125` at two places under `HalfUp` is `-0.13`, under `HalfEven` `-0.12`; a value that rounds to zero shows no sign (`-0.001` → `0.00`); `0` with `empty` set shows `0`; the `de` locale with grouping shows `-12.500,50`. Test: Task 15, `"format cell: negative halves and empty text"`. +2. **A double whose binary value is below its decimal text.** `1.005` at two places under `HalfUp` is `1.01`, as a person reads the value, though the binary double is 1.00499…. Test: Task 15, `"format cell: a double rounds from its shortest decimal text"`. +3. **A key of `except` that leaves the view.** A view selection of 10 rows less key 3, then an edit takes row 3 out of the view: the count is 9 and exact, not 8. Test: Task 17a, `"view selection: a key of except that leaves the view is dropped from it"`. +4. **A key change onto a key the client already holds.** `ChangedRow{key: 7, from: 3}` while row 7 is held and selected: the selection holds 7 once, and row 3 is fetched again. Test: Task 17c, `"key change: onto a key already held"`. +5. **Two actions over overlapping rows.** Run A names rows {1, 2}, run B rows {2, 3}; A settles: row 2 stays busy until B settles, and `busyCount` is 2 then 0. Test: Task 17c, `"busy rows: overlapping runs"`. + +--- + +## Task 15: How a column shows its cells (table.md step 9a) + +**Spec:** "How a column shows its cells" (369–455); "Values with units › Display" (2586–2593); test rows "A column's display", "Companion columns travel with their column", "Locked rows and companions" (companion half); invariant 32. + +**Files:** +- Modify: `include/morph/table/description.hpp` — `Rounding`, `ValueLabel`, `ColumnDisplay`; `TableColumn::{display, tone, tooltip}`; `TableDescription::{rowTone, rowLock}`; `validateDisplay`, `CompanionOverrides`, `columnsWithCompanions`, `companionColumns`. Glaze meta writing `Rounding` as `"halfEven"` / `"halfUp"`. +- Create: `include/morph/ui/cell_format.hpp` — `CellLook`, `DisplayUnits`, `formatCell`, `cellLook`, `cellOfOperand`, `operandOf`, `distinctTexts`, `headerText`, `headerAccessibleName`. +- Modify: `include/morph/ui/view.hpp` (`TableColumn::unit`, `CellAccent`, `TableRowView::accents`), `include/morph/ui/backend.hpp` (`TableWidget::setCellAccent`), `include/morph/ui/mount.hpp` (binds `accents` to `setCellAccent`), `include/morph/ui/testing/recording_backend.hpp`, `src/qt_quick/table.cpp`, `src/qt_quick/kinds.hpp`, `src/qt_quick/qml/MorphTable.qml`, `src/tui/table_widget.hpp`, `src/tui/table_widget.cpp`. +- Modify: `src/ui/interpreter.cpp` (a column's display parts, `tone`, `tooltip`; the default cell's text through `formatCell`; the companions in the fetched columns; companions out of the layout), `src/ui/document.cpp` (load-time checks of the new parts). +- Create: `tests/test_ui_cell_format.cpp` (add to `morph_tests` in `tests/CMakeLists.txt`, beside `test_ui_table.cpp`). +- Test: `tests/table/test_table_description.cpp`, `tests/test_ui_table_model.cpp`, `tests/qt_quick/test_qt_quick_table.cpp`, `tests/tui/test_tui_table.cpp`. +- Docs: `docs/spec/ui/document.md` (the `table` row: a column's parts `decimals`, `rounding`, `padPlaces`, `grouping`, `empty`, `unit`, `date`, `labels`, `tone`, `tooltip`), `docs/spec/ui/backend_contract.md` and `docs/spec/qt_quick/backend.md` (`setCellAccent`, a column's `unit`). + +**Interfaces:** +- Consumes: `table::TableColumn`, `TableDescription`, `describe` (Task 1); `table::Unloaded`, `ColumnInfo::level` (Task 3); `DistinctValue{text, operand, count}` (Task 8a); `UnitInfo`, `UnitId`, `UnitRegistry`, `UnitRegistryBuilder`, `Measure`, `ColumnKind::Measure` (Task 8c); `ui::FormatContext` (`include/morph/ui/expression.hpp:325`), `ui::localeFacts`, `ui::withoutGrouping` (`include/morph/ui/locale_table.hpp`); `render::formatCanonicalNumber` (`include/morph/render/locale_format.hpp:708`); `math::checkedRoundToDecimalPlaces`, `math::RoundingMode` (`include/morph/util/rational.hpp`); `table::parseDecimal`, `table::displayText` (`include/morph/table/data_source.hpp`); `ui::detail::formatDate` (`include/morph/ui/date_pattern.hpp:327`); Task 9's `WindowRows`, its fetched-column set, and `ui::detail::standInCellText`; Part E's copy (`ui::copyText`, `IViewBackend::copyText`, Task 12f), values list (`table_filter_value_`, Task 12d) and column chooser (`table_columns`, Task 12e). +- Produces (later tasks and Part G's editing rely on these exact names): + +```cpp +namespace morph::table { +enum class Rounding : std::uint8_t { HalfEven, HalfUp }; // JSON "halfEven", "halfUp" +struct ValueLabel { std::string operand; std::string text; }; +struct ColumnDisplay { + std::optional decimals; + Rounding rounding = Rounding::HalfEven; + bool padPlaces = true; + bool grouping = false; + std::string empty; + std::string unit; + std::string date; + std::vector labels; +}; +// TableColumn gains: ColumnDisplay display; std::string tone; std::string tooltip; +// TableDescription gains: std::string rowTone; std::string rowLock; +[[nodiscard]] std::expected validateDisplay(TableDescription const& description); +struct CompanionOverrides { + std::set> tone; // column ids whose document part gives `tone` + std::set> tooltip; // column ids whose document part gives `tooltip` + bool rowTone = false; // the node gives `rowTone` +}; +[[nodiscard]] std::vector columnsWithCompanions(TableDescription const& description, + std::span columns, + CompanionOverrides const& overrides = {}); +[[nodiscard]] std::vector companionColumns(TableDescription const& description); // sorted +} // namespace morph::table + +namespace morph::ui { +enum class CellLook : std::uint8_t { Text, Empty, Unloaded, Invalid }; +struct DisplayUnits { + table::UnitRegistry const* units = nullptr; // table::registryOf(description), Part C + std::map> byDimension; // dimension -> unit id the user chose +}; +[[nodiscard]] std::string formatCell(table::Cell const& cell, table::TableColumn const& column, + FormatContext const& locale, DisplayUnits const& units = {}); +[[nodiscard]] CellLook cellLook(table::Cell const& cell, table::TableColumn const& column, + DisplayUnits const& units = {}); +[[nodiscard]] table::Cell cellOfOperand(std::string_view operand, table::TableColumn const& column, + DisplayUnits const& units = {}); +[[nodiscard]] std::optional operandOf(table::Cell const& cell, table::TableColumn const& column, + DisplayUnits const& units = {}); +[[nodiscard]] std::vector distinctTexts(std::span values, + table::TableColumn const& column, FormatContext const& locale, + DisplayUnits const& units = {}); +[[nodiscard]] std::string headerText(std::string_view label, std::string_view unit); // "Mass [kg]" +[[nodiscard]] std::string headerAccessibleName(std::string_view label, std::string_view unit); // "Mass, kg" +struct CellAccent { std::optional tone; std::string tooltip; }; // view.hpp +// ui::TableColumn (view.hpp) gains: std::string unit; +// TableRowView (view.hpp) gains: std::vector> accents; // empty, or one per cell +// TableWidget (backend.hpp) gains: +// virtual void setCellAccent(Widget& row, std::size_t column, CellAccent const& accent) = 0; +} // namespace morph::ui +``` + +The spec writes the third parameter as "the session's locale". No `Locale` type exists; `ui::FormatContext` is the session's formatting (numeric facts, date patterns, display zone), and it is what the existing default cell (`textOf(*value, scope->formatting())`, `src/ui/interpreter.cpp`) formats with. `DisplayUnits` carries what a `Measure` cell needs and the three-argument call does not have. + +- [ ] **Step 1: Write the failing description tests** in `tests/table/test_table_description.cpp` + +```cpp +TEST_CASE("description: a column's display is refused where it cannot apply", "[table][description][display]") { + using namespace morph::table; + auto base = [] { + TableDescription d; + d.id = "samples"; + d.key = "id"; + d.columns = { + TableColumn{.info = {.id = "id", .kind = ColumnKind::Integer}, .label = "Id"}, + TableColumn{.info = {.id = "mass", .kind = ColumnKind::Decimal}, .label = "Mass", + .display = {.decimals = 2, .unit = "kg"}, .tone = "massTone", .tooltip = "massTip"}, + TableColumn{.info = {.id = "amount", .kind = ColumnKind::Measure}, .label = "Amount"}, + TableColumn{.info = {.id = "massTone", .kind = ColumnKind::Text}, .shown = false}, + TableColumn{.info = {.id = "massTip", .kind = ColumnKind::Text}, .shown = false}, + TableColumn{.info = {.id = "lockedBy", .kind = ColumnKind::Text}, .shown = false}, + }; + d.rowLock = "lockedBy"; + return d; + }; + REQUIRE(validateDisplay(base()).has_value()); + + auto measureDecimals = base(); + measureDecimals.columns[2].display.decimals = 3; + CHECK(validateDisplay(measureDecimals).error().code == TableErrorCode::SourceContract); + auto measureUnit = base(); + measureUnit.columns[2].display.unit = "kg"; + CHECK(validateDisplay(measureUnit).error().code == TableErrorCode::SourceContract); + auto missingTone = base(); + missingTone.columns[1].tone = "nope"; + CHECK(validateDisplay(missingTone).error().code == TableErrorCode::SourceContract); + auto numericTooltip = base(); + numericTooltip.columns[1].tooltip = "id"; + CHECK(validateDisplay(numericTooltip).error().code == TableErrorCode::SourceContract); + auto missingLock = base(); + missingLock.rowLock = "nope"; + CHECK(validateDisplay(missingLock).error().code == TableErrorCode::SourceContract); + CHECK(validateDisplay(missingLock).error().column == "nope"); +} +// Mutation: validateDisplay skipping rowLock, which accepts `missingLock`. + +TEST_CASE("description: companions travel with their column", "[table][description][display]") { + using namespace morph::table; + TableDescription d; // id, mass (tone massTone, tooltip massTip), name; rowTone rowState; rowLock lockedBy + d.key = "id"; + d.columns = { + TableColumn{.info = {.id = "id", .kind = ColumnKind::Integer}}, + TableColumn{.info = {.id = "mass", .kind = ColumnKind::Decimal}, .tone = "massTone", .tooltip = "massTip"}, + TableColumn{.info = {.id = "name", .kind = ColumnKind::Text}}, + TableColumn{.info = {.id = "massTone", .kind = ColumnKind::Text}, .shown = false}, + TableColumn{.info = {.id = "massTip", .kind = ColumnKind::Text}, .shown = false}, + TableColumn{.info = {.id = "rowState", .kind = ColumnKind::Text}, .shown = false}, + TableColumn{.info = {.id = "lockedBy", .kind = ColumnKind::Text}, .shown = false}, + }; + d.rowTone = "rowState"; + d.rowLock = "lockedBy"; + std::vector const mass{"mass"}; + std::vector const name{"name"}; + CHECK(columnsWithCompanions(d, mass) == + std::vector{"mass", "massTone", "massTip", "rowState", "lockedBy"}); + CHECK(columnsWithCompanions(d, name) == std::vector{"name", "rowState", "lockedBy"}); + CHECK(columnsWithCompanions(d, mass, {.tone = {"mass"}, .tooltip = {}, .rowTone = true}) == + std::vector{"mass", "massTip", "lockedBy"}); + CHECK(companionColumns(d) == std::vector{"lockedBy", "massTip", "massTone", "rowState"}); +} +// Mutation: companions added for every column of the description, which puts massTone in `name`'s list. + +TEST_CASE("description: a display travels in the description and changes its revision", + "[table][description][display]") { + // A description whose `mass` column has display {decimals 2, rounding HalfUp, labels [{"3","three"}]}: + // its JSON holds "rounding":"halfUp" and "labels":[{"operand":"3","text":"three"}]; it reads back equal; + // its revision differs from the same description with rounding HalfEven. +} +// Mutation: `Rounding` without glaze meta, which writes 1 for "halfUp". +``` + +Write the third case's assertions as the comment states (`CHECK(json.find("\"rounding\":\"halfUp\"") != npos)`, a read-back `==`, two `describe` revisions `!=`). + +- [ ] **Step 2: Run them; they fail to compile** (`validateDisplay`, `TableColumn::display` not declared). + +Run: `cmake --build --preset cl-debug --target morph_table_tests` +Expected: compile errors naming `validateDisplay` and `display`. + +- [ ] **Step 3: Implement the description additions** in `include/morph/table/description.hpp` with the Interfaces block's signatures. + - `validateDisplay` returns `TableError{.code = SourceContract, .column = }` for: `decimals` or `unit` set on a `Measure` column (column = that column); a `tone`, `tooltip`, `rowTone` or `rowLock` that names no column, or names a column whose kind is not `Text` (column = the named id). + - Call `validateDisplay` where Task 2 refuses a `rowScoped` table with an empty `scope` as `SourceContract` "when the table is first served" (grep `SourceContract` in `include/morph/core/registry.hpp` and `description.hpp`); a refusal there is the table's `describe` and query error. + - `columnsWithCompanions`: for each id of `columns` in order, the id, then its `tone` companion unless `overrides.tone` holds the id, then its `tooltip` companion unless `overrides.tooltip` holds it; then `rowTone` unless `overrides.rowTone`; then `rowLock`. Empty names are skipped; an id already listed is not listed again. `rowLock` travels with every row as `rowTone` does: a locked row cannot be drawn without it. + - `companionColumns`: every non-empty `tone`, `tooltip`, `rowTone` and `rowLock` name, sorted, without duplicates. + +- [ ] **Step 4: Run them; they pass.** + +Run: `ctest --test-dir build/cl-debug -R "description:" --output-on-failure` +Expected: all `description:` cases pass. + +- [ ] **Step 5: Write the failing formatter tests** in `tests/test_ui_cell_format.cpp` (add the file to `morph_tests`) + +```cpp +namespace { +using morph::table::Cell; +using morph::table::ColumnKind; +using morph::table::Rounding; +using morph::table::TableColumn; + +morph::math::Rational dec(std::string_view text) { return morph::table::parseDecimal(text).value(); } +TableColumn numeric(ColumnKind kind, morph::table::ColumnDisplay display) { + return TableColumn{.info = {.id = "v", .kind = kind}, .label = "V", .display = std::move(display)}; +} +morph::ui::FormatContext en() { return {}; } +morph::ui::FormatContext de() { + morph::ui::FormatContext context; + context.locale = "de"; + context.numeric = morph::ui::localeFacts("de").numeric; + context.datePattern = "dd.MM.yyyy"; + return context; +} +} // namespace + +TEST_CASE("format cell: a column's places and rounding", "[ui][cell_format]") { + using morph::ui::formatCell; + CHECK(formatCell(Cell{dec("0.125")}, numeric(ColumnKind::Decimal, {.decimals = 2}), en()) == "0.12"); + CHECK(formatCell(Cell{dec("0.125")}, + numeric(ColumnKind::Decimal, {.decimals = 2, .rounding = Rounding::HalfUp}), en()) == "0.13"); + CHECK(formatCell(Cell{dec("1.5")}, numeric(ColumnKind::Decimal, {.decimals = 2}), en()) == "1.50"); + CHECK(formatCell(Cell{dec("1.5")}, numeric(ColumnKind::Decimal, {.decimals = 2, .padPlaces = false}), en()) == + "1.5"); + CHECK(formatCell(Cell{dec("2")}, numeric(ColumnKind::Decimal, {.decimals = 2, .padPlaces = false}), en()) == + "2"); + CHECK(formatCell(Cell{std::int64_t{5}}, numeric(ColumnKind::Integer, {.decimals = 2}), en()) == "5.00"); + CHECK(formatCell(Cell{dec("12.50")}, numeric(ColumnKind::Decimal, {}), en()) == "12.50"); // own places + CHECK(formatCell(Cell{0.1}, numeric(ColumnKind::Number, {}), en()) == "0.1"); // shortest + CHECK(formatCell(Cell{std::int64_t{12500}}, numeric(ColumnKind::Integer, {.grouping = true}), en()) == + "12,500"); + CHECK(formatCell(Cell{std::int64_t{12500}}, numeric(ColumnKind::Integer, {}), en()) == "12500"); +} +// Mutation: `rounding` ignored (always HalfEven), which shows "0.12" under HalfUp. + +TEST_CASE("format cell: negative halves and empty text", "[ui][cell_format][review]") { + using morph::ui::formatCell; + auto const halfUp = numeric(ColumnKind::Decimal, {.decimals = 2, .rounding = Rounding::HalfUp, .empty = "–"}); + auto const halfEven = numeric(ColumnKind::Decimal, {.decimals = 2, .empty = "–"}); + CHECK(formatCell(Cell{dec("-0.125")}, halfUp, en()) == "-0.13"); + CHECK(formatCell(Cell{dec("-0.125")}, halfEven, en()) == "-0.12"); + CHECK(formatCell(Cell{dec("-0.001")}, halfEven, en()) == "0.00"); // zero shows no sign + CHECK(formatCell(Cell{std::int64_t{0}}, numeric(ColumnKind::Integer, {.empty = "–"}), en()) == "0"); + CHECK(formatCell(Cell{std::monostate{}}, halfEven, en()) == "–"); + CHECK(formatCell(Cell{std::monostate{}}, numeric(ColumnKind::Decimal, {}), en()).empty()); + CHECK(formatCell(Cell{dec("-12500.5")}, numeric(ColumnKind::Decimal, {.decimals = 2, .grouping = true}), de()) == + "-12.500,50"); + CHECK(formatCell(Cell{dec("-12500.5")}, numeric(ColumnKind::Decimal, {.decimals = 2}), de()) == "-12500,50"); +} +// Mutation: a zero value shown as `empty`, which gives "–" for 0. + +TEST_CASE("format cell: a double rounds from its shortest decimal text", "[ui][cell_format][review]") { + using morph::ui::formatCell; + auto const halfUp = numeric(ColumnKind::Number, {.decimals = 2, .rounding = Rounding::HalfUp}); + CHECK(formatCell(Cell{1.005}, halfUp, en()) == "1.01"); + CHECK(formatCell(Cell{2.675}, halfUp, en()) == "2.68"); + CHECK(formatCell(Cell{0.125}, numeric(ColumnKind::Number, {.decimals = 2}), en()) == "0.12"); + CHECK(formatCell(Cell{1e21}, numeric(ColumnKind::Number, {}), en()) == "1000000000000000000000"); +} +// Mutation: std::format("{:.2f}") on the binary value, which gives "1.00" for 1.005. + +TEST_CASE("format cell: labels, dates, quantities and measures", "[ui][cell_format]") { + using morph::ui::formatCell; + auto const available = TableColumn{ + .info = {.id = "ok", .kind = ColumnKind::Bool}, + .display = {.labels = {{.operand = "true", .text = "available"}, {.operand = "false", .text = "not available"}}}}; + CHECK(formatCell(Cell{true}, available, en()) == "available"); + CHECK(formatCell(Cell{false}, available, en()) == "not available"); + auto const coded = numeric(ColumnKind::Integer, {.labels = {{.operand = "3", .text = "three"}}}); + CHECK(formatCell(Cell{std::int64_t{3}}, coded, en()) == "three"); + CHECK(formatCell(Cell{std::int64_t{4}}, coded, en()) == "4"); + + auto const date = TableColumn{.info = {.id = "d", .kind = ColumnKind::Date}}; + CHECK(formatCell(Cell{std::int64_t{20'000}}, date, en()) == "2024-10-04"); + CHECK(formatCell(Cell{std::int64_t{20'000}}, date, de()) == "04.10.2024"); + auto const patterned = TableColumn{.info = {.id = "d", .kind = ColumnKind::Date}, .display = {.date = "dd-MMM-yyyy"}}; + CHECK(formatCell(Cell{std::int64_t{20'000}}, patterned, de()) == "04-Oct-2024"); + + auto const grams = morph::table::QuantityCell{.amount = dec("250"), .toCanonical = dec("0.001")}; + CHECK(formatCell(Cell{grams}, numeric(ColumnKind::Quantity, {.decimals = 3}), en()) == "0.250"); + + morph::table::UnitRegistryBuilder builder; + REQUIRE(builder.add({.id = "kg", .symbol = "kg", .dimension = "mass", .toBase = dec("1"), .decimals = 3})); + REQUIRE(builder.add({.id = "g", .symbol = "g", .dimension = "mass", .toBase = dec("0.001"), .decimals = 0})); + auto const registry = std::move(builder).build(); + morph::ui::DisplayUnits const units{.units = registry.get(), .byDimension = {}}; + auto const amount = TableColumn{.info = {.id = "amount", .kind = ColumnKind::Measure}}; + auto const kg = *registry->find("kg"); + auto const g = *registry->find("g"); + CHECK(formatCell(Cell{morph::table::Measure{.amount = dec("0.3"), .unit = kg}}, amount, en(), units) == "0.300 kg"); + CHECK(formatCell(Cell{morph::table::Measure{.amount = dec("200"), .unit = g}}, amount, en(), units) == "200 g"); + CHECK(formatCell(Cell{morph::table::Measure{.amount = dec("1.2345"), .unit = kg}}, amount, en(), units) == + "1.234 kg"); + morph::ui::DisplayUnits const inKg{.units = registry.get(), .byDimension = {{"mass", "kg"}}}; + CHECK(formatCell(Cell{morph::table::Measure{.amount = dec("200"), .unit = g}}, amount, en(), inKg) == "0.200 kg"); +} +// Mutation: a Measure shown at the column's `decimals` instead of its unit's, which gives "0.3 kg". + +TEST_CASE("format cell: empty, unloaded and invalid differ", "[ui][cell_format]") { + using morph::ui::CellLook; + using morph::ui::cellLook; + auto const integer = numeric(ColumnKind::Integer, {.empty = "–"}); + CHECK(cellLook(Cell{std::monostate{}}, integer) == CellLook::Empty); + CHECK(cellLook(Cell{morph::table::Unloaded{}}, integer) == CellLook::Unloaded); + CHECK(morph::ui::formatCell(Cell{morph::table::Unloaded{}}, integer, en()).empty()); + CHECK(cellLook(Cell{std::string{"abc"}}, integer) == CellLook::Invalid); + CHECK(morph::ui::formatCell(Cell{std::string{"abc"}}, integer, en()).empty()); + CHECK(cellLook(Cell{std::string{"12"}}, integer) == CellLook::Text); + CHECK(morph::ui::formatCell(Cell{std::string{"12"}}, numeric(ColumnKind::Integer, {.decimals = 1}), en()) == + "12.0"); + auto const amount = TableColumn{.info = {.id = "amount", .kind = ColumnKind::Measure}}; + CHECK(cellLook(Cell{morph::table::Measure{.amount = dec("1"), .unit = morph::table::kUnknownUnit}}, amount) == CellLook::Invalid); +} +// Mutation: `empty` shown for an unloaded cell, which gives "–". + +TEST_CASE("format cell: values that show alike keep their places in the values list", "[ui][cell_format]") { + std::vector const values{ + {.text = {}, .operand = "0.124", .count = 1}, + {.text = {}, .operand = "0.123", .count = 2}, + {.text = {}, .operand = "0.5", .count = 1}, + }; + CHECK(morph::ui::distinctTexts(values, numeric(ColumnKind::Decimal, {.decimals = 2}), en()) == + std::vector{"0.124", "0.123", "0.50"}); + CHECK(morph::ui::operandOf(Cell{dec("0.125")}, numeric(ColumnKind::Decimal, {.decimals = 2})) == "0.125"); + CHECK(morph::ui::operandOf(Cell{true}, TableColumn{.info = {.id = "b", .kind = ColumnKind::Bool}}) == "true"); + CHECK_FALSE(morph::ui::operandOf(Cell{std::monostate{}}, numeric(ColumnKind::Integer, {})).has_value()); + CHECK_FALSE(morph::ui::operandOf(Cell{morph::table::Unloaded{}}, numeric(ColumnKind::Integer, {})).has_value()); + CHECK(morph::ui::headerText("Mass", "kg") == "Mass [kg]"); + CHECK(morph::ui::headerText("Mass", "") == "Mass"); + CHECK(morph::ui::headerAccessibleName("Mass", "kg") == "Mass, kg"); +} +// Mutation: the values list formatting every entry by `decimals`, which shows "0.12" twice. +``` + +`UnitRegistryBuilder::add(UnitInfo) -> std::expected`, `build() &&`, `UnitRegistry::find` and `kUnknownUnit` are Task 8c's (Part C). The lowering fills `DisplayUnits::units` with Part C's `table::registryOf(description)` and `byDimension` from Task 8d's `ColumnLayoutValue::units`. + +- [ ] **Step 6: Run them; they fail to compile** (`morph/ui/cell_format.hpp` missing). + +Run: `cmake --build --preset cl-debug --target morph_tests` +Expected: `cannot open include file 'morph/ui/cell_format.hpp'`. + +- [ ] **Step 7: Implement `include/morph/ui/cell_format.hpp`** with the Interfaces block's signatures. The rules the tests do not fix: + - **The number path.** Every numeric cell becomes canonical decimal text (`-?digits(.digits)?`), is rounded on that text, then written by `render::formatCanonicalNumber` with `locale.numeric` when `display.grouping`, else `ui::withoutGrouping(locale.numeric)`. A `Rational` with `decimals` set rounds exactly with `math::checkedRoundToDecimalPlaces(value, DecimalPlaces{d}, mode)` (`HalfUp` → `RoundingMode::HalfAwayFromZero`), a failed round is `CellLook::Invalid`; without `decimals` it keeps its declared places (`units::detail::formatRationalDecimal`). A `double` is written with `std::to_chars(…, std::chars_format::fixed)` (its shortest round-trip text), then rounded as text. An `int64` is its digits. A `QuantityCell` is `amount × toCanonical`, multiplied with checked arithmetic (an overflow is `Invalid`). + - **Rounding text** (one private helper, used for `double` and padding): keep `d` fraction digits; the dropped tail decides: above half rounds the magnitude up, below keeps it, exactly half rounds up under `HalfUp` and to the even last kept digit under `HalfEven`. Then `padPlaces` pads to `d` digits, or else trailing zeros and a bare point are removed. A result whose digits are all zero has no sign. + - **Order of decisions.** `Unloaded` → `""`. `std::monostate` → `display.empty`. A `std::string` in an `Integer`, `Key`, `Decimal`, `Quantity`, `Number` or `Bool` column is read for the kind (`std::from_chars`, `table::parseDecimal`, `"true"`/`"false"`); text that does not read is `Invalid` and shows `""`; text that reads is formatted as the value it reads as. A string in a `Date`, `DateTime`, `Text` or `Custom` column shows as it is. Then `labels`: the first label whose `operand` equals `operandOf(cell)` gives the text (a `Decimal` or `Number` operand is compared by value: `"3"` matches 3.0). Then the kind's own text. + - **Dates.** `Date` is days since 1970-01-01, `DateTime` UTC seconds shifted by `locale.utcOffsetMinutes`; the pattern is `display.date`, else `locale.datePattern` / `locale.dateTimePattern`, else ISO (`2024-10-04`, `2024-10-04 08:30`); written by `ui::detail::formatDate`. + - **Measures.** The unit's `decimals` and `symbol` from `units.units`; with `units.byDimension` naming a unit of the cell's dimension, the amount is converted exactly (`amount × from.toBase / to.toBase`) and rounded `HalfEven` to that unit's decimals. Text is the number, one ASCII space, the symbol; the ratio unit (`""`) writes no space and no symbol. No registry, or a unit the registry does not know, is `Invalid`. + - `operandOf`: `int64` → its digits; `Rational` → `formatRationalDecimal` at its own places; `double` → shortest round-trip text; `bool` → `"true"`/`"false"`; `std::string` → itself; `QuantityCell` → its canonical amount; `Measure` → amount, a space, the unit's id; empty, unloaded and invalid → `std::nullopt`. `cellOfOperand` is its inverse for the column's kind. + - `distinctTexts`: each operand is read with `cellOfOperand` and formatted; when two entries' texts are equal, each of those entries is formatted again with `display.decimals` cleared (its own places). + +- [ ] **Step 8: Run them; they pass.** + +Run: `ctest --test-dir build/cl-debug -R "format cell:" --output-on-failure` +Expected: 7 test cases pass. + +- [ ] **Step 9: Write the failing lowering tests** in `tests/test_ui_table_model.cpp`. Each mounts a model-backed table over a description with the columns `id` (Integer, key), `mass` (Decimal, `display = {decimals 2, rounding HalfUp, unit "kg"}`, `tone = "massTone"`, `tooltip = "massTip"`), `name` (Text), the companions `massTone`, `massTip` and `lockedBy` (Text, `shown: false`) and `rowLock = "lockedBy"`, and one row `{id 1, mass 0.125, name "A", massTone "warn", massTip "Above limit", lockedBy ""}`. + +```cpp +TEST_CASE("table display: a model-backed table draws its description's display", "[ui][table][display]") { + Harness screen; + screen.mount(R"({"kind":"table","id":"t","model":"Lims_SampleModel","table":"samples"})"); + screen.replyDescribe(describeJson(companionDescription())); + screen.replyPage(kSamplesPage); + CHECK(screen.cellText(1, "mass") == "0.13"); + CHECK(screen.cellAccent(1, "mass").tone == morph::ui::Tone::Warn); + CHECK(screen.cellAccent(1, "mass").tooltip == "Above limit"); + CHECK(screen.columnOf("mass").unit == "kg"); +} +// Mutation: the default cell's text left on the stand-in formatter, which shows "0.125". + +TEST_CASE("table display: companions are fetched with their column only", "[ui][table][display]") { + Harness screen; + screen.mount(R"({"kind":"table","id":"t","model":"Lims_SampleModel","table":"samples", + "columns":[{"id":"name"}]})"); + screen.replyDescribe(describeJson(companionDescription())); + CHECK(lastQuery(screen).columns == std::vector{"id", "name", "lockedBy"}); + screen.mount(R"({"kind":"table","id":"t","model":"Lims_SampleModel","table":"samples", + "columns":[{"id":"mass","tone":{"lit":"ok"}}]})"); + screen.replyDescribe(describeJson(companionDescription())); + CHECK(lastQuery(screen).columns == std::vector{"id", "mass", "massTip", "lockedBy"}); +} +// Mutation: companions fetched for every described column, which puts massTone in the first query. + +TEST_CASE("table display: companion columns are not arranged", "[ui][table][display]") { + Harness screen; + screen.mount(R"({"kind":"table","id":"t","model":"Lims_SampleModel","table":"samples"})"); + screen.replyDescribe(describeJson(companionDescription())); + screen.replyPage(kSamplesPage); + CHECK(screen.element("t", "visibleColumns") == morph::ui::parseValue(R"(["id","mass","name"])")); + CHECK(screen.element("t", "hiddenCount") == morph::ui::Value{std::int64_t{0}}); + // Part E's column chooser (`Columns…`, dialog `table_columns`) has table_columns_show_id, _mass and _name, and + // no table_columns_show_massTone, _massTip or _lockedBy. +} +// Mutation: the chooser listing every column of the description, which lists table_columns_show_massTone. + +TEST_CASE("table display: a document's parts override the description", "[ui][table][display]") { + Harness screen; + screen.mount(R"({"kind":"table","id":"t","model":"Lims_SampleModel","table":"samples", + "columns":[{"id":"mass","decimals":1,"rounding":"halfEven","empty":"n/a"}]})"); + screen.replyDescribe(describeJson(companionDescription())); + screen.replyPage(kSamplesPage); + CHECK(screen.cellText(1, "mass") == "0.1"); +} +// Mutation: document parts read only on a `rows` table, which keeps "0.13". + +TEST_CASE("table display: an unknown tone is reported once for its column", "[ui][table][display]") { + // Two rows whose massTone is "purple": the cell has no tone, and `reportsOf(h)` holds exactly one entry, + // "column `mass`'s tone is \"purple\"; a tone is neutral, info, ok, warn or err". +} +// Mutation: reporting per row, which reports twice. + +TEST_CASE("table display: copy writes the shown text and the header with its unit", "[ui][table][display]") { + // Select row 1, "Copy with headers": clipboardText() == "Id\tMass [kg]\tName\n1\t0.13\tA". +} +// Mutation: copy writing displayText, which writes 0.125. + +TEST_CASE("table display: text never orders", "[ui][table][display]") { + // A `rows` table over [{id:1, v:10}, {id:2, v:9}] with column v {"decimals":2,"sortable":true}, sorted asc: + // the rows read "9.00", "10.00". With `filters` {"v":{"include":[{"eq":"0.13"}]}} over a row {v: 0.125} shown + // as "0.13" (decimals 2, rounding halfUp): element("t","viewCount") == 0. A Bool column labelled + // "available"/"not available", sorted asc: the row with false comes first. +} +// Mutation: a sort over the formatted text, which orders "10.00" before "9.00". +``` + +`cellText`, `cellAccent` and `columnOf` are this part's helpers (see "Shared test fixtures"). Each case answers the describe request with the companion description first: `companionDescription()`, a function in the test file returning the `TableDescription` above (`rowLock = "lockedBy"`), sent as `screen.replyDescribe(describeJson(companionDescription()))` — Part D's `replyDescribe()` serves `samplesDescription`, which has no `mass` or companions; if its `replyDescribe` takes no argument, this part adds the overload `replyDescribe(std::string json)` to `tests/ui_table_support.hpp`. `kSamplesPage` is the page JSON of the row above, written in the test. Write the commented cases' assertions exactly as stated. + +- [ ] **Step 10: Write the failing backend tests.** In `tests/qt_quick/test_qt_quick_table.cpp`: `"qt_quick table: a toned cell keeps its selection mark, a tooltip cell is described, a unit header is named with its unit"` — a table with columns `{label "Mass", unit "kg"}` and one row whose cell gets `CellAccent{Tone::Warn, "Above limit"}`, selected: the cell item's tint is the theme's `warn` surface and the row's selection mark is still drawn; the cell's `Accessible.description` is `"Above limit"` and it shows the "more here" mark; the header item's `Accessible.name` is `"Mass, kg"`. Mutation: the tint painted over the selection mark. In `tests/tui/test_tui_table.cpp`: `"tui table: a header with a unit reads Mass [kg]"` — the header row's text holds `Mass [kg]`. Mutation: `unit` ignored. + +- [ ] **Step 11: Run them; they fail** (`unit`, `setCellAccent` undeclared). + +Run: `cmake --build --preset cl-debug --target morph_tests` and `cmake --build build/cl-qt-debug --target morph_qt_quick_tests morph_tui_tests` +Expected: compile errors naming `setCellAccent`, `unit`, `accents`. + +- [ ] **Step 12: Implement the view, mount, backends and lowering.** + - `view.hpp`/`backend.hpp`/`mount.hpp`: the members in the Interfaces block. The mount calls `setCellAccent(row, column, accent)` after `setRowKey` for a row whose `accents` is not empty, and again on each change of one; `ui::TableColumn::unit` goes to `setColumns`. The recording backend records both. + - Qt Quick (`src/qt_quick/table.cpp`, `MorphTable.qml`): a toned cell is filled with the theme's tone surface under the selection and focus marks; a cell with a tooltip draws the theme's "more here" mark, shows the tooltip on hover after the theme's delay and on keyboard focus at once, and carries it as `Accessible.description`; a header with a unit draws the unit on a second, muted line and has `Accessible.name` `headerAccessibleName(label, unit)`. + - Terminal (`src/tui/table_widget.cpp`): a header is `headerText(label, unit)`; a toned cell uses the tone's colour; a tooltip is shown in the status area while the cell is focused. + - `src/ui/interpreter.cpp`: a column part's `decimals`, `rounding`, `padPlaces`, `grouping`, `empty`, `unit`, `date` and `labels` (literals) build a `table::ColumnDisplay` that overrides the description's per member; on a table over `rows` they are its only source. A part's `tone` and `tooltip` are expressions in the row's scope and win over the description's companions; the lowering then passes them in `CompanionOverrides`. The default cell's text is `formatCell(cell, column, scope->formatting(), displayUnits)`; its look decides the unloaded placeholder and the invalid mark. The fetched column set is `columnsWithCompanions(description, windowColumns, overrides)`. The columns of `companionColumns(description)` are left out of the `ColumnLayout` the lowering builds, the column chooser and the header menus. An unknown tone name is reported once per column with the message in the test. A `Measure` column with no `align` part aligns to the end, with digits of one width, as an `end` column does ("Values with units › Display"). + - **Task 9's stand-in.** Task 9 formats a model-backed table's cells through a stand-in until this task lands. It is `ui::detail::standInCellText(table::Cell const&, table::TableColumn const&)` in `include/morph/ui/table_rows.hpp` (index decision 9). Find every call with `grep -rn "standInCellText" include src tests`, replace each (Part E's copy among them) with `formatCell(cell, column, scope->formatting(), displayUnits)`, and delete the stand-in; then the same grep prints nothing. Also `grep -n "displayText(\|textOf(" src/ui/interpreter.cpp`: a call that turns a table cell into its shown text goes through `formatCell` too. If Task 9 has not landed, Task 9 calls `formatCell` directly and this bullet does nothing. + - Part E's copy (Task 12f) writes each cell with `formatCell` and each label with `headerText(label, unit)` before `ui::copyText(labels, rows)` and `IViewBackend::copyText`. Part E's values list (Task 12d, dialog `table_filter`) labels `table_filter_value_` with `distinctTexts(values, column, locale, units)[i] + " (" + count + ")"` in place of the reply's `text`. + - `src/ui/document.cpp`: `rounding` is a literal `"halfEven"` or `"halfUp"`; `decimals` a literal integer 0 to 18; `labels` a literal list of `{operand, text}`; each other value is reported at its path as the existing column parts are. + +- [ ] **Step 13: Run the task's tests and their targets; all pass.** + +Run: `ctest --test-dir build/cl-debug -R "format cell:|description:|table display:" --output-on-failure` and `ctest --test-dir build/cl-qt-debug -R "qt_quick table:|tui table:" --output-on-failure`, then the whole `morph_tests`, `morph_table_tests`, `morph_qt_quick_tests` and `morph_tui_tests`. +Expected: all pass. + +- [ ] **Step 14: Mutations.** Apply each mutation named above by hand, run its test, see it fail, revert; keep each failing line for the commit body. + +- [ ] **Step 15: Docs.** `docs/spec/ui/document.md`: the `table` row gains a column's parts `decimals`, `rounding`, `padPlaces`, `grouping`, `empty`, `unit`, `date`, `labels`, `tone`, `tooltip`, each in one clause with its rule from "How a column shows its cells". `docs/spec/ui/backend_contract.md` and `docs/spec/qt_quick/backend.md`: `setCellAccent` and a column's `unit`. Run the docs build; it is green. + +- [ ] **Step 16: Commit.** + +```bash +git add include/morph/table/description.hpp include/morph/ui/cell_format.hpp include/morph/ui/view.hpp \ + include/morph/ui/backend.hpp include/morph/ui/mount.hpp include/morph/ui/testing/recording_backend.hpp \ + src/ui/interpreter.cpp src/ui/document.cpp src/qt_quick src/tui tests docs/spec +git commit -m "feat(table): column display, tone and tooltip" +``` + +--- + +## Task 17: Acting on rows (table.md step 9b) + +Needs Tasks 9 and 15. Four commits, each its own reviewer gate. + +### Task 17a: A selection of the view + +**Spec:** "A selection of the view" (2264–2357); test rows "A selection of the view", "Resolving from elsewhere", "The view revision is membership", "Copy of a selection of the view"; invariants 33, 34; failure modes 3314–3315. + +**Files:** +- Create: `include/morph/table/row_selection.hpp` — `RowSelection`, `SelectionPolicy`, `SelectedRows`, `ResolveRequest`, `RowRef`, `resolve`, `detail::membership`. +- Modify: `include/morph/table/data_source.hpp` (`TableErrorCode::ViewChanged`, wire name `"viewChanged"` in the enum's glaze meta at `data_source.hpp:934`), `include/morph/table/query.hpp` (`QueryLimits::maxSelectedRows`; `Page::viewRevision`; `PageRows::viewRevision`, filled by `apply`), `include/morph/table/view_cache.hpp` (`MembershipDigest`; the entry keeps the digest), `include/morph/table/description.hpp` (`GenericPage::viewRevision`, written and read), `include/morph/core/registry.hpp` (`BRIDGE_REGISTER_TABLE` adds `Table..resolve`), `include/morph/forms/forms.hpp` (`DeclaresSchemaValue`). +- Modify: `include/morph/ui/table_rows.hpp` (`ViewSelection`, `TableSelection`, `CountKind`, `SelectedCount`, `SelectAllRefusal`, the `TableRows` members below, `selectAllRefusalText`), `include/morph/ui/view.hpp` (`StatusEntry`; `TableStatus::{entries, standing, standingEntries}`), `include/morph/ui/backend.hpp` (`TableWidget::setOnSelectAll`, `setSelectionView`, `setOnStatusEntry`), `include/morph/ui/mount.hpp`, the recording, Qt Quick and terminal table widgets, `include/morph/ui/interpreter.hpp` and `src/ui/interpreter.cpp` (`RunObserver`, `Session::addRunObserver`/`removeRunObserver`, `TableInput`, `tableInputsOf`; the model-backed table's selection, element values `selectionSet` and `selectedCountKind`; the `viewChanged` refusal and its "Run on these" entry; Task 12's copy of a view selection). +- Test: `tests/table/test_table_selection.cpp`, `tests/net/test_push_tables.cpp`, `tests/test_ui_table_model.cpp`. +- Docs: `docs/spec/ui/document.md` (element values `selectionSet`, `selectedCountKind`). + +**Interfaces:** +- Consumes: `ServedTable`, `TableContext`, `TableOf`, `BRIDGE_REGISTER_TABLE`, `table::scoped` (Task 2); `Match`, `CompiledFilter::evaluate` (Task 3); `TableRuntime`, `currentTableRuntime`, `TableRuntimeScope` (Task 2; `views` from Task 6), `detail::answerQuery` and `AnswerContext` (Tasks 2 and 6); `ViewCache` and its entry for a validated sort and filter (Task 6); `TableRows`, `WindowRows`, `EngineRows`, `TableCountsView` (Task 9); `TableRows::keysOf` and range selection in server mode (Task 10); the copy path and `ScreenCore::tableMessage` (Task 12c); `model::ActionRefusal` (`include/morph/core/action_refusal.hpp`); `model::recordRegistry()` (`include/morph/core/records.hpp:189`). +- Produces: + +```cpp +namespace morph::table { +template +struct RowSelection { + struct View { + FilterSpec filters; + std::vector retained; + std::string revision; + }; + std::vector keys; + std::optional view; + std::vector except; + /// The schema keyword and its value: {"x-selectionOf", "."}. + [[nodiscard]] static std::pair schemaValue(); +}; +enum class SelectionPolicy : std::uint8_t { Exact, Current }; // JSON "exact", "current" +struct SelectedRows { std::vector keys; std::int64_t dropped = 0; }; +template +struct ResolveRequest { RowSelection selection; SelectionPolicy policy = SelectionPolicy::Exact; }; +template +struct RowRef { // JSON: the key itself + RowId key; + [[nodiscard]] static std::pair schemaValue(); // {"x-rowOf", record} +}; +template +[[nodiscard]] async::Completion> +resolve(Model& model, RowSelection const& selection, TableContext const& context, SelectionPolicy policy); +// Deviation from the spec's `Model const&` (index decision 23): table functions are non-const, and `resolve` reaches +// one through Task 2's table::detail::serve(model, context), which also handles a typed table. + +// Consumed, not defined here (index decision 6): Task 2's TableRuntime{exec::IExecutor* work; ViewCache* views /* Task 6 */}, +// currentTableRuntime() and TableRuntimeScope. + +class MembershipDigest { // view_cache.hpp +public: + void add(RowId const& key) noexcept; + [[nodiscard]] std::string text() const; // 48 lowercase hex digits +}; +// QueryLimits: std::int64_t maxSelectedRows = 1'000'000; +// Page, PageRows, GenericPage: std::string viewRevision; +// TableErrorCode::ViewChanged ("viewChanged") +namespace detail { +struct Membership { std::vector rows; MembershipDigest digest; }; // rows in source order +[[nodiscard]] std::expected +membership(ServedTable const& served, FilterSpec const& filters, std::span retained, + ViewCache* views); +} +} // namespace morph::table + +namespace morph::ui { +struct ViewSelection { table::FilterSpec filters; std::vector retained; std::string revision; }; +struct TableSelection { // JSON as RowSelection's + std::vector keys; + std::optional view; + std::vector except; +}; +enum class CountKind : std::uint8_t { Exact, Approximate }; // element value "exact" / "approximate" +struct SelectedCount { std::int64_t rows = 0; CountKind kind = CountKind::Exact; }; +struct SelectAllRefusal { std::int64_t undecided = 0; std::vector levels; }; +// TableRows gains (pure virtual; EngineRows and WindowRows implement): +// [[nodiscard]] virtual std::expected selectView() const = 0; +// [[nodiscard]] virtual SelectedCount selectedCount(TableSelection const& selection) const = 0; +// virtual async::Completion, table::TableError>> +// rowsOf(std::int64_t first, std::int64_t count, std::span columns) = 0; +[[nodiscard]] std::string selectAllRefusalText(SelectAllRefusal const& refusal, FormatContext const& locale); +enum class TableInputKind : std::uint8_t { Selection, Row }; +struct TableInput { std::string member; std::string record; TableInputKind kind; }; +[[nodiscard]] std::vector tableInputsOf(std::string_view schemaJson); +struct RunStarted { std::uint64_t run; std::string const& action; Value const& body; }; +struct RunSettled { + std::uint64_t run; std::string const& action; Value const& body; bool succeeded; Value const& reply; + std::string const& refusalCode; // the ActionRefusal code, empty for none + std::function rerun; // runs the same mutation again with another body +}; +class RunObserver { +public: + virtual ~RunObserver() = default; + virtual void started(RunStarted const& run) = 0; + virtual void settled(RunSettled const& run) = 0; +}; +// Session gains: void addRunObserver(RunObserver* observer); void removeRunObserver(RunObserver* observer); +// TableWidget gains: virtual void setOnSelectAll(std::function onSelectAll) = 0; +// virtual void setSelectionView(std::vector const& except) = 0; +} // namespace morph::ui +``` + +- [ ] **Step 1: Write the failing server tests** in `tests/table/test_table_selection.cpp`. The file's new cases share one model: + +```cpp +namespace { +using namespace morph::table; +// Ten rows, keys 0..9: status "open" for 0..5, "closed" for 6..9; mass i/10. +struct SelectionModel { + std::shared_ptr source = tabletest::makeSource( + {tabletest::col("status", ColumnKind::Text), tabletest::col("mass", ColumnKind::Decimal)}, + [] { + std::vector> rows; + for (int i = 0; i < 10; ++i) { + rows.push_back({Cell{std::string{i < 6 ? "open" : "closed"}}, + Cell{tabletest::dec(std::to_string(i) + ".0")}}); + } + return rows; + }()); + QueryLimits limits{}; + [[nodiscard]] ServedTable rows(TableContext const& /*context*/) { // non-const, as the spec's table functions + return ServedTable{.description = {}, .source = source, .levels = {}, .mayRead = {}, .scope = {}, + .services = {}, .limits = limits}; + } +}; +FilterSpec open() { return FilterSpec{.columns = {{"status", ColumnFilter{.include = {{.eq = "open"}}}}}}; } +// TableContext holds a reference: one caller context outlives every call in the file. +TableContext const& caller() { + static morph::session::Context const session{}; + static TableContext const context{.session = session}; + return context; +} +// resolve with no TableRuntime settles on an InlineExecutor before it returns: `then` runs at once. +template T settledNow(morph::async::Completion& completion); // `then` into an optional, REQUIRE set +std::string revisionOf(SelectionModel& model, FilterSpec const& filters, std::vector retained = {}) { + return detail::membership(model.rows(caller()), filters, retained, nullptr).value().digest.text(); +} +} // namespace + +TEST_CASE("view selection: the view revision is membership", "[table][selection][revision]") { + SelectionModel model; + std::string const before = revisionOf(model, open()); + model.source->patchRow(RowId{std::int64_t{2}}, {Cell{std::string{"open"}}, Cell{tabletest::dec("9.5")}}); + CHECK(revisionOf(model, open()) == before); // a cell changed, membership did not + TableQuery ascending{.sort = {{.column = "mass", .dir = SortDirection::Ascending}}, .filters = open()}; + TableQuery descending{.sort = {{.column = "mass", .dir = SortDirection::Descending}}, .filters = open()}; + CHECK(apply(*model.source, ascending).value().viewRevision == before); + CHECK(apply(*model.source, descending).value().viewRevision == before); // the sort is not membership + model.source->patchRow(RowId{std::int64_t{7}}, {Cell{std::string{"open"}}, Cell{tabletest::dec("0.7")}}); + CHECK(revisionOf(model, open()) != before); // a row came in + CHECK(revisionOf(model, open(), {RowId{std::int64_t{8}}}) != revisionOf(model, open())); +} +// Mutation: the revision taken from the snapshot's version, which changes it on the first edit. + +TEST_CASE("view selection: resolve under Exact is the view less except, in source order", + "[table][selection][resolve]") { + SelectionModel model; + RowSelection<&SelectionModel::rows> selection{ + .keys = {}, .view = {{.filters = open(), .retained = {}, .revision = revisionOf(model, open())}}, + .except = {RowId{std::int64_t{3}}}}; + auto exact = resolve(model, selection, caller(), SelectionPolicy::Exact); + auto const rows = settledNow(exact).value(); + CHECK(rows.keys == std::vector{std::int64_t{0}, std::int64_t{1}, std::int64_t{2}, std::int64_t{4}, + std::int64_t{5}}); + CHECK(rows.dropped == 0); + + model.source->patchRow(RowId{std::int64_t{7}}, {Cell{std::string{"open"}}, Cell{tabletest::dec("0.7")}}); + auto changed = resolve(model, selection, caller(), SelectionPolicy::Exact); + CHECK(settledNow(changed).error().code == TableErrorCode::ViewChanged); + auto current = resolve(model, selection, caller(), SelectionPolicy::Current); + CHECK(settledNow(current).value().keys == + std::vector{std::int64_t{0}, std::int64_t{1}, std::int64_t{2}, std::int64_t{4}, std::int64_t{5}, + std::int64_t{7}}); +} +// Mutation: `viewRevision` that ignores the source snapshot (a digest of the filter alone), which resolves the +// edited view under Exact. + +TEST_CASE("view selection: a key selection resolves to the keys the caller sees", "[table][selection][resolve]") { + SelectionModel model; + // The caller's source is rows 0..4 only, through Task 2's table::scoped. + // RowSelection{.keys = {3, 1, 8}}: keys == {1, 3} (source order), dropped == 1. +} +// Mutation: dropped keys left in the reply, which returns 8. + +TEST_CASE("view selection: a view over maxSelectedRows is LimitExceeded", "[table][selection][resolve]") { + CHECK(QueryLimits{}.maxSelectedRows == 1'000'000); + SelectionModel model; + model.limits.maxSelectedRows = 5; // the open view has 6 rows + // resolve of the open view under Current: error().code == TableErrorCode::LimitExceeded. +} +// Mutation: the bound compared with `>=` against the view less except, which accepts 6 rows less one. + +TEST_CASE("view selection: retained rows are in the membership", "[table][selection][resolve]") { + // A view selection of open() with retained {8} and the revision revisionOf(model, open(), {8}): + // under Exact, keys == {0,1,2,3,4,5,8}. Retained {42}, a key the source lacks, adds nothing. +} +// Mutation: retained rows left out of `membership`, which refuses with ViewChanged. + +TEST_CASE("view selection: its JSON", "[table][selection]") { + // {"keys":[1,"a"]} reads as two keys and no view; {"view":{"filters":{"columns":{"status":{"include": + // [{"eq":"open"}]}}},"retained":[8],"revision":"r"},"except":[3]} round-trips byte for byte through glaze; + // SelectionPolicy writes "exact" and "current". +} +``` + +Write the commented cases' assertions exactly as their comments state. + +- [ ] **Step 2: Run them; they fail to compile** (`row_selection.hpp` missing). + +Run: `cmake --build --preset cl-debug --target morph_table_tests` +Expected: `cannot open include file 'morph/table/row_selection.hpp'`. + +- [ ] **Step 3: Implement the server side.** + - **`MembershipDigest`.** Membership is a set of keys, so the digest does not depend on the order it is taken in, and a retained row merged into a reply is one more `add`. For each key, `h = fnv1a64(k)` (`morph::model::fnv1a64`, `include/morph/core/payload_schema.hpp:213`) over `"i:" + decimal` for an integer key and `"s:" + text` for a text key, and `m = splitmix64(h)` (`z = h + 0x9E3779B97F4A7C15; z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9; z = (z ^ (z >> 27)) * 0x94D049BB133111EB; z ^ (z >> 31)`). The digest keeps `count`, `sum += h` and `mix += m`, all modulo 2^64; `text()` is the three as 16 lowercase hex digits each, in that order. A caller adds each key once. + - **Where it is taken.** The pass that computes a view in `apply` (and the view cache's computation, Task 6) adds the key of each row it puts in the view; the cache entry stores the `MembershipDigest` beside its order; `PageRows`, `Page` and `GenericPage` carry `viewRevision = digest.text()`, page and keyed replies alike. A distinct-values reply leaves it empty. + - **`detail::membership`.** Takes the view of `filters` with an empty sort from `views` when one is given (the entry under that key, computed by the cache's single flight when absent), else computes it with `apply`; with an empty sort the view's order is source order. Each key of `retained` the source has and the view lacks is merged in at its source row and added to a copy of the digest. A view whose size after merging is above `served.limits.maxSelectedRows` is `LimitExceeded`. + - **`resolve`.** It takes `Model&` (a stated deviation from the spec's `Model const&`, index decision 23) and gets the `ServedTable` with Task 2's `table::detail::serve(model, context)` on the caller's thread (the model's strand), so the caller's scope and `mayRead` apply and a `typed` table works as a plain one. The rest runs on `currentTableRuntime()->work` with `currentTableRuntime()->views`. The returned `Completion` comes from `Completion<…>::makeSettleable(exec)` (`include/morph/core/completion.hpp:738`), `exec` being the runtime's `work`; with no runtime, or a null `work`, the work runs inline with no view cache and `exec` is a function-local static `exec::InlineExecutor` (`include/morph/core/executor.hpp:384`), so the promise is settled and `then` runs before `resolve` returns. Tests rely on that: `settledNow` attaches `then` and `REQUIRE`s the value at once. + - **The schema.** `forms.hpp` gains `DeclaresSchemaValue` (`{ T::schemaValue() } -> std::convertible_to>`), and `annotateSchemaMarker` also writes `property[first] = second` for such a type. `RowSelection::schemaValue()` and `RowRef::schemaValue()` return `{"x-selectionOf" | "x-rowOf", name}`, `name` being the `RecordInfo::name` in `model::recordRegistry()` whose `type` is `typeid(TableOf)`, or empty when none is registered. + - **`Table..resolve`.** `BRIDGE_REGISTER_TABLE` registers it beside `Table.` and `Table..describe`: body `ResolveRequest`, result `SelectedRows`, run like `Table.`: the table's function for the caller's session, then `resolve`'s work on `tableWork`. A `TableError` surfaces as `Table.`'s errors do. + - **The runtime.** Nothing to add: `RemoteServer` installs Task 2's `TableRuntimeScope` (`work` = `ServerConfig::tableWork`, `views` = `ServerConfig::views`) around each dispatch, so a Task handler reads it when it calls `resolve`; `Table..resolve` reaches the same work through `detail::answerQuery`'s `AnswerContext`. + - **A handler's refusal.** A Task handler that gets `ViewChanged` from `resolve` throws `model::ActionRefusal{"morph.table.viewChanged", {}, error.message}`; add `[[noreturn]] void table::refuse(TableError const&)` mapping `ViewChanged` to that code and every other code to `"morph.table." + `. + +- [ ] **Step 4: Run them; they pass.** + +Run: `ctest --test-dir build/cl-debug -R "view selection:" --output-on-failure` +Expected: 6 test cases pass. + +- [ ] **Step 5: Write the failing bridge tests** in `tests/net/test_push_tables.cpp`. Register in the file a model `selnet::Rows` with table `rows` (the same ten rows; a row scope by principal: `alice` sees 0..9, `bob` sees 0..4, with `scope` = the principal), an action `TagRows{table::RowSelection<&Rows::rows> rows;}` whose Task handler awaits `resolve(*this, rows, context, Exact)`, refuses through `table::refuse` and returns `ChangedRows<&Rows::rows>` of the keys, an action `MoveRow{std::int64_t key; std::string status;}` declaring `updatesTable<&Rows::rows>`, and a model `selnet::Other` whose action `Count{table::ResolveRequest<&Rows::rows> request;}` sends `Table.rows.resolve`. + +```cpp +TEST_CASE("tables: an action resolves a view selection on its own model", "[net][tables][selection]") { + // alice: Table.rows page with filters open() -> reply.viewRevision == R. + // TagRows {view: {filters: open(), retained: [], revision: R}, except: [3]} -> rows.keys {0,1,2,4,5}. + // MoveRow {key 7, status "open"}, then TagRows with revision R -> refused, code "morph.table.viewChanged". +} +// Mutation: viewRevision that ignores the source snapshot, which tags rows under the old revision. + +TEST_CASE("tables: Table..resolve answers for the caller", "[net][tables][selection]") { + // bob runs Other.Count with {selection: {keys: [1, 3, 8]}, policy: "current"} -> keys {1, 3}, dropped 1. + // bob with a view of open() under Current -> keys {0,1,2,3,4}: no row outside bob's scope. + // With the table's limits.maxSelectedRows = 5, alice's view of open() -> LimitExceeded. +} +// Mutation: resolve run under the server's principal, which returns rows outside bob's scope. + +TEST_CASE("tables: a selection's schema names its table", "[net][tables][selection]") { + // The catalog's schema of TagRows has properties.rows["x-selectionOf"] == "selnet::Rows.rows" (the record's + // registered name); a RowRef<&Rows::rows> member has "x-rowOf" with the same name. +} +// Mutation: schemaValue returning an empty name, which leaves the client unable to mark busy rows. +``` + +Write each commented assertion exactly; use the harness Task 2 created in this file. + +- [ ] **Step 6: Run them; they fail** (`Table.rows.resolve` is not a registered action; the schema lacks `x-selectionOf`). + +Run: `cmake --build --preset linux-everything --target morph_net_tests && ctest --preset linux-everything -R "tables: .*(resolve|selection)" --output-on-failure` +Expected: FAIL. + +- [ ] **Step 7: Make them pass** with Step 3's code (the registration and the runtime scope). Run Step 6's command: PASS. + +- [ ] **Step 8: Write the failing client tests** in `tests/test_ui_table_model.cpp`. The table is a model-backed `table` with `"selectionMode":"multiple"` over the ten rows of Step 1, filtered to `open()`; a document mutation `tag` runs `TagRows` with `{"rows": {"ref": "t.selectionSet"}}`. + +```cpp +TEST_CASE("view selection: Ctrl+A selects the view and fetches no row", "[ui][table][selection]") { + Harness screen; + screen.mount(kSelectionDocument); + screen.replyDescribe(describeJson(companionDescription())); + screen.replyPage(kSamplesPage); + auto const before = lastQuery(screen); + screen.selectAll(); // the backend's Ctrl+A + CHECK(lastQuery(screen) == before); // nothing sent + CHECK(screen.element("t", "selectionSet") == + morph::ui::parseValue(R"({"view":{"filters":{"columns":{"status":{"include":[{"eq":"open"}]}}}, + "retained":[],"revision":")" + screen.viewRevision() + R"("},"except":[]})")); + CHECK(screen.element("t", "selection") == morph::ui::parseValue("[]")); + CHECK(screen.element("t", "selectedCount") == morph::ui::Value{std::int64_t{6}}); + CHECK(screen.element("t", "selectedCountKind") == morph::ui::Value{std::string{"exact"}}); + screen.ctrlClick(std::int64_t{3}); + CHECK(screen.element("t", "selectedCount") == morph::ui::Value{std::int64_t{5}}); + screen.click(std::int64_t{2}); + CHECK(screen.element("t", "selectionSet") == morph::ui::parseValue(R"({"keys":[2]})")); +} +// Mutation: select-all selecting the held keys, which sends no view and counts the held rows only. + +TEST_CASE("view selection: a key of except that leaves the view is dropped from it", "[ui][table][selection][review]") { + // selectAll, ctrlClick(3): count 5. An event names key 3 and the keyed reply leaves 3 out of the view (it + // became "closed"), total 5: selectedCount == 5, selectedCountKind == "exact", selectionSet's except == []. + // A key of except the window has not seen since (an event for it whose fetch is in flight): kind "approximate" + // and statusOf(h).message == "about 5 of 6 selected". +} +// Mutation: except keys never pruned, which counts 4. + +TEST_CASE("view selection: its life", "[ui][table][selection]") { + // selectAll; a header sort: selectionSet still has "view". A filter change: selectionSet == {"keys":[]}. +} +// Mutation: the selection ended by a sort, which empties it on the first step. + +TEST_CASE("view selection: not while membership is open", "[ui][table][selection]") { + // A reply with undecided 40: selectAll changes nothing; statusOf(h).message == "Select all is available once 40 rows are + // decided". A reply with undecided 0 and awaiting ["recipe"], `recipe` the level of the filtered column: + // statusOf(h).message == "Select all is available once recipe data has loaded". awaiting ["recipe"] where only the sorted + // column has that level: selectAll selects the view. +} +// Mutation: select-all ignoring `undecided`, which selects a view whose rows are not known. + +TEST_CASE("view selection: the rows changed since the user selected them", "[ui][table][selection]") { + // selectAll, ctrlClick(3), run tag; refuse(tag, "morph.table.viewChanged") while the latest reply has total 7: + // statusOf(h).message == "The rows changed since you selected them: 6 rows now. Run on these?" with an entry + // "Run on these". Choosing it sends TagRows again whose rows.view.revision is the latest reply's viewRevision + // and whose except is [3]. Not choosing it sends nothing. +} +// Mutation: the rerun sending the old revision, which is refused again. + +TEST_CASE("view selection: a Shift range toggles a range of except", "[ui][table][selection]") { + // Six open rows 0..5 at view rows 0..5. selectAll; ctrlClick(1): anchor 1, except == [1]. A Shift range from + // the anchor to view row 3 (selectTableRange, keys 1,2,3 through keysOf): except == [1,2,3], selectedCount == 3. + // ctrlClick(5) twice (anchor 5, selected again); a Shift range from it to view row 2 (keys 2..5): + // except == [1], selectedCount == 5. With maxRangeRows = 2, a range of 3 rows leaves except unchanged and + // statusOf(h).message == "A range selects at most 2 rows". +} +// Mutation: a Shift range in a view selection replacing the selection with the range's keys. + +TEST_CASE("view selection: copying it fetches the view's rows in order", "[ui][table][selection]") { + // 6 open rows, a window holding 2; selectAll, ctrlClick(1), ctrlClick(4); Copy: the window sends page queries + // naming the visible columns, and clipboardText() holds rows 0,2,3,5 in view order, one line each. With + // SessionOptions::maxCopyRows = 3, statusOf(h).message == "A copy holds at most 3 rows" and clipboardText() is unchanged. +} +// Mutation: copying the held rows only, which writes rows 0 and 2. + +TEST_CASE("table rows: a new description revision drops a view selection that names a removed column", + "[ui][table][selection]") { + // Extends Part D's Review Focus 1 test (Task 9, same file; index decision 16). Description rev A with columns + // id, status, c; filters on status and on c; selectAll, ctrlClick(3). A page reply names revision B, and the + // describe reply B lacks c: selectionSet == {"keys":[]} (the view selection was of a filter that named c); + // lastQuery(h).filters.columns does not contain "c"; a TagRows run afterwards carries no "c" anywhere in its body. + // With the filter on status only, the same revision change keeps the view selection, its `except` [3] and its + // filters. +} +// Mutation: setDescription leaving the view selection as it was, which sends a view whose filter names c. +``` + +`selectAll`, `clickRow` and `ctrlClickRow` (this part's `RecordingBackend` drivers) call `setOnSelectAll`'s handler and `onSelectionChange` with `Replace` / `Extend`; `screen.viewRevision()` is the `viewRevision` of the page JSON the test replied with. + +- [ ] **Step 9: Run them; they fail** (`selectionSet` is not an element value). + +Run: `cmake --build --preset cl-debug --target morph_tests && ctest --test-dir build/cl-debug -R "view selection:" --output-on-failure` +Expected: FAIL. + +- [ ] **Step 10: Implement the client side.** + - **`WindowRows`.** `selectView()` returns `SelectAllRefusal{undecided, levels}` when `counts().undecided > 0`, or when `counts().awaiting` holds a level of a column in the current filter (a column filter's column or a group filter's columns; levels from the description's `ColumnInfo::level`), `levels` then being those; otherwise `{filters, retained, viewRevision}` of the latest page reply. `selectedCount` for a view: the view's total less every key of `except` the window holds in the view, a key owed a refetch included (it still subtracts); a key of `except` a reply left out of the view is removed from `except` by the lowering. `kind` is `Exact` when the total is `Exact` and every key of `except` is held and owed nothing (`PageWindow::owed()` names none of them), else `Approximate` (so 6 less a held-but-owed 3 is "about 5 of 6"). For keys: `keys.size()`, `Exact`. `rowsOf` sends page queries of `pageSize` rows naming `columns` and completes with every row of the range. + - `rowsOf` completes with Part E's `FetchedRow{key, viewRow, cells}` (Task 12f), one per view row of the range. + - **`EngineRows`.** `selectView` returns the engine's filter, retained keys and the digest of its view; the lowering never uses it, since a table held whole selects all by keys. `rowsOf` reads the engine. + - **`selectAllRefusalText`.** `"Select all is available once " + n + (n == 1 ? " row is decided" : " rows are decided")`, `n` written with the locale's grouping; with no undecided rows, `"Select all is available once " + levels joined with " and " + " data has loaded"`. + - **The backend.** `setOnSelectAll(handler)`: when set, Ctrl+A, the select-all box and "Select all" call it instead of selecting keys. `setSelectionView(except)`: marks every row whose key is not in `except` until the next `setSelection`. In that state a Ctrl+click reports `Extend` with the keys now selected among the existing keys; the lowering reads every existing key absent from the report as added to `except`, and every present one as removed from it. A plain click reports `Replace` and ends the view selection. A `Shift` range in a view selection (Part D's Task 10a range report by view rows, `selectTableRange`) is resolved through `TableRows::keysOf(first, count)`, bounded by `SessionOptions::maxRangeRows` (over it: Task 10a's refusal through `tableMessage`, and `except` unchanged); a range from an anchor that is selected removes its keys from `except`, one from an anchor in `except` adds them. + - **A new description revision** (Part D's `setDescription`): a view selection whose filter names a column the new description lacks becomes `TableSelection{}`; one that names none is kept. + - **The lowering.** A model-backed table with `selectionMode: multiple` sets `setOnSelectAll`; the handler takes `selectView()`, posts `selectAllRefusalText(…)` through `tableMessage`, or makes the element's selection `TableSelection{.view = …}`. A filter change, and "Show only matching rows" (Task 17b), make it `TableSelection{}`; a sort change keeps it. `selectionSet` is the selection's JSON on every table (`{"keys": […]}` on a table over `rows`); `selection` is the key list, `[]` while the selection is a view; `selectedCount` and `selectedCountKind` come from `selectedCount`. The status line's count reads `"about " + n + " of " + total + " selected"` when the kind is `Approximate`. + - **Run ids.** A `RunStarted::run` is session-wide: `Session` keeps one counter, and `MutationRuntime::send` takes the next id from it for the observers (index decision 24); `MutationRuntime`'s own `_runs` (`src/ui/interpreter.cpp:2396`) is per mutation and two mutations both number their first run 1, so it is not what observers see. + - **Runs.** `MutationRuntime` calls each `RunObserver` of its session in `send` (`started`) and `settle` (`settled`, with the refusal's code and a `rerun` that calls `run` with a new body and the run's bindings). `tableInputsOf(schema)` lists the schema's top-level properties carrying `x-selectionOf` (`Selection`) or `x-rowOf` (`Row`), with the record the keyword names. Each model-backed table registers one `RunObserver` while mounted. On `settled` with code `morph.table.viewChanged`, for each `Selection` input whose body member is a view selection of this table's record, it takes `selectView()` again (same `except`), posts `"The rows changed since you selected them: " + n + " rows now. Run on these?"` through `tableMessage` with the status entry "Run on these", `n` the new `selectedCount`; choosing the entry calls `rerun` with that member replaced. + - **Copy** (Part E's Task 12f path): a view selection whose `selectedCount` is above `maxCopyRows` is refused with `tableText::copyBound(maxCopyRows)` through `tableMessage`; otherwise `rowsOf(0, total, visibleColumns)` page by page, rows of `except` left out, in view order. + +- [ ] **Step 11: Run them and their targets; all pass.** + +Run: `ctest --test-dir build/cl-debug -R "view selection:|table rows:" --output-on-failure`, then all of `morph_tests` and `morph_table_tests`; on `linux-everything`, `ctest --preset linux-everything -R "tables:" --output-on-failure` and all of `morph_net_tests`. +Expected: PASS. + +- [ ] **Step 12: Mutations, docs, commit.** Run each named mutation and keep its failing line. `docs/spec/ui/document.md`: element values `selectionSet` (every table) and `selectedCountKind`; a model-backed table's select-all selects the view. Docs build green. + +```bash +git add include/morph/table include/morph/core include/morph/forms/forms.hpp include/morph/ui src/ui src/qt_quick src/tui tests docs/spec +git commit -m "feat(table): select the whole view and resolve it on the server" +``` + +### Task 17b: Rows kept in view + +**Spec:** "Rows an action changed › Kept in view" (2382–2408); test rows "Retained rows", "Retained rows and the event"; invariant 35; failure mode 3317. + +**Files:** +- Modify: `include/morph/table/query.hpp` (`TableQuery::retain`, `QueryLimits::maxRetained`, `Page::unmatched`, `PageRows::unmatched`; `validate` bounds `retain`), `include/morph/table/description.hpp` (`GenericPage::unmatched`, written and read; `TableQuery::retain` on the wire), `include/morph/table/view_cache.hpp` (`detail::mergeRetained`, used per request), `include/morph/table/engine.hpp` (`Engine::retain`, `retained`, `isUnmatched`; `detail::JobInput::retained`), `include/morph/ui/table_rows.hpp` (`TableCountsView::retained`; `TableRows::retain`, `clearRetained`, `unmatched`, `holdRefetches`, `releaseRefetches`; `RetainResult`), `include/morph/ui/interpreter.hpp` (`SessionOptions::retainHold`), `src/ui/interpreter.cpp` (`retainChanged`, the status line entries, muted unmatched rows), `include/morph/ui/view.hpp` (`RowMarks`, `TableRowView::marks`), `include/morph/ui/backend.hpp` (`TableWidget::setRowMarks`), `include/morph/ui/mount.hpp`, the three table widgets. +- Test: `tests/table/test_table_selection.cpp`, `tests/test_ui_table_model.cpp`. +- Docs: `docs/spec/ui/document.md` (`retainChanged`), `docs/spec/ui/backend_contract.md` and `docs/spec/qt_quick/backend.md` (`setRowMarks`). + +**Interfaces:** +- Consumes: Task 17a's `MembershipDigest`, `RunObserver`, `tableInputsOf`; Task 9's `ChangedRows` reading from a reply (here `changedRowsOf`, below); `ui::ActionInfo::{effects, records}` (`include/morph/ui/document.hpp`). +- Produces: + +```cpp +namespace morph::table { +// TableQuery: std::vector retain; QueryLimits: std::size_t maxRetained = 10'000; +// Page, PageRows, GenericPage: std::vector unmatched; // indices into `rows` +// Engine: +// std::expected retain(std::vector keys); // replaces the retained set +// [[nodiscard]] std::span retained() const noexcept; +// [[nodiscard]] bool isUnmatched(std::size_t viewRow) const; +namespace detail { +struct MergedView { + std::vector order; // source rows, in view order + std::vector unmatched; // view rows of retained rows the filter rejects + std::int64_t undecided = 0; + MembershipDigest digest; +}; +[[nodiscard]] MergedView mergeRetained(std::span baseOrder, std::int64_t baseUndecided, + MembershipDigest baseDigest, std::span retainedRows, + std::function const& before, + std::function const& match, + std::function const& keyOf); +} +} // namespace morph::table + +namespace morph::ui { +enum class RetainResult : std::uint8_t { Retained, OverBound }; +// TableCountsView: std::int64_t retained = 0; // held view rows that are retained and unmatched +// TableRows gains: +// [[nodiscard]] virtual RetainResult retain(std::span keys) = 0; // adds +// virtual void clearRetained() = 0; +// [[nodiscard]] virtual bool unmatched(std::int64_t index) const = 0; +// virtual void holdRefetches(std::uint64_t run) = 0; +// virtual void releaseRefetches(std::uint64_t run) = 0; +// SessionOptions: std::chrono::milliseconds retainHold{2000}; +struct RowMarks { bool busy = false; bool locked = false; bool unmatched = false; std::string tooltip; }; // view.hpp +// TableRowView: std::optional> marks; +// TableWidget: virtual void setRowMarks(Widget& row, RowMarks const& marks) = 0; +struct ChangedRowsValue { std::vector rows; std::vector failed; }; +[[nodiscard]] ChangedRowsValue changedRowsOf(Value const& reply, ActionInfo const& action, std::string_view record); +} // namespace morph::ui +``` + +- [ ] **Step 1: Write the failing engine and server tests** in `tests/table/test_table_selection.cpp`, over Task 17a's `SelectionModel`. + +```cpp +TEST_CASE("retained rows: they stay in the view, listed in unmatched", "[table][retain]") { + SelectionModel model; + TableQuery query{.sort = {{.column = "mass", .dir = SortDirection::Descending}}, .filters = open(), + .retain = {RowId{std::int64_t{1}}, RowId{std::int64_t{7}}, RowId{std::int64_t{8}}}}; + for (std::int64_t key : {1, 7, 8}) { // the action closed row 1; rows 7 and 8 were closed already + model.source->patchRow(RowId{key}, {Cell{std::string{"closed"}}, Cell{tabletest::dec(std::to_string(key) + ".0")}}); + } + auto const page = apply(*model.source, query).value(); + std::vector keys; + for (std::size_t i = 0; i < page.rows.size(); ++i) { keys.push_back(model.source->snapshot()->rowId(page.rows[i])); } + CHECK(keys == std::vector{std::int64_t{8}, std::int64_t{7}, std::int64_t{5}, std::int64_t{4}, + std::int64_t{3}, std::int64_t{2}, std::int64_t{1}, std::int64_t{0}}); + CHECK(page.unmatched == std::vector{0, 1, 6}); + CHECK(page.total == 8); + TableQuery const plain{.sort = query.sort, .filters = open()}; + CHECK(page.viewRevision != apply(*model.source, plain).value().viewRevision); +} +// Mutation: retained rows appended after the base view instead of merged by the sort's keys. + +TEST_CASE("retained rows: the shared view entry is not changed by one request's rows", "[table][retain]") { + // Through Task 6's ViewCache: a request retaining {7}, then a request of the same sort and filter retaining + // nothing: the second reply has 5 rows, no 7, and empty `unmatched`; the cache computed the view once. +} +// Mutation: the merged view written into the shared view cache entry, which serves one session's retained rows to +// another. + +TEST_CASE("retained rows: retaining adds, never removes", "[table][retain][property]") { + // Property over 200 random tables (seeded): for random filters and a random retain set R of up to 20 keys, + // view(with R) ⊇ view(without R); view(with R) \ view(without R) == R ∩ source keys \ view(without R); + // `unmatched` names exactly the rows of R whose filter evaluates to No. +} +// Mutation: a retained row counted twice when the filter passes it too, which breaks the second equation. + +TEST_CASE("retained rows: the bound and the wire", "[table][retain]") { + CHECK(QueryLimits{}.maxRetained == 10'000); + // validate of a query retaining 10'001 keys: LimitExceeded; 10'000: accepted. A GenericPage with unmatched + // [0, 6] round-trips; a page read without "unmatched" has it empty. +} +// Mutation: no bound, which accepts 10'001. + +TEST_CASE("retained rows: a client-mode engine does the same", "[table][retain]") { + // An Engine over the same source, filter open(), sort mass desc; patch rows 1,7,8 closed; retain({1,7,8}): + // the view's keys are {8,7,5,4,3,2,1,0}; isUnmatched(0), (1), (6) are true, the rest false; setFilter(open()) + // clears the retained set (retained().empty()) and the view is {5,4,3,2,0}. +} +// Mutation: Engine::retain ignored by the filter run, which leaves 1, 7 and 8 out. +``` + +Write the commented cases' assertions exactly; `apply`'s `PageRows` holds source rows in `rows` (use its existing member if it is named differently). + +- [ ] **Step 2: Run them; they fail to compile** (`TableQuery::retain`). + +Run: `cmake --build --preset cl-debug --target morph_table_tests` +Expected: compile error naming `retain`. + +- [ ] **Step 3: Implement.** `mergeRetained`: R = the retained rows (source rows) not in `baseOrder`; sort R by `before`, which is the query's comparator over the sort's key columns with ties to the lower source row (the engine's order); merge R into `baseOrder` with the same comparator, so the result is the view the sort would give over base ∪ R. Every row of R is in the view whatever its `match` (a row whose filter is `Unknown` is in the view, not undecided): `undecided = baseUndecided −` the rows of R that were undecided; `digest` = `baseDigest` plus each key of R; `unmatched` lists the view row of each row of R whose `match` is `No`, ascending. `apply` and the view cache path compute the base view as before and call `mergeRetained` per request when `retain` is not empty; the cache entry is never written with it. The total is the merged view's size. A keyed query reads the merged view; distinct values ignore `retain`. A reply's `unmatched` holds indices into its own `rows`. `Engine::retain` stores the keys and runs the filter again with them admitted; `setFilter` clears them. + +- [ ] **Step 4: Run them; they pass.** + +Run: `ctest --test-dir build/cl-debug -R "retained rows:" --output-on-failure` +Expected: 5 test cases pass. + +- [ ] **Step 5: Write the failing client tests** in `tests/test_ui_table_model.cpp`, over the ten rows, a table with `"retainChanged": true` filtered to `open()`, and a mutation `close` running `CloseRows` (declares `updatesTable<&Rows::rows>`, result `{"rows": ChangedRows}`). + +```cpp +TEST_CASE("retained rows: the rows an action changed stay in view", "[ui][table][retain]") { + // run close; settle with {"rows":{"rows":[{"key":1},{"key":2},{"key":3}],"failed":[]}}: the next query carries + // retain [1,2,3]; answer it with those rows unmatched: they are drawn with RowMarks{.unmatched = true, + // .tooltip = "This row no longer matches the filter"}, counts().retained == 3, statusOf(h).standing == + // "3 rows shown that no longer match" with the standing entry "Show only matching rows"; it is still there after + // h.advance(7000) and a click on a row. Choosing "Show only matching rows": the next + // query has no retain, and a view selection, if any, is empty. A filter change does the same. +} +// Mutation: retained keys dropped when the next page arrives, which loses them from the second query. + +TEST_CASE("retained rows: an action over the bound retains none", "[ui][table][retain]") { + // With 9'999 keys retained, a reply naming 2 changed rows: no key is added, statusOf(h).message == + // "The changed rows are not kept in view: a table keeps at most 10,000"; the 9'999 stay. With 9'998 retained, + // the same reply adds both. +} +// Mutation: a partial add up to the bound, which retains one of the two. + +TEST_CASE("retained rows: an event before its action's reply is held", "[ui][table][retain]") { + // run close (it declares an effect on the table); fireEvent(record, [1]) before the reply: no query is sent. + // settle the run: exactly one query is sent, keyed or page per Task 7's refresh plan, carrying retain [1]. + // A second run, an event, and the scheduler advanced by retainHold (2 s) without a reply: one query is sent + // then, and none when the reply arrives later except its retained rows' refetch. + // On a table without retainChanged, the event's query is sent at once. +} +// Mutation: no hold, under which row 1 leaves the view and comes back. +``` + +Write the commented assertions exactly as stated. + +- [ ] **Step 6: Run them; they fail.** + +Run: `cmake --build --preset cl-debug --target morph_tests && ctest --test-dir build/cl-debug -R "retained rows:" --output-on-failure` +Expected: FAIL. + +- [ ] **Step 7: Implement.** + - `changedRowsOf(reply, action, record)`: for each path of the `ActionRecordInfo` of `record`, the path names a `ChangedRows`' `rows` member (`".rows"`, or `"rows"` for the result itself); the value at the path's parent is read as `{"rows": [{"key", "from"?}], "failed": [{"key", "message"}]}`. A number is an integer key, a string a text key. + - `WindowRows::retain(keys)`: the keys not already retained are added when the set stays within `maxRetained`, else none is and the result is `OverBound`; each query from then on carries the set in `retain`. `clearRetained` empties it. `unmatched(index)` is true for a held row the last reply that supplied it listed in `unmatched`; `counts().retained` counts those. `holdRefetches(run)` records the run and, while any run is recorded and for at most `SessionOptions::retainHold` from the first, makes a record event only mark what it would fetch; `releaseRefetches(run)` (or the hold's timer, on the session's scheduler) sends what was marked as one refetch. `EngineRows` maps these to `Engine::retain`, `isUnmatched`, and no-op holds. + - The lowering, on a table with `retainChanged` (a literal flag, default `false`): its `RunObserver::started` calls `holdRefetches(run)` for an action whose catalog `effects` name this table's record; `settled` calls `retain` with the keys of `changedRowsOf(reply, …).rows` (and their `from` keys, Task 17c), posts the over-bound message through `tableMessage` on `OverBound` (a refusal: transient), then `releaseRefetches(run)`. A filter change and "Show only matching rows" call `clearRetained` and end a view selection. Retained rows are not part of the stored table state. + - `RowMarks`: the mount calls `setRowMarks(row, marks)` after `setRowKey` for a row whose `marks` is set, and on every change. Qt Quick draws `unmatched` muted, `busy` with the theme's busy mark, `locked` with the theme's locked hatch, and `tooltip` as the row's tooltip; the terminal draws `unmatched` dim and prefixes a busy row with the busy glyph and a locked row with the lock glyph. + +- [ ] **Step 8: Run them and the targets; all pass.** + +Run: `ctest --test-dir build/cl-debug -R "retained rows:" --output-on-failure`, then all of `morph_tests`, `morph_table_tests`, and in `build/cl-qt-debug` `morph_qt_quick_tests`, `morph_tui_tests`. +Expected: PASS. + +- [ ] **Step 9: Mutations, docs, commit.** `docs/spec/ui/document.md`: `retainChanged`. Backend specs: `setRowMarks`. Docs build green. + +```bash +git add include/morph/table include/morph/ui src/ui src/qt_quick src/tui tests docs/spec +git commit -m "feat(table): keep rows an action changed in view" +``` + +### Task 17c: Key changes, busy, failed and locked rows + +**Spec:** "A key that changes" (2409–2420), "Rows at work, rows that failed" (2422–2443), "A locked row" (452–455); test rows "A key that changes", "Busy and failed rows", "Locked rows and companions" (the lock half); failure mode 3318. + +**Files:** +- Modify: `include/morph/table/description.hpp` (`ChangedRow::from` on Part A's `ChangedRow{RowId key;}`; `FailedRow`; `ChangedRows::failed`), `include/morph/core/records.hpp` (`RecordInfo::previousKeyField`), `include/morph/core/registry.hpp` (`BRIDGE_REGISTER_TABLE` sets it to `"from"`), `include/morph/core/changes.hpp` (`addUpdatesChange` collects it), `include/morph/ui/table_rows.hpp` (`RowStates`, `TableRows::states`, `TableRows::moveKey`, `lockText`, `refusesEdit`, `TableCountsView::busy`), `include/morph/ui/view.hpp` (`KeyMoves`, `TableOptions::keyMoves`), `include/morph/ui/backend.hpp` (`TableWidget::moveKey`), `include/morph/ui/mount.hpp`, the three table widgets, `src/ui/interpreter.cpp` (busy rows from inputs, failed marks, "Select them", "Clear marks", locked rows, key moves, element value `busyCount`). +- Test: `tests/net/test_push_tables.cpp`, `tests/test_ui_table_model.cpp`. +- Docs: `docs/spec/ui/document.md` (element value `busyCount`), `docs/spec/ui/backend_contract.md` (`moveKey`). + +**Interfaces:** +- Consumes: Task 17a's `RunObserver`, `tableInputsOf`, `TableSelection`; Task 17b's `changedRowsOf`, `retain`, `RowMarks`; Task 15's `TableDescription::rowLock`. +- Produces: + +```cpp +namespace morph::table { +struct ChangedRow { RowId key; std::optional from; }; +struct FailedRow { RowId key; std::string message; }; +// ChangedRows: std::vector rows; std::vector failed; +} +namespace morph::model { +// RecordInfo: std::optional previousKeyField; // "from" on a table's record +} +namespace morph::ui { +class RowStates { +public: + void busyKeys(std::uint64_t run, std::vector keys); + void busyView(std::uint64_t run, std::vector except, std::int64_t count); + void settle(std::uint64_t run); + void fail(std::span rows); + void clearFailed(std::span keys); + void clearFailed(); + void move(table::RowId const& from, table::RowId const& to); + [[nodiscard]] bool isBusy(table::RowId const& key) const; + [[nodiscard]] std::optional failure(table::RowId const& key) const; + [[nodiscard]] std::int64_t busyCount() const; + [[nodiscard]] std::vector failedKeys() const; // in the order they failed +}; +// TableRows gains: +// [[nodiscard]] virtual RowStates& states() = 0; +// [[nodiscard]] virtual RowStates const& states() const = 0; +// virtual void moveKey(table::RowId const& from, table::RowId const& to) = 0; // retained and states +// TableCountsView: std::int64_t busy = 0; +[[nodiscard]] std::optional lockText(TableRowRef const& row, table::TableDescription const& description); +[[nodiscard]] bool refusesEdit(TableRows const& rows, std::int64_t index, table::TableDescription const& description); +struct KeyMoves { std::uint64_t generation = 0; std::vector> moves; }; // view.hpp +// TableOptions: Prop keyMoves; +// TableWidget: virtual void moveKey(Key const& from, Key const& to) = 0; // its selection mark and current row +} +``` + +`refusesEdit` is what Part G's cell editing (editing.md "Read-only is one rule", Task 24) consults: a busy or locked row takes no cell edit. + +- [ ] **Step 1: Write the failing walk test** in `tests/net/test_push_tables.cpp`: `"tables: a changed key's event names both keys"` — the file's model gains `SaveAs{std::int64_t key;}` returning `{"rows": {"rows": [{"key": 10, "from": 2}], "failed": [{"key": 4, "message": "Locked"}]}}` and declaring `updatesTable<&Rows::rows>`; a watcher of `selnet::Rows.rows` receives one event whose keys are exactly `{10, 2}` (in any order) and not `4`. Mutation: the walk ignoring `from`, which leaves the old row's watchers untold (keys `{10}`). + +- [ ] **Step 2: Run it; it fails** (keys `{10}`). + +Run: `cmake --build --preset linux-everything --target morph_net_tests && ctest --preset linux-everything -R "tables: a changed key" --output-on-failure` +Expected: FAIL, keys `{10}`. + +- [ ] **Step 3: Implement.** `RecordInfo::previousKeyField` (absent by default); `BRIDGE_REGISTER_TABLE` sets it to `"from"` on the table's record; `addUpdatesChange` (`include/morph/core/changes.hpp:180`), after collecting `keyField` at each path, collects `*info->previousKeyField` at the same paths when it is set. A `failed` row is never at a `rows` path, so it is not collected. Run Step 2's command: PASS. + +- [ ] **Step 4: Write the failing client tests** in `tests/test_ui_table_model.cpp`, over the ten rows of Task 17a (no filter) and a description with `rowLock = "lockedBy"`; mutations `save` (`SaveAs`, input `{"rows": RowSelection}`), `touch` (`TouchRow`, input `{"row": RowRef}`), both declaring `updatesTable`. + +```cpp +TEST_CASE("key change: the session that ran the action moves what it holds", "[ui][table][keys]") { + // Rows 2 and 5 selected (keys), focus on row 2, row 2 failed from an earlier run, row 2 retained. + // run save; settle with {"rows":{"rows":[{"key":10,"from":2}],"failed":[]}}: selectionSet == {"keys":[10,5]} + // (key 10 in key 2's place), the backend got moveKey(2, 10), states().failure(10) holds the message and + // failure(2) is empty, the retained set holds 10 and not 2; fireEvent(record, [10, 2]) sends one keyed query + // naming both 10 and 2. +} +// Mutation: the client ignoring `from`, which leaves 2 selected and 10 not. + +TEST_CASE("key change: the action's event before its reply", "[ui][table][keys]") { + // Rows 2 and 5 selected, focus on 2. runMutation save; fireEvent(record, [10, 2]) and the keyed reply that + // leaves 2 out of the view and holds 10, all before the reply: selectionSet still holds 2. settleMutation with + // {"rows":{"rows":[{"key":10,"from":2}],"failed":[]}}: selectionSet == {"keys":[10,5]} and the backend got + // moveKey(2, 10). The same with a reply that names no `from`: after settle, 2 leaves the selection. +} +// Mutation: no deferral while the run is in flight, which drops 2 before `from` names it and selects [5]. + +TEST_CASE("key change: onto a key already held", "[ui][table][keys][review]") { + // Rows 3 and 7 held; 7 and 3 selected. settle with {"rows":{"rows":[{"key":7,"from":3}],"failed":[]}}: + // selectionSet == {"keys":[7]} (7 once); fireEvent(record, [7, 3]): the keyed query names 3 and 7. +} +// Mutation: a move that appends `to` without checking it is held, which selects 7 twice. + +TEST_CASE("key change: a from the client does not hold", "[ui][table][keys]") { + // settle with {"rows":{"rows":[{"key":10,"from":99}],"failed":[]}}: nothing moves (selection, focus, states + // unchanged); the event's keyed query names 10. +} +// Mutation: a move inventing state for 99, which adds 10 to the selection. + +TEST_CASE("key change: another session's event drops the old key", "[ui][table][keys]") { + // Row 2 selected and focused; fireEvent(record, [10, 2]) with no run of this session, the keyed reply without + // row 2: 2 leaves the selection, focus moves to the row now at 2's view index. +} +// Mutation: a selection kept for keys that left the view, which keeps 2. + +TEST_CASE("busy rows: a run's rows are busy until it settles", "[ui][table][busy]") { + // Keys 1, 2 selected; run save: rows 1 and 2 drawn with RowMarks{.busy = true}; busyCount == 2; + // statusOf(h).standing == "Saving 2 rows"; refusesEdit for both. refuse(save, "lims.rows.denied"): no row is busy, + // busyCount == 0. A view selection of 6 rows less 3 held as except: every held row but 3 is busy, busyCount 5. + // run touch {"row": 4}: row 4 busy. +} +// Mutation: busy state cleared only on success, which leaves rows 1 and 2 busy after the refusal. + +TEST_CASE("busy rows: overlapping runs", "[ui][table][busy][review]") { + // Run A over keys {1,2}, run B over {2,3}: busyCount == 3. Settle A: row 2 still busy, busyCount == 2. + // Settle B: busyCount == 0. +} +// Mutation: a busy set keyed by row only, which frees row 2 when A settles. + +TEST_CASE("busy rows: two mutations' runs do not share an id", "[ui][table][busy]") { + // Mutations `save` and `touch`, each running for the first time: save over {1, 2}, touch {"row": 5}. Both + // runs are busy (busyCount == 3); settle touch: rows 1 and 2 stay busy, row 5 does not (busyCount == 2); + // settle save: busyCount == 0. +} +// Mutation: run ids taken from MutationRuntime's own counter, under which settling touch (run 1) frees save's rows. + +TEST_CASE("failed rows: marked until the row's next change", "[ui][table][failed]") { + // run save over {4, 5}; settle with {"rows":{"rows":[{"key":5}],"failed":[{"key":4,"message":"Locked"}]}}: + // row 4 has the err tone and tooltip "Locked"; statusOf(h).standing == "1 row could not be saved", standingEntries labelled "Select them" and + // "Clear marks". "Select them": selectionSet == {"keys":[4]}. An event naming 5 (the action's own): 4 stays + // marked. An event naming 4 after the run settled: the mark goes. Again failed; "Clear marks": the mark goes. + // Again failed; a new run naming row 4: the mark goes when the run starts. On a rowScoped table, an event with + // no keys leaves the mark. +} +// Mutation: a failed mark cleared by any event, which clears it on the action's own event. + +TEST_CASE("locked rows: a locked row draws locked and refuses an edit", "[ui][table][locked]") { + // Row 6's lockedBy is "Being edited by another user": RowMarks{.locked = true, .tooltip = "Being edited by + // another user"}, lockText == that text, refusesEdit true; it can be selected (click(6) selects it) and copied. + // Row 5's lockedBy is "": not locked, refusesEdit false. +} +// Mutation: lockText reading `rowTone`'s column, which locks no row. +``` + +Write each commented assertion exactly as stated. + +- [ ] **Step 5: Run them; they fail.** + +Run: `cmake --build --preset cl-debug --target morph_tests && ctest --test-dir build/cl-debug -R "key change:|busy rows:|failed rows:|locked rows:" --output-on-failure` +Expected: FAIL. + +- [ ] **Step 6: Implement.** + - `RowStates`: busy runs are kept per run (a key run's keys; a view run's `except` and the `count` it was started with); `isBusy(k)` is true when any key run holds `k` or any view run does not name `k` in its `except`; `settle(run)` drops the run whatever its outcome. **`busyCount`**: with no view run in flight, the number of distinct keys across the key runs (runs {1,2} and {2,3} count 3); with a view run in flight, the largest `count` among the view runs plus the distinct keys of key runs that every view run names in its `except` (those keys are outside the view runs' rows). `fail` records each key's message; `clearFailed(keys)` and `clearFailed()` drop marks. `move(from, to)`: moves the busy membership and the failed mark of `from` to `to`; when `to` already has one, `to`'s is kept and `from`'s dropped. + - The lowering's `RunObserver`: `started` reads `tableInputsOf(Catalog::schema(action))`; for each input naming this table's record, a `Selection` member's key selection is `busyKeys(run, keys)`, its view selection `busyView(run, except, selectedCount)`, a `Row` member `busyKeys(run, {key})`; each key the run names has its failed mark cleared. `settled` calls `settle(run)`; then, on success, with `changedRowsOf(reply, …)`: `fail(failed)`; for each row with `from` that this session holds in any way (selection, focus, retained, states), `TableRows::moveKey(from, key)`, the selection's `from` replaced by `key` (kept once if `key` is already selected), and a `KeyMoves` entry for the mount, whose generation increases; a `from` the client does not hold moves nothing. An event after the run settled clears the failed marks of the keys it names; an event with no keys clears none. + - **Held while an action is in flight.** A key leaving the view drops it from the selection, the focus and `RowStates`; but while a run of this session whose action's catalog `effects` name this table's record is in flight, that drop is deferred for the keys that left: they are kept (and still counted) until every such run has settled, so the action's own event, which can precede its reply (table.md:2396–2402), does not lose the old key a `from` then names. At settle the moves are applied first, then each deferred key that is still out of the view and was not a `from` is dropped as any key that left the view. + - The mount applies each new `KeyMoves` generation's moves with `TableWidget::moveKey(from, to)`: the backend moves its selection mark and its current row from `from` to `to`. + - Drawing: a row's `RowMarks` is `{busy = isBusy, locked = lockText has a value, unmatched = TableRows::unmatched, tooltip = lockText, else failure, else the unmatched text}`; a failed row's tone is `err`. `lockText` is the row's `rowLock` cell when it is non-empty text. `refusesEdit` is busy or locked. + - Element value `busyCount`; the standing part of the status line (not `tableMessage`) holds `"Saving " + n + (n == 1 ? " row" : " rows")` while `busyCount > 0`, and `n + (n == 1 ? " row could not be saved" : " rows could not be saved")` with the standing entries "Select them" (selection = `failedKeys()`) and "Clear marks" (`clearFailed()`), while `failedKeys()` is not empty. + +- [ ] **Step 7: Run them and the targets; all pass.** + +Run: `ctest --test-dir build/cl-debug -R "key change:|busy rows:|failed rows:|locked rows:" --output-on-failure` and all of `morph_tests`; `ctest --preset linux-everything -R "tables:" --output-on-failure` and all of `morph_net_tests`; and `morph_qt_quick_tests`, `morph_tui_tests` in `build/cl-qt-debug`. +Expected: PASS. + +- [ ] **Step 8: Mutations, docs, commit.** `docs/spec/ui/document.md`: element value `busyCount`. `docs/spec/ui/backend_contract.md`: `moveKey`. Docs build green. + +```bash +git add include/morph/table include/morph/core include/morph/ui src/ui src/qt_quick src/tui tests docs/spec +git commit -m "feat(table): key changes, busy, failed and locked rows" +``` + +### Task 17d: The application's menu entries and a disabled header + +**Spec:** "The application's menu entries" (2359–2378), "The header while the application is busy" (2445–2451); test rows "The application's menu entries", "A disabled header"; invariant 31. + +**Files:** +- Modify: `include/morph/ui/view.hpp` (`TableOptions::cellMenu`, `headerEntries`, `headerEnabled`), `include/morph/ui/backend.hpp` (`TableWidget::setCellMenu`), `include/morph/ui/mount.hpp`, the three table widgets, `src/ui/interpreter.cpp` (`cellMenu`, `headerMenu`, `headerEnabled`; "Filter by this value", "Exclude this value", "Filter by this value in any "), `src/ui/document.cpp` (load-time checks). +- Test: `tests/test_ui_table_model.cpp`, `tests/qt_quick/test_qt_quick_table.cpp`. +- Docs: `docs/spec/ui/document.md` (`cellMenu`, `headerMenu`, `headerEnabled`), `docs/spec/ui/backend_contract.md` and `docs/spec/qt_quick/backend.md` (`setCellMenu`). + +**Interfaces:** +- Consumes: Part E's `TableOptions::menu(std::optional)` / `onMenu` and `TableWidget::setMenu` (Task 12b), `ui::headerMenu(HeaderMenuInput)`, `ui::menuEntries` and `MenuCommand::{Copy, CopyWithHeaders}` (Task 12a), its copy (Task 12f) and `ScreenCore::tableMessage` (Task 12c); Task 8a's `FilterEntry::oneOf`; Task 8b's `GroupFilter{group, columns, mode, match, combine, include, exclude}`; Task 15's `operandOf`, `cellLook`; the `menu` node's entry lowering in `src/ui/interpreter.cpp` (its `{label, icon, keys, checked, enabled, onClick, items, description, visible}` shape). +- Produces: + +```cpp +namespace morph::ui { +// TableOptions gains: +// std::function(Key const& row, std::size_t column)> cellMenu; // built-ins, separator, app's +// std::function(std::size_t column)> headerEntries; // the app's, after built-ins +// Prop headerEnabled = true; +// TableWidget gains: +// virtual void setCellMenu(std::function(Key const& row, std::size_t column)> entries, +// std::function const& path)> chosen) = 0; +} +``` + +`path` is the chosen entry's index at each menu level. The mount turns `MenuItem`s into `MenuEntry`s (hidden entries and the separators they strand are left out, as `Menu` does) and runs the chosen item's `onSelect`. + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_table_model.cpp`, over a model-backed table with columns `id`, `mass` (Decimal), `companyA`/`companyB` (Text, declared group `company`, label "Company") and `"selectionMode":"multiple"`: + +```cpp +TEST_CASE("table menus: the application's entries read the row, the column and the value", "[ui][table][menu]") { + // cellMenu [{"label": {"fn":"concat","args":[{"ref":"column"}," = ",{"ref":"value"}," of ",{"ref":"row.id"}]}, + // "onClick": [{"set":"picked","to":{"ref":"row.id"}}]}]: + // rightClickCell(table, 2, ): the menu of row 2's mass cell is ["Copy", "Copy with headers", "Filter by this value", "Exclude this value", + // separator, "mass = 0.2 of 2"]; choosing the last sets picked to 2. + // Rows 1 and 3 selected, a right-click on row 2: the selection is {"keys":[2]}; a right-click on row 1 while + // 1 and 3 are selected keeps both. + // headerMenu [{"label":"Units…","visible":{"fn":"eq","args":[{"ref":"column"},"mass"]}}]: menuOf(table, ) + // ends with a separator and "Units…"; the id header's has no "Units…". +} +// Mutation: the cell menu read in the table's scope, which reads `row` as null. + +TEST_CASE("table menus: filter by this value", "[ui][table][menu]") { + // "Filter by this value" on mass 0.2: filters == {"columns":{"mass":{"include":[{"oneOf":["0.2"]}]}}}, + // replacing mass's earlier entries. "Exclude this value" on 0.3 then 0.4: exclude == [{"oneOf":["0.3","0.4"]}]. + // On an empty cell: include == [{"isEmpty":true}]. On an unloaded cell both entries are disabled. + // On companyA "F": the menu has "Filter by this value in any Company", which sets + // groups == [{"group":"company","include":[{"oneOf":["F"]}]}]. +} +// Mutation: "Filter by this value" adding to include instead of replacing it, which keeps the earlier entry. + +TEST_CASE("table menus: an entry's keys work while the table has focus", "[ui][table][menu]") { + // cellMenu [{"label":"Open","keys":"Ctrl+O","onClick":[{"set":"opened","to":{"ref":"row.id"}}]}], focus on + // row 3: Ctrl+O sets opened to 3; with focus elsewhere, Ctrl+O does nothing. A query of this table names + // the columns the entries read (`id`) for every row. +} +// Mutation: keys bound only while the menu is open. + +TEST_CASE("table menus: a disabled header", "[ui][table][header]") { + // headerEnabled {"ref":"idle"}, idle false: clickHeader, clickFilter and moveColumn change nothing, + // send no query, and each sets statusOf(h).message to "Sorting, filtering and arranging are unavailable right now"; + // selecting, copying and cellMenu entries still work. idle becomes true: a header click sorts. +} +// Mutation: headerEnabled read once at mount, which leaves the header enabled after the condition turns false. +``` + +And in `tests/qt_quick/test_qt_quick_table.cpp`: `"qt_quick table: a right-click opens the cell menu on that cell and selects its row alone"` — rows 1 and 3 selected, a right-click on row 2's second cell: the backend's `onSelectionChange` reports `Replace` `{2}`, the focused cell is (2, 1), and the popup's entries are the ones `setCellMenu` returned; the menu key on the focused cell opens the same menu. Mutation: the menu opened without moving focus. + +Write each commented assertion exactly as stated. + +- [ ] **Step 2: Run them; they fail.** + +Run: `cmake --build --preset cl-debug --target morph_tests && ctest --test-dir build/cl-debug -R "table menus:" --output-on-failure`; `cmake --build build/cl-qt-debug --target morph_qt_quick_tests && ctest --test-dir build/cl-qt-debug -R "qt_quick table: a right-click" --output-on-failure` +Expected: FAIL. + +- [ ] **Step 3: Implement.** + - The lowering builds `cellMenu(row, column)`: "Copy" and "Copy with headers" (Part E's `MenuCommand::Copy` / `CopyWithHeaders`), "Filter by this value", "Exclude this value", for a column of a declared group "Filter by this value in any " + the group's label, a separator, then the document's `cellMenu` entries lowered in the row's scope with `column` (the column's id) and `value` (the cell as a document value) bound beside `row`. Its `headerEntries(column)` is the document's `headerMenu` in a scope with `column`; the lowering's `TableOptions::menu(column)` (Part E) returns `ui::menuEntries(ui::headerMenu(input))`, then a separator and `headerEntries(column)`, and `onMenu` routes a path past the built-ins to the chosen `MenuItem`. + - "Filter by this value": the column's filter becomes `{include: [{oneOf: [operandOf(cell)]}]}`; "Exclude this value": `operandOf(cell)` is appended to the column's `exclude` `oneOf` entry, made when absent; an empty cell uses `{isEmpty: true}`; both are disabled when `cellLook` is `Unloaded` or `Invalid`. The group entry sets the filter of group `` to `{group: id, include: [{oneOf: [operand]}]}`, replacing that group's filter. + - A right-click, the menu key or a long press on a cell focuses it, makes the selection that row alone when it is not selected (reported as `Replace`), and opens the menu `setCellMenu`'s `entries` returns. An entry's `keys` are bound while the table has focus and run the entry for the focused cell. The columns the entries' expressions read are fetched for every row, as `rowTone`'s are (Task 9's column analysis). + - `headerEnabled` (a condition, default `true`) is a `Prop` on `TableOptions`: while false, the mount disables the header-menu entries whose `MenuCommand` sorts, filters or arranges, `Columns` and `SortDialog` included (all but `Copy` and `CopyWithHeaders`), and the refusal itself is the lowering's: its `onSort`, `onFilter`, `onGroupFilter`, `onColumnMove` and `onColumnResize` handlers (Part E) read `headerEnabled` when called, change nothing, and post the message in the test through `ScreenCore::tableMessage`. The mount, which cannot reach `ScreenCore`, only greys the entries. Selection, scrolling, copy and the application's entries stay. + - `src/ui/document.cpp`: `cellMenu` and `headerMenu` take the `menu` node's entry shape and are reported as its entries are. + +- [ ] **Step 4: Run them and the targets; all pass.** Step 2's commands, then all of `morph_tests`, `morph_qt_quick_tests`, `morph_tui_tests`. Expected: PASS. + +- [ ] **Step 5: Mutations, docs, commit.** `docs/spec/ui/document.md`: `cellMenu`, `headerMenu`, `headerEnabled`. Backend specs: `setCellMenu`. Docs build green. + +```bash +git add include/morph/ui src/ui src/qt_quick src/tui tests docs/spec +git commit -m "feat(ui): application entries in table menus and a disabled header" +``` diff --git a/docs/superpowers/plans/2026-11-17-one-table-g-editing.md b/docs/superpowers/plans/2026-11-17-one-table-g-editing.md new file mode 100644 index 000000000..9b8ddfeaf --- /dev/null +++ b/docs/superpowers/plans/2026-11-17-one-table-g-editing.md @@ -0,0 +1,1150 @@ +# Editing in the Table — Implementation Plan (Part G) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build `docs/spec/table/editing.md`: grids of a form, derived values, cell selection with paste and fill, row operations, header bands, spans, summary rows, fixed grids, and cells of a model's table edited through `CellEdit` and `RowEdit` actions. + +**Architecture:** A grid is a form's `ObjectArray` field laid out by the table pipeline: `ui::FormRowsSource` is the `table::DataSource` over the field's `RowState`s, and `ui::GridController` owns focus, the editor, the walk, cell selection, paste, fill, clear and row operations over a `ui::GridSource`, which a form grid, a fixed grid and a model's table each implement. Every read-only decision is one function, `ui::cellReadOnly`. Derived values are rounds of the form session (`FormSession::noteCommit`, `loadDerived`) over the action's evaluation and the form's derive action; the server recomputes row members and clears derived ones. Bands, spans and summary rows are the mount's, for any table. + +**Tech Stack:** C++23, CMake (MSVC preset `cl-debug`), header-only morph, glaze, Catch2, Qt 6 Quick (offscreen `morph_qt_quick_tests` only), the terminal backend (`morph_tui_tests`). + +**Spec:** `docs/spec/table/editing.md` — all sections: "Two homes for an edit", "A grid of a form", "Cells of a model's table", "The cell", "Walking", "Selecting cells, clipboard and fill", "Rows of a grid", "Layouts", "Document vocabulary", "Invariants" (1–9), "API surface", "Tests", "Failure modes", "Neighbouring specs", "Delivery order" steps 1–6. It builds on `docs/spec/table/table.md` ("The client table model", "The renderer", "How a column shows its cells", "Acting on rows") and `docs/spec/forms/engine.md`. Roadmap: `docs/superpowers/plans/2026-11-17-one-table.md` (Tasks 19–24, its Global Constraints, standard task loop and Review Focus items 3 and 5). + +Part G of 7; index: 2026-11-17-one-table.md + +## Global Constraints + +Every constraint of the roadmap's "Global Constraints" applies; the ones that bite here, and this part's own: + +- Comments and specs state what the code does now and why; no history, no issue numbers, no commit hashes (AGENTS.md). +- Every public symbol has complete Doxygen `@param`/`@tparam`/`@return`. Docs build: `cmake -S . -B build/doc -G Ninja -DMORPH_BUILD_DOCUMENTATION=ON -DMORPH_BUILD_TESTS=OFF -DMORPH_BUILD_EXAMPLES=OFF && cmake --build build/doc --target doc` — exits 0. +- morph is public: test fixtures use the neutral `lims` domain (`Lims_RecordSpecimens`, `Lims_SetLimits`, `Lims_EditSpecimenCount`); no company or customer name anywhere. +- Commits authored `yaraslau `, message body ends with `Co-Authored-By: Claude Opus 5.5 `. +- No GUI application is launched; Qt Quick behaviour is tested only in `morph_qt_quick_tests` (ctest sets `QT_QPA_PLATFORM=offscreen`). +- Every test below names its mutation; step "Mutate" of each task applies it by hand, runs the test, sees it fail, reverts, and pastes the failing line into the commit body. +- Values the spec pins, verbatim: `deriveDebounce` default **150 ms**; bands nest at most **3** levels; `enterMoves` default `down`; `walk` default `rows`; `editMode` default `cell`; summary `position` default `bottom`; a span's `rows` and `columns` default **1**; a block row's key is `"/"`. +- A neighbouring spec takes its part only in the task whose step its row names (editing.md "Neighbouring specs"), and says nothing of it before. +- The read-only rule is one function. No code path decides read-only, the read-only surface or a place in the walk except through `ui::cellReadOnly` (Invariant 4). + +**Commands** (verified against `tests/CMakeLists.txt`, `tests/qt_quick/CMakeLists.txt`, `tests/tui/CMakeLists.txt`, `CMakePresets.json`): + +```bash +cmake --preset cl-debug # once per tree +cmake --build --preset cl-debug --target morph_tests # test_ui_*.cpp, tests/forms/*.cpp +cmake --build --preset cl-debug --target morph_table_tests # tests/table/*.cpp +ctest --test-dir build/cl-debug -R "" --output-on-failure +# Qt Quick (offscreen, ctest sets the platform) and the terminal backend, one tree: +cmake --preset cl-qt-debug -DMORPH_BUILD_QT_QUICK=ON -DMORPH_BUILD_TUI=ON +cmake --build --preset cl-qt-debug --target morph_qt_quick_tests morph_tui_tests +ctest --test-dir build/cl-qt-debug -R "" --output-on-failure +# morph_net_tests is POSIX-only: build and run it on linux-everything +cmake --preset linux-everything && cmake --build --preset linux-everything --target morph_net_tests +ctest --test-dir build/linux-everything -R "" --output-on-failure +``` + +## Interfaces consumed from Parts A–F + +Names the roadmap pins are used as it gives them. Rows marked **assumed** are not pinned by the roadmap; this part uses exactly these names, and the executor checks them against the landed code before Task 19 starts and corrects this plan (not the other part) if they differ. + +| From | Name | Pinned by | +|---|---|---| +| Task 1 | `table::TableColumn{ColumnInfo info; std::string label, group; bool sortable, filterable, shown;}`, `table::TableDescription{id, label, columns, key, groups, levels, units, revision}`, `table::describe` | roadmap | +| Task 2 | `table::ChangedRow`, `ChangedRows`, `BRIDGE_REGISTER_TABLE`, `effects::updatesTable`, the record `.` | roadmap | +| Task 3 | `table::Unloaded`, `TableErrorCode::SourceContract`, `CellEdits::commit` refusing an unloaded cell with `FieldError{"…", "not loaded yet"}` | roadmap, table.md | +| Task 8d | `table::ColumnLayout::{setPinned, pinned, value, apply}` with `value()`/`apply` over `table::ColumnLayoutValue{order, hidden, pinned, widths, groups, units}`; existing `setVisible`, `visible`, `move`, `order` | index #7, Part C | +| Task 9 | `ui::TableRows`, `EngineRows`, `WindowRows`, `TableRowRef`, `TableCountsView`, `SortChain`; `WindowRows::setDescription(table::TableDescription)` and `onStaleDescription(std::function)` (the lowering refetches the description and calls `setDescription`) | roadmap, Part D | +| Task 9 | `EngineRows` constructible over any `table::DataSource&` | **assumed** (Part D does not show its constructor) | +| Task 9 | `ui::TableStatus{counts, state, retry, message, shown}`, `Table::status`; `ui::TableColumn::id`; `ui::TableArrangement{visible, pinned, widths}`, `Table::arrangement` | index #11, #14, Part D | +| Task 10a | Mount-held focus `(row key, column id)`; `TableWidget::setFocusCell(std::size_t viewRow, std::size_t column)`, `setOnFocusMove(std::function)`, `enum class FocusMove { Up, Down, Left, Right, Home, End, First, Last, PageUp, PageDown }` | Part D | +| Task 11 | `ui::ValueRowsSource` | roadmap | +| Task 12b | `IViewBackend::copyText(std::string_view) -> bool` | index #12, Part E | +| Task 12c | `void ScreenCore::tableMessage(ElementState& table, std::string text)`; every grid and model-table message is posted through it | index #11, Part E | +| Task 15 | `ui::formatCell(table::Cell const&, table::TableColumn const&, ui::FormatContext const&, DisplayUnits const& = {}) -> std::string`; `ui::FormatContext` (`ui/expression.hpp`), `ui::localeFacts`, `ui::withoutGrouping`; `TableColumn::{display, tone, tooltip}`; `TableDescription::{rowTone, rowLock}` | index #9, Part F | +| Task 17a | `forms::DeclaresSchemaValue` (a type's `schemaValue()` returning `{keyword, name}`, written by `annotateSchemaMarker`), `name` the `RecordInfo::name` in `model::recordRegistry()` whose `type` is `typeid(TableOf)`; `TableInputKind`, `TableInput{member, record, kind}`, `tableInputsOf(std::string_view schemaJson)`, `RunObserver` | Part F | +| Task 17c | `ChangedRow::from`, `FailedRow`, `ChangedRows::failed`; `ui::RowStates` via `TableRows::states()` (`isBusy(key)`, `failure(key)`, `busyCount()`), `TableRows::moveKey(from, to)`, `ui::lockText(TableRowRef const&, TableDescription const&)`, `ui::refusesEdit(TableRows const&, std::int64_t index, TableDescription const&)` | index #10, Part F | + +## File structure + +| File | Responsibility | Task | +|---|---|---| +| `include/morph/ui/view.hpp` | `KeyOutcome`; `Table::onKey`, `onText`; `RowSpan::columns`; `TableRowView::summary`; `CellMarks`; `BandRun`; `Table::bands` | 19, 22 | +| `include/morph/ui/mount.hpp` | The focus chain, `Mounted::focusNext/focusPrevious`; cell marks; `bandRuns`, `columnUnitShown`, `bandedHeaderName`; `cutSpans`; summary rows | 19, 22 | +| `include/morph/ui/backend.hpp` | `TableWidget::setOnText`, `setCellMarks`, `setRowModified`, `setBands`, `setRowSummary` (all with no-op defaults); the Tab contract | 19, 22 | +| `include/morph/ui/form_rows.hpp` (new) | `FormRowsSource` | 19 | +| `include/morph/ui/grid.hpp` (new) | `GridColumn`, `GridOptions`, `CellFacts`, `cellReadOnly`, `GridSource`, `FormGridSource`, `GridRows`, `GridController`, `PasteResult`, `pasteMessage`, `FixedGridSource`, `ModelGridSource`, `summaryScopeRows` | 19–24 | +| `include/morph/forms/derive.hpp` (new) | `DeriveOf`, `Derived`, `derived<>()`, `clearDerived`, `DeriveOptions`, `DeriveRound`, `DeriveLoad`, `DeriveState` | 20 | +| `include/morph/forms/forms.hpp` | `FieldMeta::derived`, `withDerived()`; `x-derived`; row types' `computedFields` in the schema and in `recomputeAll` | 20 | +| `include/morph/forms/engine/field_model.hpp` | `FieldSpec::derived`, `minItems`, `maxItems` | 20, 21 | +| `include/morph/forms/engine/form_session.hpp` | Per-cell snapshots, `modified`, `rowModified`, `revert`; `field()` by row key; rounds; `insertRow`, `removeRows`, `restoreRows`; row-scoped and shared Choice options | 19–21 | +| `include/morph/core/registry.hpp` | `clearDerived` in `prepareAction` and `evaluateAction` | 20 | +| `include/morph/table/description.hpp` | `BandInfo`, `TableDescription::bands`, `checkBands`; `TableColumn::edit` | 22, 24 | +| `include/morph/table/edits.hpp` | `CellEdit`, `RowEdit`, `EditInfo`, `checkEdits`; `CellEdits` row commits, drafts by key, `moveRow`, `keepColumns` | 24 | +| `src/ui/document.cpp`, `src/ui/interpreter.cpp`, `include/morph/ui/interpreter.hpp` | Vocabulary, refusals, lowering, element values | 19–24 | +| `src/qt_quick/table.cpp` + QML, `src/tui/table_widget.cpp` | Editors in cells, marks, bands, spans, summary rows | 19, 22 | +| `tests/test_ui_grid.cpp` (new), `tests/forms/test_forms_derive.cpp` (new), `tests/grid_test_support.hpp` (new) | Grid and derive tests and their fixture | 19–23 | + +## Review Focus + +1. **An empty grid** (roadmap Review Focus 3). A grid over an `ObjectArray` with no rows: `Tab` leaves the table at once; a paste with `appendRows` adds the block's rows up to `maxItems`, without it drops every value and the status line counts them; a summary shows `count` 0 and an empty `mean`. Tests: Task 19 `"grid: an empty grid lets Tab leave at once"`, Task 21 `"grid: paste into an empty grid"`, Task 22 `"summary: over no rows"`. +2. **Bands under a stored layout** (roadmap Review Focus 5). A stored layout hides every column of a band, then one is shown again: the band reappears over that one column with its unit; a band whose columns are all pinned draws only in the pinned part. Test: Task 22 `"bands: a stored layout hides then shows a band's column"`. +3. **Clipboard text from a spreadsheet.** A spreadsheet copies `"1,5\t2\r\n3\t4\r\n"`: CRLF line ends, a trailing line break, values padded with spaces, grouped numbers (`"1.234,5"` in a comma locale). The block is 2 × 2, no value holds `\r`, no empty third row is pasted or appended, `" 12,5 "` is 12.5 and `"1.234,5"` is 1234.5. Test: Task 21 `"grid: spreadsheet clipboard text pastes as typed values"`. +4. **A row fetch that arrives before the edit's reply.** The action's change event can reach the client before its reply (table.md "Rows an action changed"), so the window refetches the edited row while the cell is still stale. The cell stays stale with the committed text until the reply, and is modified from the reply until the next fetch after it. Test: Task 24 `"cell edits: a fetch before the reply keeps the cell stale"`. +5. **A description revision that removes an edited column** (roadmap Review Focus 1, editing half). A model table holds a draft and an error in column `count` when the description's new revision drops `count`: the draft and its error are gone, `errorCount` falls by one, and nothing is committed for it. Test: Task 24 `"cell edits: a new revision drops drafts of a removed column"`. + +--- + +## Phase B — editing (`editing.md`) + +### Task 19: The focus chain and a grid of a form (editing.md step 1) + +**Spec:** editing.md "A grid of a form" (Binding, Columns are fields, Submission), "The cell" (all), "Walking", Invariants 1, 2, 4, 5; Delivery order step 1. Needs Tasks 9, 11, 15. + +**Files:** +- Modify: `include/morph/ui/view.hpp`, `include/morph/ui/backend.hpp`, `include/morph/ui/mount.hpp`, `include/morph/ui/testing/recording_backend.hpp` +- Create: `include/morph/ui/form_rows.hpp`, `include/morph/ui/grid.hpp`, `tests/grid_test_support.hpp`, `tests/test_ui_grid.cpp` (add to `morph_tests` in `tests/CMakeLists.txt`) +- Modify: `include/morph/forms/engine/form_session.hpp`, `include/morph/forms/engine/form_view.hpp` (an `ObjectArray` whose widget is `grid` draws `ui::formGrid`), `src/ui/document.cpp`, `src/ui/interpreter.cpp`, `include/morph/ui/interpreter.hpp`, `tests/ui_interpreter_support.hpp` (catalog schema `Lims_RecordSpecimens`) +- Modify: `src/qt_quick/table.cpp` and its QML, `src/tui/table_widget.cpp` (cell editors, `setOnText`, marks); `tests/qt_quick/test_qt_quick_table.cpp` +- Docs: `docs/spec/ui/view_tree.md`, `docs/spec/ui/backend_contract.md` (focus chain, `focusNext`/`focusPrevious`, `setOnText`, Tab delivered to a widget that declares it), `docs/spec/forms/engine.md` (the grid widget leaves "Not supported"; per-cell snapshots, `modified`, `revert`), `docs/spec/ui/document.md`, `docs/spec/ui/interpreter.md` (`form`/`field` on `table`, the `grid` overlay, `enterMoves`, `walk`, element values), `docs/spec/qt_quick/backend.md` (editors inside cells, cell and row marks), `docs/spec/table/table.md` (check it points to editing.md and lists `TableColumn::edit`, `TableDescription::bands`; add what is missing). + +**Interfaces:** +- Consumes: `ui::TableRows`, `EngineRows` (Task 9); `ValueRowsSource` (Task 11); `ui::formatCell` (Task 15); Task 10a's mount-held focus, `TableWidget::setFocusCell(viewRow, column)` and `setOnFocusMove(FocusMove)`; `ui::TableColumn::id`, `Table::arrangement` and `TableStatus::message` (Task 9); `ScreenCore::tableMessage` (Task 12c); `forms::FormSession`, `FieldState`, `RowState`, `forms::detail::engine::fieldControl` (existing). +- Produces: + - `view.hpp`: `struct CellRef { Key key; std::string column; bool operator==(CellRef const&) const = default; };` `Table::focus` (`std::optional>>`: when set, the table's focus is the grid's, and the mount shows it with Task 10a's `setFocusCell(viewRow, place in the layout)`) and `Table::onFocusCell` (`std::function`: a click or a `FocusMove` the mount resolves to a cell, handed to the grid instead of moving the mount's own focus); `enum class KeyOutcome : std::uint8_t { Handled, Ignored, FocusNext, FocusPrevious };` `Table::onKey` (`std::function`), `Table::keyChords` (`std::vector`), `Table::onText` (`std::function`); `struct CellMarks { bool readOnly = false; bool modified = false; bool error = false; bool stale = false; bool muted = false; bool operator==(CellMarks const&) const = default; };` `TableRowView::marks` (`std::optional>>`, one per column) and `TableRowView::modified` (`std::optional>`). + - `backend.hpp`: `virtual void TableWidget::setOnText(std::function onText) {}`; `virtual void TableWidget::setCellMarks(Widget& row, std::size_t column, CellMarks const& marks) {}`; `virtual void TableWidget::setRowModified(Widget& row, bool modified) {}`. Contract: a widget whose `setKeys` lists `Tab` or `Shift+Tab` receives them instead of moving focus. + - `mount.hpp`: `bool Mounted::focusNext(std::string_view focusId)`, `bool Mounted::focusPrevious(std::string_view focusId)`; the mount answers a table's `KeyOutcome::FocusNext`/`FocusPrevious` by moving focus along the chain from the table's widget. The chain is the mounted widgets that take focus — `TextInput`, `Button`, `Checkbox`, `Select`, `Menu`, `DateTimeInput`, `Slider`, `FilePicker`, `ActionButton`, `Table`, `Tabs`, a `Badge` with `onRemove` — that are visible and enabled, in document order (a walk of the mounted tree in child order, computed when asked). + - `form_rows.hpp`: `class FormRowsSource final : public table::DataSource` — `FormRowsSource(forms::FormSession& form, std::string arrayPath, std::vector columns)`; `columns()`, `snapshot()`, `subscribe`, `unsubscribe`; `[[nodiscard]] forms::RowState* row(table::RowId const& key) const`. A row's id is `RowId{static_cast(RowState::key())}`; rows in the array's order. A cell is the member's encoding as a cell of the column's kind (`Integer` → `int64`, `Decimal`/`Quantity` → `Rational`, `Number` → `double`, `Text`/`Enum`/`Choice` → `std::string`, `Date` → days, `DateTime` → seconds, `Bool` → `bool`), `std::monostate` when blank, and, when the draft does not encode, `std::string{draft}` — an invalid cell in every column but `Text`, where it is the text itself. A draft change of one member notifies `Updated` for its row alone. + - `grid.hpp`: + - `enum class EnterMoves : std::uint8_t { Down, Right, None };` `enum class Walk : std::uint8_t { Rows, Columns };` + - `struct GridColumn { std::string id; std::string label; std::string field; std::function value; };` — `field` empty with `value` set is a display column. + - `struct GridOptions { EnterMoves enterMoves = EnterMoves::Down; Walk walk = Walk::Rows; };` + - `struct CellFacts { forms::FieldState const* field = nullptr; bool hasEdit = true; bool overlayReadOnly = false; bool companionReadOnly = false; bool busy = false; bool locked = false; bool unloaded = false; };` and `[[nodiscard]] bool cellReadOnly(CellFacts const& facts);` — true when `field` is null or `!hasEdit`, the field's `readonly()` (which covers `x-readonly`, `x-computed`, `x-derived` and a `readonlyWhen` reaching it), or any other member is true. + - `class GridSource` (pure virtual): `rows() const -> std::vector` (view order); `columns() const -> std::vector` (visible, layout order); `readOnly(Key const&, std::string_view column) const -> bool` (calls `cellReadOnly`); `covered(Key const&, std::string_view) const -> bool`; `draft(Key const&, std::string_view) const -> std::string`; `text(Key const&, std::string_view) const -> std::string` (display text, `formatCell`); `error(Key const&, std::string_view) const -> std::optional`; `write(Key const&, std::string_view column, std::string draft) -> void` (a user's draft); `committed() -> void` (an edit is meant; one call per change); `marks(Key const&, std::string_view) const -> CellMarks`. + - `class FormGridSource final : public GridSource` — `FormGridSource(forms::FormSession& form, std::string arrayPath, std::vector columns, FormatContext const& format)` (`text()` is the draft while blank or not encoding, else `formatCell` of the cell with the column's display places set to the field's places); `[[nodiscard]] TableRows& view()`; `void setLayout(std::vector visible)` (the visible column ids in layout order, which the lowering feeds from Task 9's `Table::arrangement`; default every column in declared order); `[[nodiscard]] forms::FieldState* field(Key const& row, std::string_view column) const`; `modifiedCount() const -> std::size_t`, `modifiedRows() const -> std::size_t`, `errorCount() const -> std::size_t` (all tracked). + - `class GridRows final : public TableRows` over an `EngineRows` of a `FormRowsSource`: the engine's view, less the rows added since the sort or the filter last changed, which follow it in the order they were added (they pass the filter until it changes). + - `class GridController` — `GridController(reactive::Runtime& runtime, GridSource& source, GridOptions options)`; `[[nodiscard]] std::optional focus() const` (tracked); `void focusCell(CellRef cell)` (a click or an arrow; refused for a covered cell); `[[nodiscard]] std::optional editing() const` (tracked); `KeyOutcome press(std::string_view chord)`; `void type(std::string_view text)` (on a focused cell with no editor: opens the editor with `text` replacing the value; with the editor open: the editor's whole new text, as a `TextInput` edit reports it); `void leave()` (focus left the table: commits); `[[nodiscard]] std::string const& message() const` (tracked; the latest refusal or notice, which the lowering posts with `ScreenCore::tableMessage`); `[[nodiscard]] Node node(TableOptions options)` (the table node, wiring `onKey`, `onText`, focus and marks). + - `ui::formGrid(forms::FormSession&, forms::FieldState& array, std::vector, GridOptions) -> Node` (used by `form_view.hpp` and by the lowering). + - `form_session.hpp`: `FormSession::field(path)` also takes `rows#.member` (a row by key); `[[nodiscard]] bool modified(std::string_view path) const` (tracked: reads the field's draft and unit; true when they differ from what the field held when the form was last loaded, or the field's row was added since; false for a read-only member); `[[nodiscard]] bool rowModified(std::string_view arrayPath, std::uint64_t key) const`; `void revert(std::string_view path)` (programmatic: restores the loaded draft and unit; a row path reverts each member; reverting a row added since the load removes it). The load record is `std::map>` by `FieldState::path()`, `struct LoadedCell { std::string text; std::size_t unit; }`, written for every member of every `ObjectArray` row by `prefill`, `reset`, construction, a field `refill` loads and a field `acceptSent` ends the edit of. + - Document: `table` takes `form` (a form node id) and `field` (a path) in place of `rows`; a form field overlay takes `"widget": "grid"` with `grid`, a part holding a table node's keys; `enterMoves` (`down`/`right`/`none`), `walk` (`rows`/`columns`); element values `modifiedCount`, `modifiedRows`, `errorCount`, `editing` (`{key, column}` or `null`). Loader refusals: `form` with `rows`; a column `field` naming no member of the row type ("Refused when the document loads"). + - Status-line copy (posted through `ScreenCore::tableMessage`): typing on a read-only cell: `"This cell is read-only"`. + +- [ ] **Step 1: Write the fixture** `tests/grid_test_support.hpp` — namespace `morph::test::grid`: + +```cpp +// A form whose `specimens` rows are entered in a grid. `density` is computed, `limit` derived (Task 20 adds the +// `x-derived` reading; until then its `x-readonly` makes it read-only). `reviewer` freezes the grid by rule. +inline constexpr std::string_view kSpecimens = R"({"type":"object","title":"Specimens","properties":{ + "sampleId":{"type":"integer","title":"Sample","x-order":0}, + "reviewer":{"type":["string","null"],"title":"Reviewer","x-order":1}, + "specimens":{"type":"array","title":"Specimens","x-order":2,"minItems":0,"maxItems":5, + "items":{"type":"object","properties":{ + "label":{"type":["string","null"],"title":"Label","x-order":0}, + "mass":{"type":["object","null"],"title":"Mass","x-decimalPlaces":2,"x-order":1}, + "volume":{"type":["object","null"],"title":"Volume","x-decimalPlaces":1,"x-order":2}, + "density":{"type":["object","null"],"title":"Density","x-decimalPlaces":3,"x-readonly":true, + "x-computed":{"inputs":["mass","volume"]},"x-order":3}, + "limit":{"type":["object","null"],"title":"Limit","x-decimalPlaces":2,"x-readonly":true,"x-derived":true, + "x-order":4}, + "note":{"type":["string","null"],"title":"Note","x-order":5}}}}}, + "required":["sampleId"], + "x-rules":[{"kind":"readonlyWhen","fields":["specimens"],"when":{"kind":"engaged","fields":["reviewer"]}}]})"; + +struct GridFixture { + morph::test::formsengine::Harness h; + std::unique_ptr form; + std::unique_ptr source; + std::unique_ptr grid; + // Loads `rows` blank rows (keys 1..rows) with sampleId 1, in a comma locale, columns in schema order. + explicit GridFixture(std::size_t rows, ui::GridOptions options = {}, std::string schema = std::string{kSpecimens}); + [[nodiscard]] std::string json(std::string_view path) const; // the field's encoded JSON, "" when none +}; +inline ui::CellRef at(std::int64_t key, std::string column) { return ui::CellRef{ui::Key{key}, std::move(column)}; } +``` + +`GridFixture` prefills `{"sampleId":1,"specimens":[{},…]}`, builds the session with `FormSessionOptions{.locale = {.decimalSeparator = ",", .groupSeparator = "."}}` and the source with a `ui::FormatContext` of `ui::withoutGrouping(ui::localeFacts("de"))`, so a cell of `1` at two places shows `1,00`. Add the same schema to `corpusCatalog().schema` in `tests/ui_interpreter_support.hpp` as `Lims_RecordSpecimens`. + +- [ ] **Step 2: Write the failing tests** in `tests/test_ui_grid.cpp`: + +```cpp +// Mutation: FormGridSource::write parsing the number itself (std::stod after replacing ','), not through the field. +TEST_CASE("grid: a cell encodes as its field does", "[ui][grid]") { + GridFixture f{3}; + f.grid->focusCell(at(1, "mass")); + f.grid->type("12,5"); + CHECK(f.grid->press("Enter") == ui::KeyOutcome::Handled); + CHECK(f.json("specimens[0].mass") == R"({"num":1250,"den":100,"dp":2})"); + CHECK(f.grid->focus() == at(2, "mass")); // enterMoves: down + f.grid->type("1,234"); + f.grid->press("Enter"); + CHECK(f.form->field("specimens[1].mass").draft() == "1,234"); // kept as typed + CHECK(f.form->field("specimens[1].mass").error().peek()->code == "too-precise"); + CHECK_FALSE(f.form->ready()); + CHECK(f.source->marks(ui::Key{std::int64_t{2}}, "mass").error); + f.grid->focusCell(at(2, "mass")); + CHECK(f.grid->message() == "Too many decimal places"); // the forms error, while the cell has focus +} + +// Mutation: GridSource::readOnly testing spec().readOnly only (the readonlyWhen section fails). +TEST_CASE("grid: read-only is one rule", "[ui][grid]") { + GridFixture f{2}; + SECTION("computed and derived members") { + f.grid->focusCell(at(1, "volume")); + f.grid->type("2,0"); + f.grid->press("Tab"); + CHECK(f.grid->focus() == at(1, "note")); // density and limit skipped + f.grid->press("Shift+Tab"); + CHECK(f.grid->focus() == at(1, "volume")); + for (std::string const column : {"density", "limit"}) { + f.grid->focusCell(at(1, column)); // an arrow or a click reaches it + f.grid->type("5"); + CHECK_FALSE(f.grid->editing().has_value()); + CHECK(f.grid->message() == "This cell is read-only"); + CHECK(f.source->marks(ui::Key{std::int64_t{1}}, column).readOnly); + } + } + SECTION("a readonlyWhen rule") { + f.h.type(*f.form, "reviewer", "ana"); + f.grid->focusCell(at(1, "mass")); + f.grid->type("5"); + CHECK_FALSE(f.grid->editing().has_value()); + CHECK(f.source->marks(ui::Key{std::int64_t{1}}, "mass").readOnly); + CHECK(f.grid->press("Tab") == ui::KeyOutcome::FocusNext); // no editable cell is left + } +} + +// Mutation: Tab wrapping to the first editable cell, which never leaves the grid. +TEST_CASE("grid: the walk", "[ui][grid]") { + SECTION("enterMoves down fills a column") { + GridFixture f{3}; + f.grid->focusCell(at(1, "mass")); + for (auto const* text : {"1", "2", "3"}) { f.grid->type(text); f.grid->press("Enter"); } + CHECK(f.json("specimens[2].mass") == R"({"num":300,"den":100,"dp":2})"); + CHECK(f.grid->focus() == at(3, "mass")); // no row below and no appendRows: it stays + } + SECTION("Tab on the last editable cell leaves") { + GridFixture f{2}; + f.grid->focusCell(at(2, "note")); + CHECK(f.grid->press("Tab") == ui::KeyOutcome::FocusNext); + f.grid->focusCell(at(1, "label")); + CHECK(f.grid->press("Shift+Tab") == ui::KeyOutcome::FocusPrevious); + } + SECTION("walk: columns goes down first") { + GridFixture f{2, ui::GridOptions{.walk = ui::Walk::Columns}}; + f.grid->focusCell(at(1, "mass")); + f.grid->press("Tab"); + CHECK(f.grid->focus() == at(2, "mass")); + f.grid->press("Tab"); + CHECK(f.grid->focus() == at(1, "volume")); + } + SECTION("Escape restores the draft the cell had when the editor opened") { + GridFixture f{1}; + f.grid->focusCell(at(1, "label")); + f.grid->type("a"); + f.grid->press("Enter"); + f.grid->focusCell(at(1, "label")); + f.grid->type("b"); + f.grid->press("Escape"); + CHECK(f.form->field("specimens[0].label").draft() == "a"); + } +} + +// Review Focus 1, walk half. Mutation: an empty grid answering Tab with Handled. +TEST_CASE("grid: an empty grid lets Tab leave at once", "[ui][grid]") { + GridFixture f{0}; + CHECK_FALSE(f.grid->focus().has_value()); + CHECK(f.grid->press("Tab") == ui::KeyOutcome::FocusNext); + CHECK(f.grid->press("Shift+Tab") == ui::KeyOutcome::FocusPrevious); +} + +// Mutation: FormRowsSource giving the engine's view order to the array (a source that reorders the field's rows). +TEST_CASE("grid: a sorted grid keeps its record", "[ui][grid]") { + GridFixture f{3}; + for (auto const& [key, text] : {std::pair{1, "3"}, {2, "1"}, {3, "2"}}) { + f.grid->focusCell(at(key, "mass")); f.grid->type(text); f.grid->press("Enter"); + } + REQUIRE(f.source->view().setSort({table::SortKey{.column = "mass"}}).has_value()); + CHECK(f.source->rows() == std::vector{ui::Key{std::int64_t{2}}, ui::Key{std::int64_t{3}}, ui::Key{std::int64_t{1}}}); + f.grid->focusCell(at(3, "mass")); f.grid->type("9"); f.grid->press("Enter"); + CHECK(f.json("specimens[2].mass") == R"({"num":900,"den":100,"dp":2})"); // in place, entered order + CHECK(f.json("specimens[0].mass") == R"({"num":300,"den":100,"dp":2})"); + std::uint64_t const added = f.form->field("specimens").addRow(); + f.grid->focusCell(at(static_cast(added), "mass")); f.grid->type("0"); f.grid->press("Enter"); + CHECK(f.source->rows().back() == ui::Key{static_cast(added)}); // at the end until the sort changes + REQUIRE(f.source->view().setSort({}).has_value()); + REQUIRE(f.source->view().setSort({table::SortKey{.column = "mass"}}).has_value()); + CHECK(f.source->rows().front() == ui::Key{static_cast(added)}); // 0,00 sorts first +} + +// Mutation: modified() comparing the whole ObjectArray's snapshot, which marks every cell. +TEST_CASE("grid: what changed", "[ui][grid]") { + GridFixture f{2}; + f.grid->focusCell(at(1, "mass")); f.grid->type("4"); f.grid->press("Enter"); + CHECK(f.form->modified("specimens#1.mass")); + CHECK_FALSE(f.form->modified("specimens#1.volume")); + CHECK_FALSE(f.form->modified("specimens#2.mass")); + CHECK(f.form->rowModified("specimens", 1)); + CHECK_FALSE(f.form->rowModified("specimens", 2)); + CHECK(f.source->modifiedCount() == 1); + CHECK(f.source->modifiedRows() == 1); + f.form->revert("specimens#1.mass"); + CHECK(f.form->field("specimens[0].mass").draft().empty()); + CHECK(f.source->modifiedCount() == 0); + std::uint64_t const added = f.form->field("specimens").addRow(); + CHECK(f.form->rowModified("specimens", added)); // a row added since the load, as a whole +} +``` + +Interpreter-level, same file, through `ui::testing::Harness` and a document holding `{"kind":"form","id":"rec","model":"sample","action":"Lims_RecordSpecimens","submit":"explicit"}`, a text input `before`, `{"kind":"table","id":"grid","testId":"grid","form":"rec","field":"specimens","columns":[{"id":"mass","field":"mass"},{"id":"note","field":"note"}]}` and a text input `after`: + +```cpp +// Mutation: lowering the form's submit before the open editor commits. +TEST_CASE("grid: submission commits the open editor and places field errors", "[ui][grid][interpreter]") { + // (the Harness formats in `en`) prefill sampleId 4 and two blank specimens; focus (1, mass), type "7.5" with the + // editor open; click form_submit + // CHECK the submitted body's specimens[0].mass == {"num":750,"den":100,"dp":2} + // fail the call with field error {"specimens[1].mass": "must be positive"} + // CHECK readJson(*screen, R"({"ref":"grid.errorCount"})") == "1" and the cell (2, mass) tooltip == "must be positive" +} +// Mutation: the mount ignoring KeyOutcome::FocusNext. +TEST_CASE("grid: Tab on the last editable cell moves focus to the next control", "[ui][grid][interpreter]") { + // focus (2, note); h.backend.press(h.byTestId("Table","grid"), "Tab") + // CHECK h.backend.focused() == h.byTestId("TextInput","after") + // press "Shift+Tab" on (1, mass): CHECK focused() == byTestId("TextInput","before") +} +TEST_CASE("grid: a column naming no member is refused when the document loads", "[ui][grid][interpreter]") { + // a column {"id":"x","field":"nosuch"}: loadDocumentText fails; the message names "nosuch" +} +``` + +These three are written out in full Catch2 with the Harness calls named in the comments; the comments are the assertions. + +Qt Quick, `tests/qt_quick/test_qt_quick_table.cpp`: + +```cpp +// Mutation: the QML cell ignoring setOnText. +TEST_CASE("qt_quick grid: typing on a focused cell opens its editor, Tab walks and leaves", "[qt_quick][grid]") { + // A mounted formGrid of two rows between two text inputs: QTest::keyClicks "5" on (1, mass) shows a TextField + // holding "5" in that cell; Tab commits it (field draft "5") and focuses (1, volume); Tab on (2, note) gives + // focus to the second text input's item. +} +``` + +- [ ] **Step 3: Run to see them fail** + +Run: `cmake --build --preset cl-debug --target morph_tests` +Expected: compile errors — `morph/ui/grid.hpp` not found, `FormSession::modified` not declared. + +- [ ] **Step 4: Implement the focus chain** in `view.hpp`, `backend.hpp`, `mount.hpp` with the signatures above; `RecordingBackend` records `setOnText` as a callback and gains `void typeInto(int tableId, std::string text)`; it records `setCellMarks` as property `marks:` (`r`,`m`,`e`,`s`,`u` letters for the set flags) and `setRowModified` as `modified`. + +- [ ] **Step 5: Implement per-cell snapshots** in `form_session.hpp`: `field()` by `#key`, `modified`, `rowModified`, `revert`, the load record. + +- [ ] **Step 6: Implement `form_rows.hpp` and `grid.hpp`** with the signatures above. The walk, which the signatures do not determine: + +```text +next(from, forward): // view positions at the moment of the call + cells = rows() × columns() in walk order (Rows: row-major; Columns: column-major) + step from `from`'s position (or before the first / after the last when there is no focus) + return the first cell with !readOnly && !covered, or none +Tab: commit an open editor; next(focus, true) -> focus it, Handled; none -> FocusNext +Shift+Tab: commit; next(focus, false) -> focus it, Handled; none -> FocusPrevious +Enter (editor open): commit; Down: first editable cell in the same column below; Right: next(focus, true) within + the row; None: stay. Nothing found: stay. Enter, F2 or a double click with no editor opens it on an + editable cell; on a read-only cell it runs the table's onActivate. +Space on a Boolean cell: toggles and commits, no editor. Delete with no editor: blanks the cell (write ""). +Arrows: move to any cell (read-only included, covered cells passed over); Ctrl+arrow to the last cell that way. +``` + +Commit writes the editor's text with `write` and calls `committed()` once. + +- [ ] **Step 7: Implement the lowering and the loader** in `src/ui/document.cpp`, `src/ui/interpreter.cpp`: `table` with `form`/`field` lowers to `formGrid` over the form element's session; the `grid` overlay adds `overrides.byField(id, …)` drawing the same; element values; refusals. `form_view.hpp` draws an `ObjectArray` whose `spec().widget == "grid"` with `ui::formGrid`. + +- [ ] **Step 8: Implement editors in cells** in the Qt Quick and terminal tables: `setOnText`, `setCellMarks`, `setRowModified`, `Tab`/`Shift+Tab` delivered to a table that lists them. + +- [ ] **Step 9: Run the tests** + +Run: `cmake --build --preset cl-debug --target morph_tests && ctest --test-dir build/cl-debug -R "grid:" --output-on-failure` +Expected: all `grid:` cases pass. Then `ctest --test-dir build/cl-debug -R "form|table|mount" --output-on-failure` — no regressions. Qt Quick: `ctest --test-dir build/cl-qt-debug -R "qt_quick grid" --output-on-failure` — pass. + +- [ ] **Step 10: Mutate** each named mutation once; record the failing lines. + +- [ ] **Step 11: Docs** per this task's Docs list; run the docs build (new public symbols). + +- [ ] **Step 12: Commit** + +```bash +git add include/morph/ui include/morph/forms/engine src/ui src/qt_quick src/tui tests docs/spec +git commit -m "feat(ui): grids of a form" +``` + +### Task 20: Derived values (editing.md step 2) + +**Spec:** editing.md "Derived values", "Columns are fields" (Per row), "The editor" (choices that depend on the row), Invariants 2, 3, 9; Failure modes rows "An evaluation or a derive call fails", "A reply arrives after a newer one", "A derive reply names a path that is not a derived member"; Delivery order step 2. + +**Files:** +- Create: `include/morph/forms/derive.hpp`, `tests/forms/test_forms_derive.cpp` (add to `morph_tests`) +- Modify: `include/morph/forms/forms.hpp`, `include/morph/forms/engine/field_model.hpp`, `include/morph/forms/engine/form_session.hpp`, `include/morph/core/registry.hpp`, `include/morph/ui/grid.hpp`, `src/ui/document.cpp`, `src/ui/interpreter.cpp`, `tests/test_ui_grid.cpp`, `tests/grid_test_support.hpp`, `tests/qt_quick/test_qt_quick_table.cpp` +- Docs: `docs/spec/forms/engine.md` (rounds, `loadDerived`, `deriveDebounce`, row-dependent Choices), `docs/spec/forms/forms.md` (`computedFields` on a row type as the exception to "top-level only"; `derived()` and `x-derived`, untouched by `recomputeAll`), `docs/spec/core/registry.md` (`DeriveOf`, `Derived`; `evaluate` recomputing row members; `clearDerived`), `docs/spec/ui/document.md` and `docs/spec/ui/interpreter.md` (`derive`, `deriveDebounce`; a column's `field` expression and per-row overlays). + +**Interfaces:** +- Consumes: Task 19's `GridColumn`, `FormGridSource`, `GridController`, `GridSource::committed`; `ActionDispatcher::evaluate`/`evaluateAction`/`prepareAction` (existing); `ModelChannel::execute(…, CallOptions{.evaluate = true})` (existing); `reactive::Scheduler::after` (existing). +- Produces: + - `derive.hpp`: `template struct DeriveOf { Action body; };` `struct Derived { struct Value { std::string path; glz::raw_json value; }; std::vector values; };` (JSON `{"values":[{"path":…,"value":…}]}`); `template [[nodiscard]] FieldMeta derived(std::string_view label = {}, std::string_view help = {}) noexcept` (= `describe(label, help).withDerived()`); `template void clearDerived(A& action)` (value-initialises every member whose `FieldMeta` is `derived`, in `A` and in each element of a `std::vector` member of a reflectable aggregate, recursively); `enum class DeriveState : std::uint8_t { Current, Running, Failed };` `struct DeriveOptions { std::function(std::string bodyJson)> evaluate; std::function(std::string deriveBodyJson)> derive; reactive::Scheduler* scheduler = nullptr; std::chrono::milliseconds debounce{150}; };` `struct DeriveRound { std::uint64_t sequence = 0; std::string body; std::map, std::less<>> rowKeys; };` `struct DeriveLoad { std::size_t loaded = 0; std::vector refused; };` + - `forms.hpp`: `bool FieldMeta::derived{false};` `constexpr FieldMeta withDerived() const noexcept`; the schema writes `"x-derived": true` and leaves the member out of `required`; `annotateNestedAggregate` writes `x-computed`/`x-readonly` for `Sub::computedFields`; `recomputeAll` recomputes each element of every `std::vector` member with `HasComputedFields` (recursively) **before** `A`'s own entries, so a top-level total reads recomputed rows. + - `field_model.hpp`: `bool FieldSpec::derived = false;` read from `x-derived`; `FieldState::readonly()` is true for a derived member. + - `registry.hpp`: `prepareAction` and `evaluateAction` call `forms::clearDerived(action)` before `recomputeAll`. + - `form_session.hpp`: `void deriveWith(DeriveOptions options)`; `void noteCommit()`; `void recalculate()`; `[[nodiscard]] DeriveState deriveState() const` (tracked); `[[nodiscard]] std::string partialBody() const` (every top-level field that encodes; an `Object` or `ObjectArray` member by member, leaving out blank and non-encoding members; every row written, in order, `{}` for a row with nothing); `[[nodiscard]] DeriveRound captureRound()` (next sequence, `partialBody()`, the row keys of every `ObjectArray` by `FieldState::path()`); `DeriveLoad loadDerived(DeriveRound const& round, Derived const& reply)` (derived members only); `DeriveLoad loadDerived(DeriveRound const& round, std::string_view evaluatedBody)` (computed members only, the reply's `errors` and `ready` ignored). Both resolve `name[i]` through `round.rowKeys[name][i]` and the row's current position by key, drop a row since removed, refuse a path that names no member of their kind, write programmatically and move no snapshot. `std::uint64_t FieldState::insertRow(std::size_t position)` (a user action; position past the end appends). + - Choices: a row member's `x-optionsDependsOn` names are looked up in its row, then among the top-level fields; a row member's options are fetched only after `void FieldState::openOptions()` (what a cell editor calls when it opens); identical `(optionsAction, request)` pairs share one fetch for the session's life, a failed fetch is not kept. + - `grid.hpp`: `GridColumn` gains `std::function fieldOf` (a member per row; tracked) and per-row overlays `decimals` (`std::function(forms::RowState const&)>`), `unit` (`std::function(…)>`), `readonly` (`std::function`), `options` (`std::function>(…)>`, option JSON literals kept), `placeholder` (`std::function`). A column with `fieldOf` has `ColumnKind::Text` in `FormRowsSource`, its cell the member's draft. Derived and computed cells draw `CellMarks::muted` while `deriveState()` is `Running` and `stale` while `Failed`; the status line then offers `"Recalculate"`. + - Document: `form` takes `derive` (an action id of the form's model) and `deriveDebounce` (a literal integer, milliseconds); a grid column's `field` may be an expression in the row's scope, and takes `decimals`, `unit`, `readonly`, `options`, `placeholder` as expressions in the row's scope. The lowering calls `deriveWith` with `evaluate` = the form alias's channel `execute(action, body, {.evaluate = true})` and `derive` = `execute(derive, R"({"body":)" + body + "}")`, and reports each refused path once. + +- [ ] **Step 1: Write the failing server tests** in `tests/forms/test_forms_derive.cpp`: + +```cpp +struct DeriveBox { + std::int64_t count = 0; + std::int64_t perBox = 0; + std::int64_t total = 0; // computed: count * perBox + std::int64_t limit = 0; // derived: the model's + static constexpr auto computedFields = morph::forms::computeList( + morph::forms::computed<&DeriveBox::total, &DeriveBox::count, &DeriveBox::perBox>( + [](auto const& box) { return box.count * box.perBox; })); + static const std::array fieldMetadata; +}; +inline const std::array DeriveBox::fieldMetadata{morph::forms::derived<&DeriveBox::limit>()}; + +struct RecordBoxes { + std::vector boxes; + std::int64_t sum = 0; // computed: the boxes' totals + static constexpr auto computedFields = morph::forms::computeList(morph::forms::computed<&RecordBoxes::sum, &RecordBoxes::boxes>( + [](auto const& record) { std::int64_t s = 0; for (auto const& b : record.boxes) s += b.total; return s; })); +}; +struct DeriveBoxModel { + static inline RecordBoxes last{}; + bool execute(RecordBoxes const& record) { last = record; return true; } // sets no derived member +}; +BRIDGE_REGISTER_MODEL(DeriveBoxModel, "T_DeriveBoxes") +BRIDGE_REGISTER_ACTION(DeriveBoxModel, RecordBoxes, "T_RecordBoxes") + +constexpr std::string_view kForged = R"({"boxes":[{"count":2,"perBox":3,"total":999,"limit":555}],"sum":999})"; + +// Mutation: recomputeAll skipping row members (total stays 999, sum 999); a second run with clearDerived removed +// from prepareAction (limit stays 555). +TEST_CASE("derive: the server keeps recomputeAll's values and the model's", "[forms][derive]") { + auto holder = morph::model::detail::defaultRegistry().create("T_DeriveBoxes"); + morph::model::detail::defaultDispatcher().dispatch("T_DeriveBoxes", "T_RecordBoxes", *holder, kForged); + RecordBoxes const& kept = DeriveBoxModel::last; + CHECK(kept.boxes.at(0).total == 6); + CHECK(kept.sum == 6); + CHECK(kept.boxes.at(0).limit == 0); // the client's 555 is cleared; the model set nothing + auto const evaluated = morph::model::detail::defaultDispatcher().evaluate("T_DeriveBoxes", "T_RecordBoxes", kForged); + CHECK(evaluated.body == R"({"boxes":[{"count":2,"perBox":3,"total":6,"limit":0}],"sum":6})"); +} + +// Mutation: annotateNestedAggregate not writing x-computed for a row type. +TEST_CASE("derive: a row type's schema marks computed and derived members", "[forms][derive]") { + auto dom = glz::read_json(morph::forms::schemaJson()).value(); + auto& box = dom["properties"]["boxes"]["items"]; // DeriveBox is used once: glaze inlines it + CHECK(glz::write_json(box["properties"]["total"]["x-readonly"]).value() == "true"); + CHECK(glz::write_json(box["properties"]["total"]["x-computed"]["inputs"]).value() == R"(["count","perBox"])"); + CHECK(glz::write_json(box["properties"]["limit"]["x-derived"]).value() == "true"); + CHECK(glz::write_json(box["required"]).value() == R"(["count","perBox"])"); +} +``` + +- [ ] **Step 2: Write the failing round tests** in `tests/test_ui_grid.cpp`. `GridFixture` gains `reactive::testing::ManualScheduler scheduler;` two `formsengine::FakeServer` recorders `evaluate` and `derive` whose responders return `std::nullopt` (every call stays pending until resolved by index), and `void enableDerive()`, which calls `form->deriveWith({.evaluate = …, .derive = …, .scheduler = &scheduler})` (debounce 150 ms) over them. + +```cpp +using namespace std::chrono_literals; +std::string const kNoValues = R"({"values":[]})"; + +// Mutation: loadDerived resolving `specimens[i]` against the current rows (the value lands on the inserted row). +TEST_CASE("grid: derived values", "[ui][grid][derive]") { + GridFixture f{3}; + f.enableDerive(); + f.grid->focusCell(at(1, "mass")); f.grid->type("4,00"); f.grid->press("Enter"); + f.grid->type("abc"); f.grid->press("Enter"); // (2, mass): does not encode, left out + f.scheduler.advance(149ms); f.h.settle(); + CHECK(f.evaluate.submits().empty()); + f.scheduler.advance(1ms); f.h.settle(); + REQUIRE(f.evaluate.submits().size() == 1); + REQUIRE(f.derive.submits().size() == 1); + CHECK(f.evaluate.submits().at(0).body == R"({"sampleId":1,"specimens":[{"mass":{"num":400,"den":100,"dp":2}},{},{}]})"); + CHECK(f.derive.submits().at(0).body == R"({"body":{"sampleId":1,"specimens":[{"mass":{"num":400,"den":100,"dp":2}},{},{}]}})"); + CHECK(f.form->deriveState() == forms::DeriveState::Running); + CHECK(f.source->marks(ui::Key{std::int64_t{1}}, "limit").muted); + + f.form->field("specimens").insertRow(1); // above rows 2 and 3 + f.grid->focusCell(at(2, "note")); f.grid->type("typed"); f.grid->press("Enter"); // a commit during the round + f.derive.resolveSubmit(0, R"({"values":[{"path":"specimens[1].limit","value":{"num":7,"den":1,"dp":2}}, + {"path":"specimens[1].note","value":"server"}]})"); + f.evaluate.resolveSubmit(0, R"({"sampleId":1,"specimens":[{"mass":{"num":400,"den":100,"dp":2}, + "density":{"num":2,"den":1,"dp":3}},{},{}],"errors":{},"ready":true})"); + f.h.settle(); + CHECK(f.json("specimens[2].limit") == R"({"num":700,"den":100,"dp":2})"); // row key 2, now at position 2 + CHECK(f.json("specimens[1].limit").empty()); // the inserted row is untouched + CHECK(f.json("specimens[0].density") == R"({"num":2000,"den":1000,"dp":3})"); + CHECK(f.form->field("specimens#2.note").draft() == "typed"); // not a derived member: refused + CHECK(f.form->modified("specimens#2.note")); + CHECK(f.evaluate.submits().size() == 2); // the stale round: one more, at once + CHECK(f.derive.submits().size() == 2); + f.evaluate.resolveSubmit(1, R"({"errors":{},"ready":false})"); + f.derive.resolveSubmit(1, kNoValues); + f.h.settle(); + CHECK(f.form->deriveState() == forms::DeriveState::Current); + + // A failed round keeps the values, marks them stale, and Recalculate runs one more. + f.grid->focusCell(at(1, "mass")); f.grid->type("5"); f.grid->press("Enter"); + f.scheduler.advance(150ms); f.h.settle(); + f.evaluate.rejectSubmit(2, "unreachable"); + f.derive.resolveSubmit(2, kNoValues); + f.h.settle(); + CHECK(f.form->deriveState() == forms::DeriveState::Failed); + CHECK(f.source->marks(ui::Key{std::int64_t{2}}, "limit").stale); + CHECK(f.json("specimens[2].limit") == R"({"num":700,"den":100,"dp":2})"); + CHECK(f.grid->message() == "Derived values could not be recalculated. Recalculate"); + f.form->recalculate(); f.h.settle(); + CHECK(f.evaluate.submits().size() == 4); +} + +// Mutation: loadDerived without the sequence check (the late older reply overwrites 9 with 8). +TEST_CASE("grid: an older derive reply is ignored", "[ui][grid][derive]") { + GridFixture f{2}; + auto const older = f.form->captureRound(); + auto const newer = f.form->captureRound(); + auto const value = [](std::string json) { return forms::Derived{.values = {{"specimens[1].limit", glz::raw_json{std::move(json)}}}}; }; + CHECK(f.form->loadDerived(newer, value(R"({"num":9,"den":1,"dp":2})")).loaded == 1); + CHECK(f.form->loadDerived(older, value(R"({"num":8,"den":1,"dp":2})")).loaded == 0); + CHECK(f.json("specimens[1].limit") == R"({"num":900,"den":100,"dp":2})"); + f.form->field("specimens").removeRow(2); + auto const gone = f.form->captureRound(); + CHECK(f.form->loadDerived(gone, value(R"({"num":1,"den":1,"dp":2})")).loaded == 0); // that row is gone +} + +// Mutation: a derive reply that moves the loaded snapshot of the members it writes. +TEST_CASE("grid: a derive reply moves no snapshot", "[ui][grid][derive]") { + GridFixture f{1}; + f.grid->focusCell(at(1, "mass")); f.grid->type("4"); f.grid->press("Enter"); + auto const round = f.form->captureRound(); + auto const load = f.form->loadDerived(round, forms::Derived{.values = {{"specimens[0].limit", glz::raw_json{R"({"num":1,"den":1,"dp":2})"}}, + {"specimens[0].mass", glz::raw_json{R"({"num":5,"den":1,"dp":2})"}}}}); + CHECK(load.loaded == 1); + CHECK(load.refused == std::vector{"specimens[0].mass"}); + CHECK(f.form->modified("specimens#1.mass")); + CHECK(f.form->field("specimens[0].mass").draft() == "4"); + CHECK_FALSE(f.form->modified("specimens#1.limit")); // a read-only member draws no modified mark +} + +// Mutation: the column's field resolved once for every row. +TEST_CASE("grid: kind per row", "[ui][grid][derive]") { + // schema: rows {isChoice: boolean, choice: enum ["low","high"], amount: decimal 1 place, mass: decimal 2 places}; + // column {id: "value", fieldOf: isChoice ? "choice" : "amount"}; column "mass" with decimals overlay row 1 -> 1, row 2 -> 3 + // rows: [{isChoice: true}, {isChoice: false}] + CHECK(f.source->field(ui::Key{std::int64_t{1}}, "value")->spec().kind == forms::FieldKind::Enum); + CHECK(f.source->field(ui::Key{std::int64_t{2}}, "value")->spec().kind == forms::FieldKind::Decimal); + f.grid->focusCell(at(2, "value")); f.grid->type("3,5"); f.grid->press("Enter"); + CHECK(f.json("rows[1].amount") == R"({"num":35,"den":10,"dp":1})"); + f.grid->focusCell(at(1, "mass")); f.grid->type("1,25"); f.grid->press("Enter"); + CHECK(f.form->field("rows[0].mass").error().peek()->code == "too-precise"); + f.grid->focusCell(at(2, "mass")); f.grid->type("1,255"); f.grid->press("Enter"); + CHECK(f.json("rows[1].mass") == R"({"num":1255,"den":1000,"dp":3})"); +} + +// Mutation: a fetch per row (no shared fetch). +TEST_CASE("grid: choices per row", "[ui][grid][derive]") { + // rows {material: string, grade: Choice "Lims_ListGrades", x-optionsDependsOn ["material","site"]}; top-level "site" + // prefill site "north", six rows with material A, A, B, B, C, A + CHECK(f.h.server.fetches().empty()); // nothing is fetched before an editor opens + for (std::int64_t key = 1; key <= 6; ++key) { f.grid->focusCell(at(key, "grade")); f.grid->press("F2"); f.grid->press("Escape"); } + REQUIRE(f.h.server.fetches().size() == 3); + CHECK(f.h.server.fetches().at(0).body == R"({"material":"A","site":"north"})"); + CHECK(f.h.server.fetches().at(1).body == R"({"material":"B","site":"north"})"); + CHECK(f.h.server.fetches().at(2).body == R"({"material":"C","site":"north"})"); +} +``` + +Qt Quick: `"qt_quick grid: a column's kind follows its row"` — row 1's `value` cell opens a ComboBox, row 2's a TextField. Mutation as "kind per row". + +- [ ] **Step 3: Run to see them fail** + +Run: `cmake --build --preset cl-debug --target morph_tests` +Expected: compile errors — `morph/forms/derive.hpp` not found; `FormSession::deriveWith` not declared. + +- [ ] **Step 4: Implement `derive.hpp`, `FieldMeta::derived`, the schema, `recomputeAll` over rows and `clearDerived` in `prepareAction`/`evaluateAction`.** + +- [ ] **Step 5: Implement rounds in `FormSession`.** The order the signatures do not determine: + +```text +noteCommit(): a round in flight: mark it stale. Else: no scheduler: startRound() now; else restart the timer + for options.debounce, whose expiry calls startRound() +startRound(): round = captureRound(); state = Running; send evaluate (if set) and derive (if set) with round.body +on a reply: loadDerived(round, reply). loadDerived itself ignores a round whose sequence is below the last one it + loaded for that kind of reply (evaluation or derive), and records the sequence it loads. +both settled: any failed: state = Failed, else Current; stale: clear it and startRound() at once +recalculate(): startRound() unless a round is in flight +``` + +- [ ] **Step 6: Implement row-scoped Choices** (`openOptions`, lookup order, the shared fetch) and the per-row column parts in `grid.hpp`; the lowering's `derive`, `deriveDebounce`, column expressions and overlays. + +- [ ] **Step 7: Run the tests** + +Run: `cmake --build --preset cl-debug --target morph_tests && ctest --test-dir build/cl-debug -R "derive|grid:|computed|evaluate|forms" --output-on-failure` +Expected: all pass, including the existing `test_computed_fields.cpp` and `test_ui_evaluate.cpp` cases. + +- [ ] **Step 8: Mutate**, **Step 9: Docs** (and the docs build), **Step 10: Commit** `feat(forms): derived values for grids`. + +### Task 21: Selecting cells, paste and rows of a grid (editing.md step 3) + +**Spec:** editing.md "Selecting cells, clipboard and fill", "Rows of a grid", Invariant 7; Failure modes rows on paste, insert and delete, `selectionMode: "cells"` on a model's table. + +**Files:** +- Modify: `include/morph/ui/grid.hpp`, `include/morph/forms/engine/field_model.hpp`, `include/morph/forms/engine/form_session.hpp`, `src/ui/document.cpp`, `src/ui/interpreter.cpp`, `src/qt_quick/table.cpp` and QML, `src/tui/table_widget.cpp`; `tests/test_ui_grid.cpp` +- Docs: `docs/spec/ui/document.md`, `docs/spec/ui/interpreter.md` (`selectionMode: "cells"` and its value, `appendRows`, `rowsFixed`, the row menu entries). + +**Interfaces:** +- Consumes: Task 19's `GridController`, `GridSource`; Task 20's `FieldState::insertRow`; Task 12b's `IViewBackend::copyText` (the lowering hands it `GridController::copy()`) and Task 12c's `ScreenCore::tableMessage`. +- Produces: + - `field_model.hpp`: `std::optional FieldSpec::minItems, maxItems;` read from an array's `minItems`/`maxItems`. + - `form_session.hpp`: `struct RemovedRow { std::size_t position; std::shared_ptr row; };` `std::vector FieldState::removeRows(std::span keys);` `void FieldState::restoreRows(std::vector rows);` (each back at its position, ascending). + - `grid.hpp`: `struct CellRange { CellRef anchor; CellRef focus; bool operator==(CellRange const&) const = default; };` `GridOptions` gains `bool appendRows = false; bool rowsFixed = false;` `GridController` gains `[[nodiscard]] std::optional selection() const` (tracked), `void select(std::optional range)`, `[[nodiscard]] std::string copy() const` (tab-separated display text of the rectangle, view order, layout order; a tab or line break inside a cell written as a space), `PasteResult paste(std::string_view text)`, `void fillDown()`, `void clear()`, `void insertRow(bool below)`, `void deleteRows()`, `void undo()`, `[[nodiscard]] RowOps rowOps() const` (tracked); `struct PasteResult { std::size_t written = 0; std::size_t readOnly = 0; std::size_t pastRows = 0; std::size_t pastColumns = 0; };` `[[nodiscard]] std::string pasteMessage(PasteResult const& result);` `struct RowOps { bool insert; bool remove; std::string why; };` `GridSource` gains `appendRow() -> std::optional`, `insertRow(std::size_t position) -> std::optional`, `removeRows(std::span) -> bool` (true when one held a value), `undoRemove() -> void`, `rowCount() -> std::size_t`, `bounds() -> std::pair>` (minItems, maxItems). + - Chords: `Shift`+arrows extend from the anchor; `Ctrl+A` the whole grid; `Ctrl+C` copy; `Ctrl+V` paste; `Ctrl+D` fill down; `Delete` clear; `Ctrl+Shift+=` insert below; `Ctrl+-` delete; a row header click selects its row, a column header's selection gesture its column. + - Copy, exact: paste message `"Pasted {written} values"`, then, when anything was dropped, `"; "` and the parts present joined by `" and "`: `"{readOnly} on read-only cells"`, `"{pastRows} past the last row"`, `"{pastColumns} past the last column"`, then `" were dropped"`. Deletion of rows holding values: `"Deleted {n} rows. Undo"` (`"Deleted 1 row. Undo"`), Undo until the next change. Disabled entries: at `maxItems` `"A grid holds at most {maxItems} rows"`, at `minItems` `"A grid holds at least {minItems} rows"`; with `rowsFixed` no entry is offered. + - Paste, which the signatures do not determine: lines split on `\n`, a trailing `\r` removed from each, a final empty line dropped; cells split on `\t`; each value trimmed of spaces. Start: the focused cell; in a grid with no rows, the first visible column of the first row the paste adds. Row `r0+i` past the last row: appended when `appendRows` and the count is below `maxItems`, else every value of that line counts in `pastRows`. Column `c0+j` past the last visible column counts in `pastColumns`. A read-only or covered target counts in `readOnly`. Every other value is written with `write` (the cell's draft, in the user's locale), all in one runtime batch, then `committed()` once. Fill and clear: editable cells of the rectangle only, one batch, one `committed()`. + - Document: `selectionMode: "cells"` (the default for a grid of a form; refused on a model's table), `selection`/`onSelectionChange` with `{"anchor": {"key", "column"}, "focus": {"key", "column"}}` or `null`; `appendRows`, `rowsFixed` (literal flags). + +- [ ] **Step 1: Write the failing tests** in `tests/test_ui_grid.cpp`: + +```cpp +// Mutation: CellRange stored as view positions (the sort moves the corners onto other rows). +TEST_CASE("grid: cell selection", "[ui][grid][cells]") { + GridFixture f{3}; + for (auto const& [key, text] : {std::pair{1, "3"}, {2, "1"}, {3, "2"}}) { + f.grid->focusCell(at(key, "mass")); f.grid->type(text); f.grid->press("Enter"); + } + f.grid->focusCell(at(1, "mass")); + f.grid->press("Shift+Down"); f.grid->press("Shift+Right"); + CHECK(f.grid->selection() == ui::CellRange{at(1, "mass"), at(2, "volume")}); + REQUIRE(f.source->view().setSort({table::SortKey{.column = "mass"}}).has_value()); + CHECK(f.grid->selection() == ui::CellRange{at(1, "mass"), at(2, "volume")}); // corners kept by key + CHECK(f.grid->copy() == "1,00\t\n2,00\t\n3,00\t"); // rows 2, 3, 1 now lie between the corners' positions +} +TEST_CASE("grid: cells on a model's table are refused when the document loads", "[ui][grid][cells]") { + // {"kind":"table","model":"m","table":"specimens","selectionMode":"cells"}: loadDocumentText fails, message names selectionMode +} + +// Mutation: a paste that shifts a value past a read-only cell into the next editable one. +TEST_CASE("grid: paste, fill, clear", "[ui][grid][cells]") { + std::string schema{kSpecimens}; + schema.replace(schema.find(R"("maxItems":5)"), 12, R"("maxItems":2)"); + GridFixture f{1, ui::GridOptions{.appendRows = true}, schema}; + f.enableDerive(); + f.grid->focusCell(at(1, "volume")); // columns: volume, density (read-only), limit (read-only), note + auto const result = f.grid->paste("1\t2\t3\ta\n4\t5\t6\tb\n7\t8\t9\tc"); // 3 rows × 4 columns + CHECK(result.written == 4); // volume and note of two rows + CHECK(result.readOnly == 4); + CHECK(result.pastRows == 4); + CHECK(f.grid->message() == "Pasted 4 values; 4 on read-only cells and 4 past the last row were dropped"); + CHECK(f.form->field("specimens[0].note").draft() == "a"); // the block keeps its shape + CHECK(f.form->field("specimens[1].volume").draft() == "4"); + CHECK(f.form->field("specimens").rows().size() == 2); // one row added, to maxItems + f.scheduler.advance(150ms); f.h.settle(); + CHECK(f.evaluate.submits().size() == 1); // one paste, one round + f.grid->select(ui::CellRange{at(1, "volume"), at(2, "note")}); + f.grid->fillDown(); + CHECK(f.form->field("specimens[1].volume").draft() == "1"); + CHECK(f.form->field("specimens[1].note").draft() == "a"); + f.grid->clear(); + CHECK(f.form->field("specimens[0].volume").draft().empty()); + CHECK(f.form->field("specimens[1].note").draft().empty()); + CHECK(f.form->field("specimens[0].density").draft().empty()); // read-only: never written +} + +// Review Focus 1, paste half. Mutation: an empty grid's paste starting nowhere and appending nothing. +TEST_CASE("grid: paste into an empty grid", "[ui][grid][cells]") { + SECTION("with appendRows") { + GridFixture f{0, ui::GridOptions{.appendRows = true}}; + auto const result = f.grid->paste("x\t1\ny\t2\nz\t3"); + CHECK(result.written == 6); + CHECK(f.form->field("specimens").rows().size() == 3); + CHECK(f.form->field("specimens[2].label").draft() == "z"); + CHECK(f.form->field("specimens[2].mass").draft() == "3"); + } + SECTION("without appendRows") { + GridFixture f{0}; + auto const result = f.grid->paste("x\t1\ny\t2"); + CHECK(result.written == 0); + CHECK(result.pastRows == 4); + CHECK(f.form->field("specimens").rows().empty()); + CHECK(f.grid->message() == "Pasted 0 values; 4 past the last row were dropped"); + } +} + +// Review Focus 3. Mutation: lines split on "\n" only, keeping "\r" and the trailing empty line. +TEST_CASE("grid: spreadsheet clipboard text pastes as typed values", "[ui][grid][cells]") { + GridFixture f{2, ui::GridOptions{.appendRows = true}}; + f.grid->focusCell(at(1, "mass")); + auto const result = f.grid->paste(" 12,5 \t1.234,5\r\n3\t4\r\n"); + CHECK(result.written == 4); + CHECK(result.pastRows == 0); + CHECK(f.form->field("specimens").rows().size() == 2); // no third row from the trailing line break + CHECK(f.json("specimens[0].mass") == R"({"num":1250,"den":100,"dp":2})"); + CHECK(f.json("specimens[0].volume") == R"({"num":12345,"den":10,"dp":1})"); + CHECK(f.form->field("specimens[1].volume").draft() == "4"); +} + +// Mutation: insert and delete without checking minItems and maxItems. +TEST_CASE("grid: rows of a grid", "[ui][grid][rows]") { + std::string schema{kSpecimens}; + schema.replace(schema.find(R"("minItems":0)"), 12, R"("minItems":1)"); + schema.replace(schema.find(R"("maxItems":5)"), 12, R"("maxItems":2)"); + GridFixture f{1, {}, schema}; + f.grid->focusCell(at(1, "label")); + CHECK_FALSE(f.grid->rowOps().remove); + CHECK(f.grid->rowOps().why == "A grid holds at least 1 rows"); + f.grid->press("Ctrl+Shift+="); + CHECK(f.form->field("specimens").rows().size() == 2); + CHECK_FALSE(f.grid->rowOps().insert); + CHECK(f.grid->rowOps().why == "A grid holds at most 2 rows"); + f.grid->type("kept"); f.grid->press("Enter"); // (1, label) + f.grid->focusCell(at(1, "label")); + f.grid->press("Ctrl+-"); + CHECK(f.form->field("specimens").rows().size() == 1); + CHECK(f.grid->message() == "Deleted 1 row. Undo"); + f.grid->undo(); + CHECK(f.form->field("specimens[0].label").draft() == "kept"); + SECTION("rowsFixed offers none") { + GridFixture fixed{2, ui::GridOptions{.rowsFixed = true}}; + CHECK_FALSE(fixed.grid->rowOps().insert); + CHECK_FALSE(fixed.grid->rowOps().remove); + fixed.grid->focusCell(at(2, "note")); + CHECK(fixed.grid->press("Tab") == ui::KeyOutcome::FocusNext); + } + SECTION("appendRows adds a row on Tab past the end") { + GridFixture grow{1, ui::GridOptions{.appendRows = true}}; + grow.grid->focusCell(at(1, "note")); + CHECK(grow.grid->press("Tab") == ui::KeyOutcome::Handled); + CHECK(grow.form->field("specimens").rows().size() == 2); + CHECK(grow.grid->focus() == at(2, "label")); + } +} +``` + +- [ ] **Step 2: Run to see them fail** — `cmake --build --preset cl-debug --target morph_tests`; expected: `GridController::paste` not declared. + +- [ ] **Step 3: Implement** `minItems`/`maxItems`, `removeRows`/`restoreRows`, then the selection, copy, paste, fill, clear and row operations in `grid.hpp`; then the lowering, the loader's refusal, and the backends' row header and column header selection gestures. + +- [ ] **Step 4: Run** `ctest --test-dir build/cl-debug -R "grid:" --output-on-failure` — all pass. + +- [ ] **Step 5: Mutate**, **Step 6: Docs**, **Step 7: Commit** `feat(ui): cell selection, paste and row operations in grids`. + +### Task 22: Bands, spans and summary rows (editing.md step 4) + +**Spec:** editing.md "Header bands", "Spans", "Summary rows", Invariant 8; Failure modes rows on bands, spans and `summaries`; table.md "Column groups" (a group draws as a one-level band). + +**Files:** +- Modify: `include/morph/table/description.hpp`, `include/morph/ui/view.hpp`, `include/morph/ui/mount.hpp`, `include/morph/ui/backend.hpp`, `include/morph/ui/grid.hpp`, `include/morph/ui/testing/recording_backend.hpp`, `src/ui/document.cpp`, `src/ui/interpreter.cpp`, `include/morph/ui/interpreter.hpp`, `src/qt_quick/table.cpp` and QML, `src/tui/table_widget.cpp` +- Tests: `tests/test_ui_mount.cpp`, `tests/test_ui_grid.cpp`, `tests/table/test_table_description.cpp`, `tests/qt_quick/test_qt_quick_table.cpp`, `tests/tui/test_tui_table.cpp` +- Docs: `docs/spec/ui/document.md`, `docs/spec/ui/interpreter.md` (`bands`, a span part's `columns`, `summaries` on `table` and `group`), `docs/spec/ui/view_tree.md` (spans across columns, summary rows), `docs/spec/qt_quick/backend.md` (bands and spans). + +**Interfaces:** +- Consumes: `table::ColumnLayout` (Task 8d), `ui::TableColumn::id` and `Table::arrangement` (Task 9; the mount maps the arrangement's visible indices to ids for `bandRuns`), `ui::formatCell` (Task 15), Task 19's `GridSource::covered`, `GridRows`, `TableRows`. +- Produces: + - `description.hpp`: `struct BandInfo { std::string label; std::string unit; std::vector columns; std::vector bands; };` `std::vector TableDescription::bands;` `[[nodiscard]] std::expected checkBands(std::span bands, std::span columns);` — `SourceContract` for a nesting past 3 levels, a band naming no column, a child over a column its parent lacks, a column in two bands of one level, or a column id the table does not have; the message names the band's label. + - `view.hpp`: `struct BandRun { std::size_t level = 0; std::string label; std::string unit; std::size_t first = 0; std::size_t count = 0; bool pinned = false; bool operator==(BandRun const&) const = default; };` `std::vector Table::bands;` `std::size_t RowSpan::columns = 1;` `bool TableRowView::summary = false;` + - `mount.hpp`: `[[nodiscard]] std::vector bandRuns(std::span bands, std::span visible, std::size_t pinned);` — `visible` the window's visible column ids in layout order; one run per maximal run of a band's columns standing together, split at the pinned edge; levels from 0 (outermost); a band with no visible column yields none. `[[nodiscard]] bool columnUnitShown(std::span bands, std::string_view columnId, std::string_view unit);` (false when an enclosing band's unit equals `unit`). `[[nodiscard]] std::string bandedHeaderName(std::span bands, std::string_view columnId, std::string_view label);` (outermost band first, joined by `", "`). `struct CellSpan { std::string column; std::size_t rows = 1; std::size_t columns = 1; };` `[[nodiscard]] std::size_t cutColumns(CellSpan const& span, std::span declared, std::span visible);` — the span covers its column, then each next visible column while it is the next declared one, up to `columns`. + - `backend.hpp`: `virtual void TableWidget::setBands(std::vector const& runs) {}`; `virtual void TableWidget::setRowSummary(Widget& row) {}` (a summary row: drawn on the theme's `cellSummary` surface, never selected, activated, focused for an edit or pasted into; copied when a copy covers it). `setRowSpans` carries `RowSpan::columns`. + - `grid.hpp`: `[[nodiscard]] ValueList summaryScopeRows(TableRows const& rows);` — the records of the view's rows, in view order (a grid row's record holds each member's encoded value, `null` when blank or not encoding). `void FormGridSource::setSpans(std::function(forms::RowState const&)> spansOf);` — `covered(key, column)` is true for a cell another row's span covers (rows below, columns right by `cutColumns`), while the grid is unsorted and under its windowing thresholds. + - Document: `bands` on `table` (parts `{label, unit, columns, bands}`, refused as `checkBands` refuses); a `rowSpans` part's `columns`; `summaries` on `table` and on a `group` part, parts `{key, label, position, cells}` (`position` `bottom` or `top`; `cells` maps column ids to expressions over `rows` and `column`), refused on a model-backed table when the document loads. A summary row's key is `"\x1Fsummary:"` + its key (a group's: `"\x1Fsummary:"` + the run's key + `"/"` + its key). A summary cell's test id is `summary__` (a group's `summary___`). A user-formed or declared column group draws as a band of one level. + +- [ ] **Step 1: Write the failing tests.** `tests/test_ui_mount.cpp`: + +```cpp +namespace { +std::vector const kMassBand{{.label = "Mass retained", .unit = "g", .columns = {"m8", "m4", "m2"}}}; +std::vector const kIds{"name", "m8", "m4", "m2", "total"}; +} + +// Mutation: a band drawn from its first to its last column's place, which covers a foreign column. +TEST_CASE("bands: a band follows the layout", "[ui][mount][bands]") { + CHECK(ui::bandRuns(kMassBand, kIds, 0) == std::vector{{0, "Mass retained", "g", 1, 3, false}}); + std::vector const moved{"name", "m8", "m2", "total", "m4"}; + CHECK(ui::bandRuns(kMassBand, moved, 0) == + std::vector{{0, "Mass retained", "g", 1, 2, false}, {0, "Mass retained", "g", 4, 1, false}}); + CHECK(ui::bandRuns(kMassBand, kIds, 2) == + std::vector{{0, "Mass retained", "g", 1, 1, true}, {0, "Mass retained", "g", 2, 2, false}}); + CHECK(ui::bandRuns(kMassBand, std::vector{"name", "total"}, 0).empty()); + CHECK_FALSE(ui::columnUnitShown(kMassBand, "m8", "g")); + CHECK(ui::columnUnitShown(kMassBand, "total", "g")); + CHECK(ui::bandedHeaderName(kMassBand, "m4", "sieve 4 mm") == "Mass retained, sieve 4 mm"); +} + +// Review Focus 2. Mutation: bandRuns computed once from the declared order and not from the layout's visible columns. +TEST_CASE("bands: a stored layout hides then shows a band's column", "[ui][mount][bands]") { + table::ColumnLayout layout{kIds}; + table::ColumnLayout stored{kIds}; + for (auto const* id : {"m8", "m4", "m2"}) stored.setVisible(id, false); + layout.apply(stored.value()); + CHECK(ui::bandRuns(kMassBand, layout.visible(), layout.pinned()).empty()); + layout.setVisible("m4", true); + CHECK(ui::bandRuns(kMassBand, layout.visible(), layout.pinned()) == + std::vector{{0, "Mass retained", "g", 1, 1, false}}); + table::ColumnLayout pinnedAll{kIds}; + pinnedAll.setPinned(4); + CHECK(ui::bandRuns(kMassBand, pinnedAll.visible(), pinnedAll.pinned()) == + std::vector{{0, "Mass retained", "g", 1, 3, true}}); +} + +// Mutation: covered cells left focusable (the walk lands on one). +TEST_CASE("spans: a cell spanning rows and columns covers them", "[ui][mount][spans]") { + std::vector const declared{"a", "b", "c", "d"}; + CHECK(ui::cutColumns({.column = "a", .rows = 2, .columns = 3}, declared, declared) == 3); + CHECK(ui::cutColumns({.column = "a", .rows = 2, .columns = 3}, declared, std::vector{"a", "c", "b", "d"}) == 1); + CHECK(ui::cutColumns({.column = "a", .rows = 2, .columns = 3}, declared, std::vector{"a", "b", "d"}) == 2); + // and, through RecordingBackend, a table row giving {a, rows 2, columns 3} reports spans "a:2x3" for that row +} +``` + +`tests/table/test_table_description.cpp`: + +```cpp +// Mutation: checkBands not walking child bands. +TEST_CASE("description: bands are checked", "[table][description][bands]") { + // columns m8, m4, m2, total + CHECK(table::checkBands(kMassBand, columns).has_value()); + // four levels deep; a band over no column; a child over "total" which its parent lacks; "m8" in two bands of one level + // each: CHECK(error().code == table::TableErrorCode::SourceContract) and the message contains the band's label +} +``` + +`tests/test_ui_grid.cpp`: + +```cpp +// Mutation: a paste writing into a covered cell (the spanning cell's value changes twice, or the covered field is written). +TEST_CASE("grid: spans pass the walk and drop pasted values", "[ui][grid][spans]") { + GridFixture f{3}; + f.source->setSpans([](forms::RowState const& row) { + return row.key() == 1 ? std::vector{{.column = "mass", .rows = 2, .columns = 2}} : std::vector{}; + }); + CHECK(f.source->covered(ui::Key{std::int64_t{2}}, "mass")); + CHECK(f.source->covered(ui::Key{std::int64_t{1}}, "volume")); + CHECK(f.source->covered(ui::Key{std::int64_t{2}}, "volume")); + f.grid->focusCell(at(1, "mass")); + f.grid->press("Tab"); + CHECK(f.grid->focus() == at(1, "note")); // volume covered, density and limit read-only + f.grid->focusCell(at(1, "mass")); f.grid->type("1"); f.grid->press("Enter"); + CHECK(f.grid->focus() == at(3, "mass")); // (2, mass) is covered + f.grid->focusCell(at(1, "mass")); + auto const result = f.grid->paste("1\n2\n3"); + CHECK(result.written == 2); + CHECK(result.readOnly == 1); + CHECK(f.form->field("specimens[1].mass").draft().empty()); + table::ColumnLayout layout{{"label", "mass", "volume", "density", "limit", "note"}}; + // moving volume away cuts the span: with volume after note, (1, volume) is no longer covered + layout.move("volume", 5); + f.source->setLayout(layout.visible()); // the grid's visible columns follow the layout + CHECK_FALSE(f.source->covered(ui::Key{std::int64_t{1}}, "volume")); +} + +// Mutation: summaryScopeRows reading the source's snapshot, which ignores the filter. +TEST_CASE("summary: rows follow the filter", "[ui][grid][summary]") { + GridFixture f{3}; + for (auto const& [key, text] : {std::pair{1, "10"}, {2, "12"}, {3, "14"}}) { + f.grid->focusCell(at(key, "mass")); f.grid->type(text); f.grid->press("Enter"); + } + REQUIRE(f.source->view().setFilter(table::FilterSpec{.columns = {{"mass", + table::ColumnFilter{.include = {table::FilterEntry{.gt = "11"}}}}}}).has_value()); + ValueList const rows = ui::summaryScopeRows(f.source->view()); + REQUIRE(rows.size() == 2); + CHECK(canonicalJson(*rows.at(0).field("mass")) == R"({"num":1200,"den":100,"dp":2})"); +} +``` + +Interpreter cases (Harness, which formats in `en`; document: the `Lims_RecordSpecimens` form and a grid with `"summaries":[{"key":"stats","cells":{"mass":{"fn":"mean","args":[{"map":{"ref":"rows"},"as":"r","to":{"ref":"r.mass"}}]},"label":{"fn":"count","args":[{"ref":"rows"}]}}},{"key":"sd","cells":{"mass":{"fn":"stdev","args":[{"map":{"ref":"rows"},"as":"r","to":{"ref":"r.mass"}},2]}}}]`): + +```cpp +// Mutation: summaries evaluated once at lowering (they do not follow a committed value). +TEST_CASE("summary: mean and stdev follow every committed value", "[ui][grid][summary][interpreter]") { + // masses 10.00 / 12.00 / 14.00 typed through the grid + CHECK(h.text("summary_stats_mass") == "12.00"); + CHECK(h.text("summary_sd_mass") == "2.00"); + CHECK(h.text("summary_stats_label") == "3"); + // type 16.00 into row 3 + CHECK(h.text("summary_stats_mass") == "12.67"); // 38/3 at the column's 2 places +} +// Review Focus 1, summary half. Mutation: count over the source rows or mean of [] shown as "0". +TEST_CASE("summary: over no rows", "[ui][grid][summary][interpreter]") { + // the same document with no specimens + CHECK(h.text("summary_stats_label") == "0"); + CHECK(h.text("summary_stats_mass").empty()); +} +TEST_CASE("summary: group summaries are per run", "[ui][grid][summary][interpreter]") { + // group by r.label with summaries [{key: "m", cells: {mass: mean}}]; labels A, A, B; masses 10, 12, 14 + CHECK(h.text("summary_0_m_mass") == "11.00"); + CHECK(h.text("summary_1_m_mass") == "14.00"); +} +TEST_CASE("summary: a model-backed table refuses summaries", "[ui][summary][interpreter]") { + // {"kind":"table","model":"m","table":"specimens","summaries":[…]} and a group part with summaries: both refused +} +``` + +Qt Quick: `"qt_quick table: a band draws over each run of its columns and across the pinned edge"` — after `setBands({{0,"Mass retained","g",1,2,false},{0,"Mass retained","g",4,1,false}})` the header holds two band items whose x and width equal the covered columns' extents. TUI: `"tui table: bands draw above the header"` — the rendered header's first line holds `Mass retained [g]` over columns 1–3. + +- [ ] **Step 2: Run to see them fail** — `cmake --build --preset cl-debug --target morph_tests`; expected: `ui::bandRuns` not declared. + +- [ ] **Step 3: Implement** `BandInfo`, `checkBands`, `bandRuns`, `columnUnitShown`, `bandedHeaderName`, `cutColumns`; the mount's bands (over the column window's columns), its column spans and summary rows; `summaryScopeRows`; the lowering and loader; then the Qt Quick and terminal drawing. + +- [ ] **Step 4: Run** `ctest --test-dir build/cl-debug -R "bands|spans|summary|description" --output-on-failure` (and the Qt Quick and TUI patterns `qt_quick table` and `tui table`) — all pass; the existing row-span cases in `test_ui_table_row_span.cpp` still pass with `columns` 1. + +- [ ] **Step 5: Mutate**, **Step 6: Docs**, **Step 7: Commit** `feat(ui): header bands, spans and summary rows`. + +### Task 23: Fixed grids (editing.md step 5) + +**Spec:** editing.md "Fixed grids" (with blocks), Failure modes rows "Two `layoutRows` parts with one key" and "`form` and `rows` both on a table, `layoutRows` with `rows` or `field`, or a `{field}` cell in a fixed grid without `form`". + +**Files:** `include/morph/ui/grid.hpp`, `src/ui/document.cpp`, `src/ui/interpreter.cpp`, `include/morph/ui/interpreter.hpp`, `tests/ui_interpreter_support.hpp` (schema `Lims_SetLimits`), `tests/test_ui_grid.cpp`; docs `docs/spec/ui/document.md`, `docs/spec/ui/interpreter.md` (`layoutRows`, cell parts, blocks). + +**Interfaces:** +- Consumes: Task 19–21 `GridSource`, `GridController`, `cellReadOnly`; Task 22 `CellSpan`, `cutColumns`. +- Produces: `grid.hpp`: `struct FixedCell { std::string field; std::function value; std::string text; std::size_t rows = 1; std::size_t columns = 1; std::function readonly; std::function()> decimals; };` `struct FixedRow { std::string key; std::string label; std::map> cells; };` `struct FixedBlock { std::string forEach; std::vector rows; };` `class FixedGridSource final : public GridSource` — `FixedGridSource(forms::FormSession& form, std::vector> rows, std::vector columns)`; row keys are `Key{std::string}`: a fixed row's `key`, a block row's `"/"`; a block's `{field}` path is relative to the specimen's row (`specimens#.`); a cell with `{text}` or `{value}` is read-only. Not sorted, filtered or rearranged by rows; columns may be hidden and resized. + +- [ ] **Step 1: Write the failing tests** — `Lims_SetLimits` schema: `limits` object with `density {min, max}` (decimals, 3 places), `moisture {min, max}` (decimals, 1 place), `method` (enum `["A","B"]`), `count` (integer); and `specimens` (array of `{mass: decimal 2}`). + +```cpp +// Mutation: the walk asking the field's spec of the column's first row for every row (one kind per column). +TEST_CASE("fixed grid: four parameters walk, paste and validate per cell", "[ui][grid][fixed]") { + // rows: density {min: limits.density.min, max: limits.density.max}; moisture {min, max}; + // method {min: {field: limits.method, columns: 2}}; count {min: limits.count, max: {text: "—"}} + f.grid->focusCell(ui::CellRef{ui::Key{std::string{"density"}}, "min"}); + std::vector> walked; + for (int i = 0; i < 5; ++i) { f.grid->press("Tab"); walked.emplace_back(std::get(f.grid->focus()->key), f.grid->focus()->column); } + CHECK(walked == std::vector>{ + {"density", "max"}, {"moisture", "min"}, {"moisture", "max"}, {"method", "min"}, {"count", "min"}}); + CHECK(f.grid->press("Tab") == ui::KeyOutcome::FocusNext); // count.max is text: read-only + f.grid->focusCell(ui::CellRef{ui::Key{std::string{"density"}}, "min"}); + auto const result = f.grid->paste("1,2345\t2\n0,55\tx"); + CHECK(result.written == 4); + CHECK(f.form->field("limits.density.min").error().peek()->code == "too-precise"); // 3 places + CHECK(f.json("limits.density.max") == R"({"num":2000,"den":1000,"dp":3})"); + CHECK(f.form->field("limits.moisture.min").error().peek()->code == "too-precise"); // 1 place + CHECK(f.form->field("limits.moisture.max").error().peek()->code == "malformed"); +} + +// Mutation: block row keys by position, which move every row's state on insertion. +TEST_CASE("fixed grid: a forEach block repeats per specimen with stable keys", "[ui][grid][fixed]") { + // rows: [{forEach: "specimens", rows: [{key: "mass", cells: {v: {field: "mass"}}}]}], two specimens (keys 1, 2) + CHECK(f.source->rows() == std::vector{ui::Key{std::string{"1/mass"}}, ui::Key{std::string{"2/mass"}}}); + f.grid->focusCell(ui::CellRef{ui::Key{std::string{"2/mass"}}, "v"}); f.grid->type("5"); f.grid->press("Enter"); + std::uint64_t const added = f.form->field("specimens").insertRow(0); + CHECK(f.source->rows().front() == ui::Key{std::to_string(added) + "/mass"}); + CHECK(f.source->draft(ui::Key{std::string{"2/mass"}}, "v") == "5"); +} + +TEST_CASE("fixed grid: the loader refuses what the spec refuses", "[ui][grid][fixed]") { + // each refused by loadDocumentText: two layoutRows parts with key "density"; layoutRows with rows; + // layoutRows with field; a {field} cell without form; form with rows +} +``` + +- [ ] **Step 2: Run to see them fail**; **Step 3: Implement** `FixedGridSource` and the lowering of `layoutRows` (expanding blocks on each change of the block's array rows); **Step 4: Run** `ctest --test-dir build/cl-debug -R "fixed grid" --output-on-failure` — pass; **Step 5: Mutate**; **Step 6: Docs**; **Step 7: Commit** `feat(ui): fixed grids`. + +### Task 24: Cells of a model's table (editing.md step 6) + +**Spec:** editing.md "Cells of a model's table", "Read-only and computed" (the model-table half), "Walking" (moving by view position), Invariant 6; Failure modes rows on edit kinds, `editMode`, refused edits, `FieldError`/`FailedRow`; table.md "Rows an action changed" (a key that changes), "Rows at work, rows that failed". Needs Tasks 9 and 17. + +**Files:** +- Modify: `include/morph/table/edits.hpp`, `include/morph/table/description.hpp`, `include/morph/ui/interpreter.hpp` and `src/ui/interpreter.cpp` (`TableInputKind::CellEdit` in `tableInputsOf`, so a `CellEdit`/`RowEdit` run marks its row busy through `RowStates`), `include/morph/ui/grid.hpp`, `src/ui/document.cpp`, `src/ui/interpreter.cpp`, `include/morph/ui/interpreter.hpp`, `tests/ui_interpreter_support.hpp` (schemas `Lims_EditSpecimenCount`, `Lims_EditSpecimenRow`) +- Tests: `tests/table/test_table_pending.cpp` (the `CellEdits` cases, beside its existing ones), `tests/table/test_table_description.cpp`, `tests/test_ui_table_model.cpp`, `tests/net/test_push_tables.cpp` +- Docs: `docs/spec/ui/document.md`, `docs/spec/ui/interpreter.md` (`editMode`, an editable model column's lowering), `docs/spec/table/engine.md` (`CellEdits` row commits, drafts by key). + +**Interfaces:** +- Consumes: `TableRows`, `WindowRows::setDescription`, `onStaleDescription` (Task 9); `TableRows::states()` (`isBusy`, `failure`), `TableRows::moveKey`, `lockText`, `refusesEdit`, `FailedRow`, `ChangedRows::failed` (Task 17c); `DeclaresSchemaValue`, `model::recordRegistry()`, `TableInputKind`, `tableInputsOf` (Task 17a); `ScreenCore::tableMessage` (Task 12c); `Session::can(action)` (existing); `forms::FormModel::fromSchema`, `forms::FormSession` (existing); Task 19's `GridController`, `GridSource`, `cellReadOnly`. +- Produces: + - `edits.hpp`: `template struct CellEdit { RowId key; T value; };` and `template struct RowEdit { RowId key; Values values; };`, each with a `schemaValue()` (Task 17a's `DeclaresSchemaValue`) returning `{"x-cellEditOf", name}` / `{"x-rowEditOf", name}`, `name` the `RecordInfo::name` in `model::recordRegistry()` whose `type` is `typeid(TableOf)`; `TableInputKind` gains `CellEdit` and `tableInputsOf` lists both keywords, the busy key being the body's `key`; `struct EditInfo { std::string action; std::string member; std::string readonly; };` `[[nodiscard]] std::expected checkEdits(TableDescription const& description, std::function(std::string_view action)> const& schemaOf);` — `SourceContract` when a column's kind does not fit its field's kind, or `member` is not a member of `values`; the fits are `Integer` ↔ {Integer, Slider, Enum, Choice}, `Key` ↔ {Integer, Choice}, `Decimal` ↔ {Decimal}, `Quantity` ↔ {Quantity}, `Measure` ↔ {Quantity}, `Number` ↔ {Number, Slider}, `Text` ↔ {Text, Multiline, Enum, Choice}, `Date` ↔ {Date, EpochDate}, `DateTime` ↔ {DateTime, EpochDateTime}, `Bool` ↔ {Boolean}, `Custom` ↔ {Text, Enum, Choice}. Task 2's describe path calls it with the dispatcher's action schemas. + - `edits.hpp`, `CellEdits` additions: `struct EditCell { std::string column; std::string json; };` `struct RowEditRequest { RowId row; std::vector cells; };` `using RowEditDone = std::function>)>;` `using RowEditCommit = std::function;` `explicit CellEdits(RowEditCommit commit);` (a model's table: no patcher, no engine); `void commitRow(RowEditRequest request);` (queued with the row's other edits, in order); `void setDraft(RowId const& row, std::string_view column, std::string text, std::optional error);` `[[nodiscard]] std::optional draft(RowId const&, std::string_view) const;` `void clearDraft(RowId const&, std::string_view);` `[[nodiscard]] bool modified(RowId const&, std::string_view) const;` (from a success until `rowFetched`); `void rowFetched(RowId const& row);` (ends the row's modified marks and its settled drafts; a stale cell is left as it is); `void moveRow(RowId const& from, RowId const& to);` (drafts, errors, modified marks and queued edits); `void keepColumns(std::span columns);` (drops drafts, errors and marks of every other column); `[[nodiscard]] std::size_t errorCount() const;` `[[nodiscard]] std::size_t modifiedCount() const;` `stale` and `error` see row commits too. `TableColumn::edit` (`std::optional`). + - `grid.hpp`: `struct ModelColumn { std::string id; std::optional edit; std::optional field; };` `enum class EditMode : std::uint8_t { Cell, Row };` `class ModelGridSource final : public GridSource` — `ModelGridSource(TableRows& rows, table::CellEdits& edits, std::vector columns, EditMode mode, std::function may)`; read-only through `cellReadOnly` with `hasEdit = edit && may(edit->action)`, `companionReadOnly` from the `EditInfo::readonly` column's `true`, `busy` and `locked` from `refusesEdit(rows, index, description)` (busy is `rows.states().isBusy(key)`, locked is `lockText(row, description)`), with one rule on top: a row whose only busy run is this source's own `CellEdits` queue is not refused; `unloaded` from the row's cell. A draft is encoded with the column's `FieldSpec` (`forms::encodeScalar`); an editor is a one-field `forms::FormSession` over that spec. Cell mode: a commit sends `RowEditRequest{key, {{column, json}}}`; row mode: a row's cells are collected while focus stays in the row and sent as one request when focus leaves it or on `Ctrl+Enter`; a refusal sets each `FieldError` on its cell and, while the table has focus, moves focus to the first refused cell. A success whose `ChangedRows` does not name the row calls `rowFetched` at once. `void ModelGridSource::moveKey(Key const& from, Key const& to)` calls `rows.moveKey(from, to)` and `edits.moveRow(from, to)`; the lowering of an editable table calls it where Task 17c's lowering calls `TableRows::moveKey` for a `ChangedRow::from`. `void ModelGridSource::setDescription(table::TableDescription const&)` calls `keepColumns` with its column ids; the lowering calls it wherever it calls `WindowRows::setDescription`. A `FailedRow` for the row is a `FieldError` on each cell of the request. + - Status-line copy (through `ScreenCore::tableMessage`): an edit refused at a cell: unloaded `"This value is not loaded yet"`, busy `"This row is being saved"`, locked: the row's lock text. + - Document: `editMode` (`cell`, the default, or `row`) on a model's table; refused on any other table. + +- [ ] **Step 1: Write the failing table tests** — `tests/table/test_table_pending.cpp`: + +```cpp +namespace { +struct RowCommits { + std::vector sent; + std::vector done; + table::RowEditCommit commit() { return [this](auto const& r, auto d) { sent.push_back(r); done.push_back(std::move(d)); }; } +}; +table::RowId const k1{std::int64_t{1}}; +} + +// Mutation: rowFetched clearing a stale cell's draft. +TEST_CASE("cell edits: stale until the reply, modified until the next fetch", "[table][edits]") { + RowCommits c; + table::CellEdits edits{c.commit()}; + edits.setDraft(k1, "count", "9", std::nullopt); + edits.commitRow({k1, {{"count", "9"}}}); + CHECK(edits.stale(k1, "count")); + c.done.at(0)(std::expected>{}); + CHECK_FALSE(edits.stale(k1, "count")); + CHECK(edits.modified(k1, "count")); + CHECK(edits.draft(k1, "count") == "9"); + edits.rowFetched(k1); + CHECK_FALSE(edits.modified(k1, "count")); + CHECK_FALSE(edits.draft(k1, "count").has_value()); +} + +// Review Focus 4. Mutation: rowFetched ending stale edits too. +TEST_CASE("cell edits: a fetch before the reply keeps the cell stale", "[table][edits]") { + RowCommits c; + table::CellEdits edits{c.commit()}; + edits.setDraft(k1, "count", "9", std::nullopt); + edits.commitRow({k1, {{"count", "9"}}}); + edits.rowFetched(k1); // the change event's refetch, before the reply + CHECK(edits.stale(k1, "count")); + CHECK(edits.draft(k1, "count") == "9"); + c.done.at(0)(std::expected>{}); + CHECK(edits.modified(k1, "count")); + edits.rowFetched(k1); + CHECK_FALSE(edits.modified(k1, "count")); +} + +// Mutation: a row's second request started before the first settles. +TEST_CASE("cell edits: a row's edits run one at a time, a refusal lands on each cell", "[table][edits]") { + RowCommits c; + table::CellEdits edits{c.commit()}; + edits.commitRow({k1, {{"label", R"("a")"}, {"note", R"("b")"}}}); + edits.commitRow({k1, {{"note", R"("c")"}}}); + CHECK(c.sent.size() == 1); + c.done.at(0)(std::unexpected(std::vector{{"label", "taken"}, {"note", "too long"}})); + CHECK(edits.error(k1, "label") == "taken"); + CHECK(edits.error(k1, "note") == "too long"); + CHECK(c.sent.size() == 2); + CHECK(edits.errorCount() == 2); +} + +// Mutation: moveRow moving only drafts (the error stays on the old key). +TEST_CASE("cell edits: drafts are held by key and follow a key change", "[table][edits]") { + table::CellEdits edits{[](auto const&, auto) {}}; + edits.setDraft(table::RowId{std::int64_t{5}}, "count", "abc", "Not a number"); + CHECK(edits.errorCount() == 1); + edits.moveRow(table::RowId{std::int64_t{5}}, table::RowId{std::int64_t{9}}); + CHECK_FALSE(edits.draft(table::RowId{std::int64_t{5}}, "count").has_value()); + CHECK(edits.draft(table::RowId{std::int64_t{9}}, "count") == "abc"); + CHECK(edits.errorCount() == 1); +} + +// Review Focus 5. Mutation: keepColumns dropping drafts but not their errors. +TEST_CASE("cell edits: a new revision drops drafts of a removed column", "[table][edits]") { + table::CellEdits edits{[](auto const&, auto) {}}; + edits.setDraft(k1, "count", "abc", "Not a number"); + edits.setDraft(k1, "note", "kept", std::nullopt); + std::vector const kept{"label", "note"}; + edits.keepColumns(kept); + CHECK_FALSE(edits.draft(k1, "count").has_value()); + CHECK(edits.draft(k1, "note") == "kept"); + CHECK(edits.errorCount() == 0); +} +``` + +`tests/table/test_table_description.cpp`: + +```cpp +// Mutation: checkEdits not comparing kinds. +TEST_CASE("description: an edit whose field does not fit its column is a source contract", "[table][description][edits]") { + // column "count" (Integer) with edit {action "Lims_EditSpecimenCount"} whose value schema is {"type":"string"}: + CHECK(table::checkEdits(description, schemaOf).error().code == table::TableErrorCode::SourceContract); + // column "label" (Text) with edit {action "Lims_EditSpecimenRow", member "nosuch"}: SourceContract, message names "nosuch" + // with value {"type":"integer"} and member "label" present: has_value() +} +``` + +- [ ] **Step 2: Write the failing UI tests** — `tests/test_ui_table_model.cpp`, over a `FakeRows : ui::TableRows` in the test (rows by key in a vector, a `RowStates` its `states()` returns, a `rowLock` companion column for locks, unloaded cells, `moveKey` recording its calls) and a `ModelGridSource` whose `may` reads a `std::set` the test edits. Columns: `label` and `remark` (Text, edit `Lims_EditSpecimenRow`, members `label` and `remark`), `count` (Integer, edit `Lims_EditSpecimenCount`, readonly companion `countFrozen`), `note` (no edit). `f.commits` is a `RowCommits`. + +```cpp +// Mutation: the busy test left out of ModelGridSource::readOnly (row 3 takes the edit). +TEST_CASE("model table: read-only is one rule", "[ui][table][edit]") { + // rows 1..5; row 2 count unloaded; row 3 busy (states().busyKeys(7, {3}): another action's run); row 4's rowLock + // cell "Being edited by another user"; + // row 5 countFrozen true + for (std::int64_t key : {2, 3, 4, 5}) CHECK(f.source->readOnly(ui::Key{key}, "count")); + CHECK(f.source->readOnly(ui::Key{std::int64_t{1}}, "note")); // no edit action + f.grid->focusCell(at(2, "count")); // arrows and clicks reach read-only cells + f.grid->type("7"); + CHECK(f.grid->message() == "This value is not loaded yet"); + f.grid->focusCell(at(3, "count")); f.grid->type("7"); + CHECK(f.grid->message() == "This row is being saved"); + f.grid->focusCell(at(4, "count")); f.grid->type("7"); + CHECK(f.grid->message() == "Being edited by another user"); + CHECK(f.commits.sent.empty()); + f.grid->focusCell(at(1, "count")); f.grid->type("3"); f.grid->press("Enter"); + CHECK(f.commits.sent.size() == 1); + CHECK(f.grid->focus() == at(1, "count")); // Enter down passes rows 2–5, all read-only in count: it stays + f.permitted = {"Lims_EditSpecimenRow"}; // count's action refused + CHECK(f.source->readOnly(ui::Key{std::int64_t{1}}, "count")); +} + +// Mutation: the walk following the edited row instead of the view position at the commit. +TEST_CASE("model table: Enter after an edit that moves its row lands on the row that was below", "[ui][table][edit]") { + // view [1, 2, 3], all editable; focus (1, count), type "9", Enter; FakeRows re-sorts to [2, 3, 1] + CHECK(f.grid->focus() == at(2, "count")); +} + +// Mutation: drafts held by the window (FakeRows dropping row 5 loses them). +TEST_CASE("model table: cells of a model's table", "[ui][table][edit]") { + // cell mode: focus (1, count), type "9", Enter -> one RowEditRequest{1, {{"count","9"}}}; marks stale; + // reply ok with ChangedRows naming 1 -> marks modified; FakeRows refetches 1 -> not modified + // refusal: type "11" on (2, count) -> reply FieldError{"value","must be at most 10"} -> error "must be at most 10", + // draft "11" kept + // scroll: type "abc" on (5, count) -> errorCount 1; FakeRows drops then restores row 5 -> draft "abc", error kept + // from: f.source->moveKey(5, 9) -> draft and error on (9, count); FakeRows recorded moveKey(5, 9) + // row mode: EditMode::Row; type "a" on (1, label), Tab, type "b" on (1, remark), Enter -> nothing sent; + // focusCell (2, label) -> one RowEditRequest{1, {{"label", "\"a\""}, {"remark", "\"b\""}}}; + // a refusal FieldError{"label", "taken"} -> error on (1, label) and focus back on (1, label); + // on row 2, type "c" on (2, label) and press Ctrl+Enter -> one request for row 2 with focus still there +} +``` + +`tests/net/test_push_tables.cpp`, over Task 2's fixture with a model registering table `specimens`, `Lims_EditSpecimenCount : table::CellEdit<&SpecimensModel::specimens, std::int64_t>` declaring `effects::updatesTable<&SpecimensModel::specimens>` and returning `ChangedRows`: + +```cpp +// Mutation: the client sending {"key":…, "value":…} under the wrong action id (the model never sees it). +TEST_CASE("push tables: a cell edit reaches the model and the change event refetches the row", "[net][tables][edit]") { + // client commits count = 4 on row 1: the server model's row 1 count == 4; the other client's window refetches + // row 1; the editing client's cell is not modified after its refetch; the action's schema carries + // "x-cellEditOf":".specimens" +} +TEST_CASE("push tables: a principal refused the edit action sees the column read-only", "[net][tables][edit]") { + // the authoriser refuses Lims_EditSpecimenCount for principal "viewer": the catalog's permitted set lacks it, and + // the lowered count column is read-only for that client +} +``` + +Write each commented case out in Catch2 with the fixture's calls; the comments are the assertions and their order. + +- [ ] **Step 3: Run to see them fail** — `cmake --build --preset cl-debug --target morph_table_tests`; expected: `table::RowEditRequest` not declared. + +- [ ] **Step 4: Implement** `CellEdit`, `RowEdit`, `EditInfo`, `TableColumn::edit`, `checkEdits` (called from Task 2's describe path), the `CellEdits` additions; busy rows from `x-cellEditOf`/`x-rowEditOf` inputs in `table_rows.hpp`; `ModelGridSource`; the lowering of editable model columns, `editMode` and its refusal. + +- [ ] **Step 5: Run** + +Run: `ctest --test-dir build/cl-debug -R "cell edits|description|model table|pending" --output-on-failure`, and on `linux-everything` (where `morph_net_tests` builds) `ctest --test-dir build/linux-everything -R "push tables" --output-on-failure` +Expected: all pass, including every existing case in `test_table_pending.cpp`, `test_table_reentrancy.cpp` and `test_table_selection.cpp`. + +- [ ] **Step 6: Mutate**, **Step 7: Docs** (docs build), **Step 8: Commit** `feat(table): edit cells of a model's table`. + +--- + +## Decisions where the spec leaves a choice + +Each is what this plan implements; the task that lands it states it in the spec it updates. + +- **A row busy with its own cell edits still takes edits.** editing.md says edits to one row are queued in order (line 221) and that a busy row takes no edit (lines 236–237, 288, Invariant 6 at 482); table.md makes the row of a `CellEdit` busy (lines 2425–2427). Read together, a second cell of a row could never be entered while the first is saving. `ModelGridSource` refuses an edit for a row busy with any other action, and queues one for a row whose only work is its own `CellEdits` queue (Task 24). +- **A row-mode refusal brings focus back.** editing.md line 228–230 commits a row "when focus leaves it" and on refusal "keeps focus in the row". Focus has left by then; it is moved back to the first refused cell while the table has focus (Task 24). +- **Read-only cells draw no modified mark.** editing.md line 300 marks any cell whose value differs from the loaded one; its test row (521) wants typing to mark "that cell and its row, not its neighbours", while a derive reply changes the neighbouring computed cells of that row. `FormSession::modified` is false for read-only members (Task 19). +- **A deletion offers Undo; it does not ask first.** editing.md line 358–359 "asks once, in the table's status line, with Undo"; its test row (518) "offers Undo". Rows are deleted at once and the status line says `"Deleted {n} rows. Undo"` (Task 21). +- **Status-line copy the spec does not fix** is pinned in Tasks 19, 20, 21 and 24 (`"This cell is read-only"`, `"Derived values could not be recalculated. Recalculate"`, the paste and row messages, the refusal messages). +- **Busy rows from a `CellEdit` input** are listed in the roadmap's Task 17c, which lands before `CellEdit` exists; Task 24 adds `TableInputKind::CellEdit` (`x-cellEditOf`, `x-rowEditOf`) to Task 17a's `tableInputsOf`. The own-queue exemption is a rule of `ModelGridSource` on top of `refusesEdit`, which stays as Part F defines it. +- **`CellEdits` cases live in `tests/table/test_table_pending.cpp`**, beside that class's existing cases, in addition to the roadmap's `tests/test_ui_table_model.cpp` and `tests/net/test_push_tables.cpp`. `tests/forms/` is a new directory whose files join `morph_tests`. + +## Self-review notes + +- **Coverage.** Every editing.md Tests row has a test: "A grid is a form", "Read-only is one rule (grid)", "The walk" (grid), "A sorted grid keeps its record", "Submission", "What changed" — Task 19; "Derived values", "The server decides", "Kind per row", "Choices per row" — Task 20; "Cell selection", "Paste, fill, clear", "Rows of a grid" — Task 21; "Bands", "Spans", "Summary rows" — Task 22; "Fixed grids" — Task 23; "Cells of a model's table", "Read-only is one rule (model table)", "The walk" (model table) — Task 24. Invariants: 1 (Task 19, the grid writes only through `FieldState`), 2 (Task 20 "derived values": a typed member survives the reply), 3 (Task 20: an older reply ignored, values land by row key), 4 (`cellReadOnly`, Tasks 19 and 24), 5 (Task 19 "a sorted grid keeps its record"), 6 (Task 24 "read-only is one rule"), 7 (Task 21 "paste, fill, clear"), 8 (Task 22 "spans"), 9 (Task 20 "the server decides"). Every Failure modes row has its refusal or its test in the task that lands the key. +- **Not covered by a test of its own:** the accessible band name is asserted only at function level (`bandedHeaderName`); the Qt Quick backend's live region and its reading order are left to `qt_quick/backend.md`'s existing conformance. diff --git a/docs/superpowers/plans/2026-11-17-one-table.md b/docs/superpowers/plans/2026-11-17-one-table.md index b2209388c..68179d302 100644 --- a/docs/superpowers/plans/2026-11-17-one-table.md +++ b/docs/superpowers/plans/2026-11-17-one-table.md @@ -41,6 +41,54 @@ cmake --build --preset cl-debug --target ctest --test-dir build/cl-debug -R "" --output-on-failure ``` +## The detailed parts + +This document is the index. Each task is planned in full — signatures, test cases with their values, commands — in one of seven parts, which keep this document's task numbers: + +| Part | Tasks | File | +|---|---|---| +| A | 0, 0a, 1, 2 | `2026-11-17-one-table-a-foundation.md` | +| B | 3–7 | `2026-11-17-one-table-b-slow-columns.md` | +| C | 8a–8d | `2026-11-17-one-table-c-engine-config.md` | +| D | 9–11 | `2026-11-17-one-table-d-screen.md` | +| E | 12–14, 16 | `2026-11-17-one-table-e-configure.md` | +| F | 15, 17a–17d | `2026-11-17-one-table-f-display-and-acting.md` | +| G | 19–24 | `2026-11-17-one-table-g-editing.md` | + +## Cross-part decisions + +Where the parts, this index and the spec named one thing differently, this is the name every part uses. A part that disagrees with a line here is wrong. + +1. **Server options.** The spec's `RemoteServerOptions` is the code's `ServerConfig` (`core/remote.hpp`): `ServerConfig::tableWork` (Task 2), `ServerConfig::views` (Task 6). +2. **`TableQuery::columns` and `QueryLimits::maxColumns` land in Task 1**, as Delivery step 1 says. Task 4 adds `TableQuery::keys`, `maxKeys` and the keyed reply only. +3. **`Page::{unloadedRows, undecided, awaiting, failed}` and `UnloadedRow` land in Task 3b**; Task 4 adds their generic-page members and `indices`. +4. **`effects::IChangeObserver`, `effects::AnnounceSink`, `table::changeKeys`, `table::rowIdOf` land in Task 5** (the level cache is an observer); Task 6 adds `announce` and `attach`. +5. **The answer path** is `table::detail::answerQuery(ServedTable const&, TableQuery const&, AnswerContext const& context = {})` in `description.hpp`. Task 2 defines it with an empty `struct AnswerContext {}`; Task 6 adds `ViewCache* views`. +6. **Where table work runs.** `table::TableRuntime{exec::IExecutor* work = nullptr;}`, `table::currentTableRuntime() -> TableRuntime const*` and the RAII `table::TableRuntimeScope`, installed by `RemoteServer` around a dispatch, are Task 2's (no `exec::detail::tlsTableWork`). Task 6 adds `ViewCache* views` to it; `QueryTable::invoke` builds `AnswerContext` from it; Task 17a's `resolve` uses it. +7. **The column layout's value** is Task 8d's `table::ColumnLayoutValue{order, hidden, pinned, widths, groups, units}`. +8. **`PageWindow`'s accessors** are Task 7's: `key(i)`, `cell(i, column)`, `holds(i, column)`, `revision()`. `WindowRows` reads through them. +9. **The cell formatter** is `ui::formatCell(table::Cell const&, table::TableColumn const&, ui::FormatContext const&, DisplayUnits const& = {}) -> std::string` (Task 15). There is no `Locale` type; `ui::FormatContext` (`ui/expression.hpp`) carries numbers, date patterns and zone. Until Task 15, Task 9's stand-in is `ui::detail::standInCellText(table::Cell const&, table::TableColumn const&)`. +10. **Row states** are Task 17c's `ui::RowStates` reached by `TableRows::states()`, with `isBusy(key)`, `failure(key)`, `busyCount()`, and `TableRows::moveKey(from, to)`; locks through `ui::lockText(row, description)` and `ui::refusesEdit(rows, index, description)`. Task 24 uses these and declares no `busy`, `lockedBy` or `onKeyMoved`. +11. **The status line** is Task 9d's (`TableStatus{counts, state, retry, message, shown}`, `TableWidget::setStatus`, `setOnRetry`, `Table::status`), and so is its message channel `ScreenCore::tableMessage(ElementState&, std::string)` with its 6 s hold (`kStatusMessageHold`) and the `status: false` routing to notifications. Task 12 and later tasks post through it; Task 17a adds the overload taking entries. +12. **Copy** goes through `IViewBackend::copyText(std::string_view)` (the spec's `Backend::copyText`), Task 12b. +13. **Range selection in server mode** (`maxRangeRows`, the view-row report) is Task 10a's; Task 12f tests copy over it at its bound. +14. **`ui::TableColumn::id`, `TableArrangement`, a column part's `kind`, the `status` and `persist` keys** are Task 9's and 11b's, not Task 12's. +15. **"One order on every surface"** is tested for `rows` and model-backed tables in Task 11b; Task 13 extends the same test with a collection. +16. **Review Focus 1** is tested at Task 9 with what exists then (sort, filter, fetched columns); Task 17a extends it to a view selection and Task 24 to a cell draft. +17. **Order inside Task 8:** 8a → 8b → 8d (8b moves `ColumnGroupInfo` into `data_source.hpp`, which 8d's `ColumnLayout` uses); 8c needs Tasks 1, 2 and 5 and may land after 8b in either order with 8d. +18. **Build configurations.** Unit and interpreter tests are in `morph_tests`; Qt Quick and TUI tests need `cmake --preset cl-qt-debug -DMORPH_BUILD_QT_QUICK=ON -DMORPH_BUILD_TUI=ON`; `morph_net_tests` is POSIX-only and runs on `linux-everything`. +19. **Status-line entries and standing states.** Task 17a adds `ui::StatusEntry{std::string id; std::string label;}` (equality by value; handlers are held by the lowering, keyed by `id`) and `TableStatus::entries`, in `view.hpp` beside Task 9's `TableStatus`. Standing states — rows saving, rows that failed, retained rows that no longer match — are a persistent part of `TableStatus` derived from `RowStates::busyCount()`, `failedKeys()` and `counts().retained`, not posted through `tableMessage`, which carries transient refusals only; `tableMessage` has an overload taking entries. +20. **Test helpers** for table screens live in `tests/ui_table_support.hpp`. The first task that needs a helper adds it; every later task reuses it by its name and adds no second helper for the same thing. Task 9 adds `Harness`, `tableCatalog`, `replyDescribe`, `replyPage`, `lastQuery`, `statusOf`; Task 12 the menu and column drivers (`menuOf`, `clickHeader`, `clickFilter`, `moveColumn`, `clipboardText`); Task 17a `element`, `fireEvent`, `runMutation`, `settleMutation`, `refuseMutation`, `cellText`, `rowMarks`, `selectAll`, `rightClickCell`. +21. **`EngineRows`** is constructed as Task 9 declares it (`EngineRows(std::shared_ptr, RecordOf, table::EngineOptions)`); grids build it over a `std::shared_ptr` to their source. +22. **Measures outside the engine.** `ValueRowsSource` reads a `{num, den, unit}` object as a `Measure` (Task 11a). A stored display unit that names a unit the registry no longer has is dropped when the layout is applied to a description (Task 12c, which has the registry; `ColumnLayout` has none). +23. **Calling a table function.** Table functions are non-const, as the spec declares them. Task 2 provides the one way to get a table's `ServedTable` from its model: `table::detail::serve(Model& model, TableContext const& context) -> ServedTable`, which handles both a plain function and a `typed` registration. `QueryTable::invoke`, `DescribeTable::invoke` and Task 17a's `resolve` all call it, on the strand; `resolve` takes `Model&`, a stated deviation from the spec's `Model const&`. +24. **Run ids** used for busy rows and held refetches are session-wide (one counter per `Session`), not per `MutationRuntime`. +25. **`TableQuery::retry`** (`bool retry = false`) lands in Task 4, with its wire round-trip; Task 5 passes it to every want it causes, order and display. +26. **A user-formed column group** (a layout group, not a declared one) has no values list: its menu offers conditions and the match line only, since `DistinctRequest::group` names a declared group. +27. **The clipboard driver** is `RecordingBackend::clipboardText()` (the last text copied), added by Task 12b. +28. **Cells are a mount concept from Task 9d.** `TableSource` gives a per-cell view (`cellView(column, rowSignal) -> Node`) and a row-level view, and names its always-fetched columns; a pooled cell (Task 10a) owns its scope and a row signal and is rebound by setting it. `RecordOf` returns an owning row record, memoised by `EngineRows`. +29. **A cell's load state** (`Loaded`, `Unloaded`, `Failed`, from `TableRows::unloadedLevels`) reaches the mount and both backends in Task 9d/10b: a fixed-width placeholder for unloaded, an error mark for failed. + ## Review Focus 1. **A description that changes revision while state is held.** A view selection, retained rows, a focused cell or a cell draft held when the description takes a new revision (a column removed): the client refetches the description, fits its layout, drops what names a removed column, and never sends a query naming it. Test goes to Task 9 ("revision mid-selection"). @@ -105,7 +153,7 @@ ctest --test-dir build/cl-debug -R "" --output-on-failure **Spec:** "A model's tables", "The table's record", "Typed rows are a convenience". **Files:** -- Modify: `include/morph/table/description.hpp` (`TableContext`, `ServedTable`, `ChangedRow`, `ChangedRows`, `TableOf`, `table::typed`, `table::scoped`, `table::rowScoped`, `table::derivedFrom`), `include/morph/core/registry.hpp` and `include/morph/core/effects.hpp` (`BRIDGE_REGISTER_TABLE`, `effects::updatesTable`, `effects::changesTable`, the result walk's slot for `ChangedRows`), `include/morph/core/remote.hpp` (`RemoteServerOptions::tableWork`) +- Modify: `include/morph/table/description.hpp` (`TableContext`, `ServedTable`, `ChangedRow`, `ChangedRows`, `TableOf`, `table::typed`, `table::scoped`, `table::rowScoped`, `table::derivedFrom`), `include/morph/core/registry.hpp` and `include/morph/core/effects.hpp` (`BRIDGE_REGISTER_TABLE`, `effects::updatesTable`, `effects::changesTable`, the result walk's slot for `ChangedRows`), `include/morph/core/remote.hpp` (`ServerConfig::tableWork`, `table::TableRuntime`), `include/morph/table/query.hpp` (`TableQuery::columns`, `QueryLimits::maxColumns` — Task 1) - Modify: `tests/table/wide_table_model.*` (`bench::WideTableModel` with tables `wide`, `small`, `scoped`; actions `EditWideRow`, `ReplaceWideRows`) - Create: `tests/net/test_push_tables.cpp` (in `morph_net_tests`) - Docs: `docs/spec/core/registry.md`, `docs/spec/core/backend.md` ("Fan-out") per table.md's Neighbouring specs, step 1. @@ -137,7 +185,7 @@ ctest --test-dir build/cl-debug -R "" --output-on-failure **Files:** `include/morph/table/query.hpp`, `description.hpp` (generic page members); tests in `test_table_query.cpp`, `test_table_description.cpp`. **Interfaces:** -- Produces: `TableQuery::{columns, keys}`, `Page::{unloadedRows, undecided, awaiting, failed, indices}`, `UnloadedRow`, `UnloadedLevels`, the same on `GenericPage`; `validate` bounds per the spec. +- Produces: `TableQuery::keys`, `QueryLimits::maxKeys`, `Page::indices`, `UnloadedLevels`, and on `GenericPage` the members Task 3b put on `Page`; `validate` bounds per the spec. - [ ] Tests: spec row "`TableQuery` on the wire"; a keyed reply carries the view's totals and each row's view row. Gates W3, W4. - [ ] Commit: `feat(table): partial pages and keyed queries`. @@ -227,7 +275,7 @@ Four commits, one per subsystem, each its own reviewer gate. **Spec:** "Configuring a table" (columns, sort chains, filters from the header, column groups' menu, copy), "Keyboard, focus and assistive technology" (range selection in server mode). -**Files:** `include/morph/ui/interpreter.hpp` (`layout`, `onLayoutChange`, `configurable`, `locked`, `status`, `persist`, a column's `kind`), the Qt Quick and terminal backends (header and table menus, drag, resize, the sort dialog, the column chooser, the filter menu, the status line), `Backend::copyText`; tests `tests/test_ui_table_model.cpp`, `tests/qt_quick/`. +**Files:** `include/morph/ui/interpreter.hpp` (`layout`, `onLayoutChange`, `configurable`, `locked`), the Qt Quick and terminal backends (header and table menus, drag, resize, the sort dialog, the column chooser, the filter menu, the status line), `IViewBackend::copyText`; tests `tests/test_ui_table_model.cpp`, `tests/qt_quick/`. - [ ] Tests: spec rows "Sort chains", "Copy", "The status line" (messages and refusals), the menu halves of "Column layout", "Distinct values" and "Column groups". Gates D7, D9, B3, B4. Screenshot baselines of existing screens change here, as intended; regenerate them in this commit. - [ ] Commit: `feat(ui): configure a table from its header`. @@ -254,7 +302,7 @@ Four commits, one per subsystem, each its own reviewer gate. **Spec:** "How a column shows its cells". -**Files:** `include/morph/table/description.hpp` (`ColumnDisplay`, `Rounding{HalfEven, HalfUp}`, `ValueLabel`; `TableColumn::{display, tone, tooltip}`; `TableDescription::{rowTone, rowLock}`), create `include/morph/ui/cell_format.hpp` (`ui::formatCell(Cell const&, TableColumn const&, Locale const&) -> std::string`), the mount and backends (tone surfaces, tooltips, the "more here" mark, header units); tests `tests/test_ui_cell_format.cpp`, `test_table_description.cpp`, `tests/test_ui_table_model.cpp`. +**Files:** `include/morph/table/description.hpp` (`ColumnDisplay`, `Rounding{HalfEven, HalfUp}`, `ValueLabel`; `TableColumn::{display, tone, tooltip}`; `TableDescription::{rowTone, rowLock}`), create `include/morph/ui/cell_format.hpp` (`ui::formatCell(Cell const&, TableColumn const&, FormatContext const&, DisplayUnits const& = {}) -> std::string`), the mount and backends (tone surfaces, tooltips, the "more here" mark, header units); tests `tests/test_ui_cell_format.cpp`, `test_table_description.cpp`, `tests/test_ui_table_model.cpp`. **Interfaces:** - Consumes: `ui::locale_table` (number and date facts per locale). @@ -362,7 +410,7 @@ Not a task until a figure justifies it (table.md step 10: repairing a cached vie Tasks run in the order above, except: - Task 0a has no dependency and may run first or beside Task 0. -- Tasks 8a to 8d depend only on Task 3 and may run in parallel lanes; they share `filter.hpp`, so they merge one at a time. +- Task 8 runs 8a, 8b, then 8d; 8c needs Tasks 1, 2 and 5 and follows 8b. They share `filter.hpp`, so they merge one at a time. - Task 15 may land before Task 9 (then Task 9 uses `formatCell` directly). - Task 16 depends only on Task 12. - Phase B starts after Tasks 9, 11 and 15 (Task 19), and Task 24 waits for Task 17.