diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 7ae494d1f..328e51fb1 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -16,6 +16,8 @@ This page tracks significant updates to the QuestDB documentation. ### New +- [Audited views](/docs/security/audited-views/) - QuestDB Enterprise views created `WITH AUDIT` record every read in `sys.view_audit`, with the principal, the time, and the resolved values of the view's `AUDITED` parameters, covering the `params` JSON format, what counts as a read, audited views read through other audited views, the `AUDIT VIEW` permission, replication, and limitations, plus the [`view.audit.*` settings](/docs/configuration/audited-views/) +- [Declared value lists](/docs/query/sql/declare/#value-lists) - `DECLARE @symbols := ('BTC-USDT', 'ETH-USDT')` names the values of an `IN` filter once, including lists of bind variables and list parameters in views - [Memory limits](/docs/configuration/cairo-engine/#memory-limits) - New section covering the per-query, materialized view refresh, WAL apply, and live view refresh memory limits, what counts toward them, and what happens on a breach, plus the previously undocumented [`cairo.mat.view.max.refresh.retries`](/docs/configuration/materialized-views/#cairomatviewmaxrefreshretries), [`cairo.mat.view.refresh.busy.retry.limit`](/docs/configuration/materialized-views/#cairomatviewrefreshbusyretrylimit), [`cairo.mat.view.refresh.busy.retry.timeout`](/docs/configuration/materialized-views/#cairomatviewrefreshbusyretrytimeout), [`cairo.write.back.off.timeout.on.mem.pressure`](/docs/configuration/cairo-engine/#cairowritebackofftimeoutonmempressure), [`ram.usage.limit.bytes`](/docs/configuration/cairo-engine/#ramusagelimitbytes), and [`ram.usage.limit.percent`](/docs/configuration/cairo-engine/#ramusagelimitpercent) keys - [RBAC memory limits](/docs/security/rbac/#memory-limits) - Per-user, per-group, and per-service-account query memory limits in QuestDB Enterprise: `SET MEMORY LIMIT` on `ALTER USER`, `ALTER GROUP`, and `ALTER SERVICE ACCOUNT`, how limits resolve, the `SET MEMORY LIMIT` permission, and the upgrade migration - [ALTER GROUP](/docs/query/sql/acl/alter-group/) - New reference page covering `SET MEMORY LIMIT` and external alias mapping diff --git a/documentation/concepts/views.md b/documentation/concepts/views.md index 411b38ff6..0f89df0b8 100644 --- a/documentation/concepts/views.md +++ b/documentation/concepts/views.md @@ -186,6 +186,23 @@ CREATE VIEW mixed_params AS ( DECLARE @limit := 50 SELECT * FROM mixed_params ``` +### List parameters + +A parameter can hold a list of values for an `IN` filter. A caller can override +it with a list of any length: + +```questdb-sql +CREATE VIEW trades_for AS ( + DECLARE OVERRIDABLE @symbols := ('BTC-USDT', 'ETH-USDT') + SELECT timestamp, symbol, price FROM trades WHERE symbol IN @symbols +) + +-- A list of one needs the trailing comma +DECLARE @symbols := ('SOL-USDT',) SELECT * FROM trades_for +``` + +See [value lists](/docs/query/sql/declare/#value-lists) for the rules. + ## View hierarchies Views can reference other views, tables, and materialized views: @@ -442,6 +459,23 @@ GRANT SELECT ON desk_a_trades TO desk_a_users; For more details on permissions, see [Role-Based Access Control (RBAC)](/docs/security/rbac/). +### Audited views (Enterprise) + +A view created `WITH AUDIT` records every read of it in the `sys.view_audit` +table: who read it, when, and the values its `AUDITED` parameters resolved to, +including a caller's overrides and bind variables. Use it to keep an audit +trail of access to sensitive data: + +```questdb-sql +CREATE VIEW trades_by_symbol AS ( + DECLARE OVERRIDABLE AUDITED @symbols := ('BTC-USDT', 'ETH-USDT') + SELECT timestamp, symbol, price, amount FROM trades WHERE symbol IN @symbols +) WITH AUDIT; +``` + +See [Audited views](/docs/security/audited-views/) for what a read records, the +audit table, permissions, and limitations. + ## Performance considerations ### Views don't cache results @@ -496,3 +530,4 @@ EXPLAIN SELECT * FROM my_view WHERE symbol = 'AAPL' - [Materialized Views](/docs/concepts/materialized-views/): Incrementally maintained `SAMPLE BY` aggregates - [Live views](/docs/concepts/live-views/): Incrementally maintained row-per-input window-function results - [DECLARE](/docs/query/sql/declare/): Parameter declaration for views + - [Audited views](/docs/security/audited-views/): Record every read of a view (Enterprise) diff --git a/documentation/configuration/audited-views.md b/documentation/configuration/audited-views.md new file mode 100644 index 000000000..e4372dbaa --- /dev/null +++ b/documentation/configuration/audited-views.md @@ -0,0 +1,53 @@ +--- +title: Audited views +description: Configuration settings for audited views in QuestDB Enterprise. +--- + +:::note + +Audited views are [Enterprise](/enterprise/) only. + +::: + +An audited view records each read of it in the `sys.view_audit` table. These +settings control the in-memory queue that carries rows from the reading query +to the background job that writes them, and the storage policy the table is +created with. + +For details, see [Audited views](/docs/security/audited-views/). + +## view.audit.queue.capacity + +- **Default**: `4096` +- **Reloadable**: no + +Number of audit rows the queue holds between the queries that read audited +views and the job that writes them to `sys.view_audit`. The value is rounded up +to a power of two, and the queue is allocated on the heap at startup. + +Recording never makes a read wait. When the queue is full, the read still runs, +its row is dropped, and the server logs `view audit queue is full, dropping +rows`. Raise the capacity if that message appears during bursts of audited +reads. Auditing is lossy by design: see +[Delivery](/docs/security/audited-views/#delivery) for every case in which a +read goes unrecorded. + +## view.audit.storage.policy + +- **Default**: `TO PARQUET 1d` +- **Reloadable**: no + +[Storage policy](/docs/concepts/storage-policy/) that `sys.view_audit` is +created with. The default converts each daily partition to Parquet one day +after the partition ends. An audit trail is append-only and read cold, and +every partition converts once whatever the threshold, so a longer one keeps +native files around without saving any work. + +The setting applies only when the server creates the table at startup, which +a primary does and a replica does not: a replica takes the table, and its +policy, from the primary. It does not change the policy of a table that already +exists: use +[`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/) +for that. Set the property to an empty value to create the table with no +storage policy. If the server rejects the policy, it logs the error and creates +the table without one. diff --git a/documentation/configuration/overview.md b/documentation/configuration/overview.md index e0b9eb159..b5475ebc9 100644 --- a/documentation/configuration/overview.md +++ b/documentation/configuration/overview.md @@ -528,6 +528,7 @@ http.net.connection.sndbuf=2m | Section | Description                                                                                  | Enterprise only | |---------|-------------|:----------:| +| [Audited views](/docs/configuration/audited-views/) | Audit trail of view reads | ✓ | | [Cairo engine](/docs/configuration/cairo-engine/) | SQL engine settings | | | [Cold storage](/docs/configuration/cold-storage/) | Historical partitions on object storage | ✓ | | [COPY settings](/docs/configuration/copy-settings/) | CSV import and Parquet export | | diff --git a/documentation/query/sql/alter-view.md b/documentation/query/sql/alter-view.md index 982418ff9..75185f9f2 100644 --- a/documentation/query/sql/alter-view.md +++ b/documentation/query/sql/alter-view.md @@ -111,6 +111,7 @@ ALTER VIEW trades_filtered AS ( | `Invalid column` | Column in new query doesn't exist | | `circular dependency detected` | New definition would create circular reference | | `Access denied [ALTER VIEW on view_name]` | User lacks `ALTER VIEW` permission (Enterprise) | +| `Access denied [AUDIT VIEW]` | The view is audited and the user lacks the `AUDIT VIEW` permission (Enterprise) | | `Access denied [SELECT on table_name]` | User lacks SELECT on tables in new definition (Enterprise) | ## Behavior @@ -120,6 +121,8 @@ ALTER VIEW trades_filtered AS ( - Dependent views may become invalid if the altered view's output changes - Use `CREATE OR REPLACE VIEW` as an alternative if you want to create the view when it doesn't exist +- An [audited view](/docs/security/audited-views/) stays audited, and altering + one also requires the database-level `AUDIT VIEW` permission (Enterprise) ### Definer permissions transfer (Enterprise) diff --git a/documentation/query/sql/create-view.md b/documentation/query/sql/create-view.md index 33eacf2c6..38f69387d 100644 --- a/documentation/query/sql/create-view.md +++ b/documentation/query/sql/create-view.md @@ -14,6 +14,7 @@ documentation. ```questdb-sql CREATE [ OR REPLACE ] VIEW [ IF NOT EXISTS ] view_name AS ( query ) + [ WITH AUDIT ] [ OWNED BY owner_name ] ``` ## Parameters @@ -24,6 +25,10 @@ CREATE [ OR REPLACE ] VIEW [ IF NOT EXISTS ] view_name AS ( query ) | `OR REPLACE` | Replaces existing view or creates new one | | `view_name` | Name of the view (case-insensitive, Unicode supported) | | `query` | SELECT statement defining the view | +| `WITH AUDIT` | Enterprise only. Records every read of the view. See [WITH AUDIT](#with-audit-enterprise) | +| `OWNED BY` | Enterprise only. Assigns the view's owner. See [OWNED BY](#owned-by-enterprise) | + +`WITH AUDIT` and `OWNED BY` may appear in either order. ## Examples @@ -227,6 +232,27 @@ CREATE VIEW trades_summary AS ( OWNED BY 'analysts'; ``` +## WITH AUDIT (Enterprise) + +`WITH AUDIT` makes the view an [audited view](/docs/security/audited-views/): +every read of it records a row in `sys.view_audit`, with the principal, the +time, and the resolved values of the variables the view declares `AUDITED`. + +```questdb-sql title="Create an audited view" +CREATE VIEW trades_by_symbol AS ( + DECLARE OVERRIDABLE AUDITED @symbols := ('BTC-USDT', 'ETH-USDT') + SELECT timestamp, symbol, price, amount + FROM trades + WHERE symbol IN @symbols +) WITH AUDIT; +``` + +Creating a view `WITH AUDIT` requires the `AUDIT VIEW` permission in addition +to `CREATE VIEW`. The view stays audited through `ALTER VIEW` and +`CREATE OR REPLACE VIEW`, which do not accept `WITH AUDIT` for an existing +view, and which also require `AUDIT VIEW` over an audited one. To audit an +existing view, drop it and create it again. + ## See also - [Views concept](/docs/concepts/views/) @@ -234,3 +260,4 @@ OWNED BY 'analysts'; - [DROP VIEW](/docs/query/sql/drop-view/) - [COMPILE VIEW](/docs/query/sql/compile-view/) - [DECLARE](/docs/query/sql/declare/) +- [Audited views](/docs/security/audited-views/) diff --git a/documentation/query/sql/declare.md b/documentation/query/sql/declare.md index e74316881..8f1d6b626 100644 --- a/documentation/query/sql/declare.md +++ b/documentation/query/sql/declare.md @@ -16,19 +16,32 @@ DECLARE @variable := expression [, @variable := expression ...] SELECT ... ``` -```questdb-sql title="Inside a view definition (with optional OVERRIDABLE)" -DECLARE [OVERRIDABLE] @variable := expression - [, [OVERRIDABLE] @variable := expression ...] +```questdb-sql title="Inside a view definition (with optional OVERRIDABLE and AUDITED)" +DECLARE [OVERRIDABLE] [AUDITED] @variable := expression + [, [OVERRIDABLE] [AUDITED] @variable := expression ...] [WITH ...] SELECT ... ``` +```questdb-sql title="Value list, for the right-hand side of IN" +DECLARE @variable := ( value [, value ...] [,] ) +``` + The `OVERRIDABLE` keyword only takes effect inside a [view definition](/docs/query/sql/create-view/#declare-with-overridable). It marks a variable as a parameter that the caller of the view can override at query time. Variables without `OVERRIDABLE` use the value set in the view and cannot be changed by the caller. +The `AUDITED` keyword only takes effect inside the definition of an +[audited view](/docs/security/audited-views/), a QuestDB Enterprise feature. It +marks a variable whose resolved value each read of the view records in the +audit trail. It is independent of `OVERRIDABLE`, and the two may appear in +either order. Elsewhere it is accepted and has no effect. + +A [value list](#value-lists) declares the set of values an `IN` filter tests +against. + ## Mechanics The `DECLARE` keyword comes before the `SELECT` clause in your query: @@ -190,6 +203,66 @@ FROM second; | 10 | 9 | +### Value lists + +A parenthesised, comma-separated list declares the values of an `IN` filter +once, so that a query or a view can name the set instead of spelling it out: + +```questdb-sql title="Declare the values of an IN filter" +DECLARE @symbols := ('BTC-USDT', 'ETH-USDT') +SELECT timestamp, symbol, price, amount +FROM trades +WHERE symbol IN @symbols AND timestamp IN '$now-1h..$now'; +``` + +The list is expanded into the `IN` when the query is parsed, so the query above +is exactly `symbol IN ('BTC-USDT', 'ETH-USDT')`. Each member keeps its own type, +and every form of `IN` works as it does with a written-out list, including +`NOT IN` and interval scans on the designated timestamp. + +- `IN @symbols` and `IN (@symbols)` are equivalent. +- A list mixes with literals: `symbol IN ('SOL-USDT', @symbols)`. +- One list variable can be assigned to another: `@majors := @symbols`. +- A list of one needs a trailing comma, `('BTC-USDT',)`. Without it, + `('BTC-USDT')` is a parenthesised value. A trailing comma is also accepted + after the last member of a longer list. +- A bracketed sub-query, `(SELECT ...)` or `(DECLARE ... SELECT ...)`, is a + sub-query and not a list. + +Members can be bind variables, which lets one prepared statement filter on a +different set of values each time: + +```questdb-sql title="A list of bind variables" +DECLARE @symbols := ($1, $2) +SELECT timestamp, symbol, price FROM trades WHERE symbol IN @symbols; +``` + +In a [view](/docs/concepts/views/#parameterized-views), an `OVERRIDABLE` list +can be overridden with a list of a different length: + +```questdb-sql title="A list parameter in a view" +CREATE VIEW trades_for AS ( + DECLARE OVERRIDABLE @symbols := ('BTC-USDT', 'ETH-USDT') + SELECT timestamp, symbol, price FROM trades WHERE symbol IN @symbols +); + +DECLARE @symbols := ('SOL-USDT',) SELECT * FROM trades_for; +``` + +A list has no value of its own, so it can only be used on the right-hand side +of `IN`. These fail: + +| Query | Error | +| ------------------------------------------------------ | --------------------------------------------------------- | +| `SELECT @symbols`, `WHERE symbol = @symbols` | `declared list can only be used on the right-hand side of IN` | +| `OVER (PARTITION BY @symbols)` | `declared list can only be used on the right-hand side of IN` | +| `@all := (@symbols, 'SOL-USDT')` | `declared list can only be used on the right-hand side of IN` | +| `@x := ('BTC-USDT', ('ETH-USDT', 'SOL-USDT'))` | `nested lists are not supported` | +| `@x := ()` | `value expected in list` | + +To combine a list with more values, write both in the `IN`: +`symbol IN (@symbols, 'SOL-USDT')`. + ### Bind variables `DECLARE` syntax will work with prepared statements over PG Wire, so long as the client library @@ -241,18 +314,6 @@ how many places you need to update the constant. However, not all expressions are supported. The following are explicitly disallowed: -#### Bracket lists - -```questdb-sql title="bracket lists are not allowed" -DECLARE - @symbols := ('BTC-USDT', 'ETH-USDT') -SELECT timestamp, price, symbol -FROM trades -WHERE symbol IN @symbols; - --- error: unexpected bind expression - bracket lists not supported -``` - #### SQL statement fragments ```questdb-sql title="sql fragments are not allowed" diff --git a/documentation/query/sql/drop-view.md b/documentation/query/sql/drop-view.md index 95e52b9c0..ac94ee561 100644 --- a/documentation/query/sql/drop-view.md +++ b/documentation/query/sql/drop-view.md @@ -97,6 +97,10 @@ GRANT DROP VIEW ON view1, view2 TO username; When a user creates a view, they are automatically granted all permissions including `DROP VIEW` on that view. +Dropping an [audited view](/docs/security/audited-views/) also requires the +database-level `AUDIT VIEW` permission, because dropping is the only way a view +stops being audited. + ## See also - [Views concept](/docs/concepts/views/) diff --git a/documentation/security/audited-views.md b/documentation/security/audited-views.md new file mode 100644 index 000000000..81a76112b --- /dev/null +++ b/documentation/security/audited-views.md @@ -0,0 +1,413 @@ +--- +title: Audited views +sidebar_label: Audited views +description: + Record every read of a QuestDB Enterprise view in sys.view_audit, with the + principal, the time, and the parameter values each read resolved to. +--- + +import { EnterpriseNote } from "@site/src/components/EnterpriseNote" + + + Record who read a view, when, and with which parameters. + + +An audited view records every read of it in the `sys.view_audit` table: the +principal that ran the query, when the read finished, how long it took, and the +values its parameters resolved to. Use it where you have to answer who looked +at which data, for example a view that exposes trades or positions for a symbol +and time range that the caller chooses. + +A view created `WITH AUDIT` records one row per read. Each of its +[`DECLARE`](/docs/query/sql/declare/) variables marked `AUDITED` adds its +resolved value to the row's `params` column, as JSON. A caller's override, a +bind variable, or a default built on `now()` is recorded as the value that read +actually ran with, so the trail describes the data each read covered. + +## Quick start + +```questdb-sql title="Create an audited view" +CREATE VIEW trades_by_symbol AS ( + DECLARE + OVERRIDABLE AUDITED @symbols := ('BTC-USDT', 'ETH-USDT'), + AUDITED @since := dateadd('h', -1, now()) + SELECT timestamp, symbol, side, price, amount + FROM trades + WHERE symbol IN @symbols AND timestamp >= @since +) WITH AUDIT; +``` + +```questdb-sql title="Read it, overriding the symbols" +DECLARE @symbols := ('SOL-USDT',) +SELECT * FROM trades_by_symbol; +``` + +```questdb-sql title="Inspect the trail" +SELECT ts, principal, view_name, params, status +FROM 'sys.view_audit' +WHERE view_name = 'trades_by_symbol' +ORDER BY ts DESC +LIMIT 10; +``` + +| ts | principal | view_name | params | status | +| --------------------------- | --------- | ---------------- | ------------------------------------------------------------- | ------ | +| 2026-09-19T10:15:02.418331Z | analyst | trades_by_symbol | `{"since":"2026-09-19T09:15:02.417950Z","symbols":["SOL-USDT"]}` | ok | + +Rows are written in the background and appear in the table shortly after the +read. + +## Syntax + +```questdb-sql title="Create an audited view" +CREATE VIEW [ IF NOT EXISTS ] viewName AS ( query ) + WITH AUDIT [ OWNED BY ownerName ] +``` + +```questdb-sql title="Mark the parameters to record" +DECLARE [ OVERRIDABLE ] [ AUDITED ] @variable := expression + [, [ OVERRIDABLE ] [ AUDITED ] @variable := expression ...] +``` + +- `WITH AUDIT` and `OWNED BY` may appear in either order. `WITH AUDIT` requires + the [`AUDIT VIEW`](#permissions) permission. +- `AUDITED` and `OVERRIDABLE` are independent and may appear in either order. + `AUDITED` takes effect only in the body of a view created `WITH AUDIT`. + Elsewhere it is accepted and has no effect. +- A view created `WITH AUDIT` with no `AUDITED` variables still records every + read, with `{}` in `params`. + +## Choose what a read records + +Mark a variable `AUDITED` when its value describes which data a read covered: + +- **`OVERRIDABLE AUDITED`**: a parameter the caller can set. The row records the + caller's value, or the view's default when the caller sets nothing. +- **`AUDITED` only**: a parameter the caller cannot change. Worth recording when + its value changes between reads, such as a window built on `now()`, because + the trail is otherwise the only place that value is kept. + +Only the view's own declarations count. A caller cannot add a parameter to the +row, or remove one, by declaring variables of their own, `AUDITED` or not. + +Values are evaluated for each execution. A prepared statement that binds a +view parameter to a bind variable records the values bound on each execution: + +```questdb-sql title="One plan, a row per execution with its own values" +DECLARE @symbols := ($1, $2) +SELECT * FROM trades_by_symbol; +``` + +:::tip + +Record resolved values rather than text that resolves later. A +[TICK](/docs/query/operators/tick/) string such as `'$now-1h..$now'` is recorded +as that text, not as the time range it resolved to. To keep the range in the +trail, declare the bounds as timestamps, as `@since` does above. + +::: + +## The audit table + +The server creates `sys.view_audit` at startup. The `sys.` prefix follows +[`cairo.system.table.prefix`](/docs/configuration/cairo-engine/#cairosystemtableprefix). +It is a WAL table partitioned by day, with this schema: + +| Column | Type | Description | +| ---------------- | ----------- | ------------------------------------------------------------------------------------------------------------ | +| `ts` | `TIMESTAMP` | When the read finished and the row was recorded. The designated timestamp. | +| `principal` | `SYMBOL` | The user or service account that ran the query. | +| `view_name` | `SYMBOL` | The name of the audited view. | +| `params` | `VARCHAR` | The resolved values of the view's `AUDITED` variables, as a JSON object. `NULL` when they could not be evaluated. | +| `latency_micros` | `LONG` | How long the read took, from opening it to closing it, in microseconds. | +| `status` | `SYMBOL` | `ok`, or `error` when the read failed or was cancelled. A failed read is recorded because it was attempted. | +| `view_id` | `INT` | The view's internal id. A view that is dropped and created again under the same name gets a new id. | + +Reads that stream page frames, such as Parquet export, record the row when the +read starts, so their `latency_micros` covers only opening the read. + +### The params column + +`params` is canonical JSON, so two reads with the same values produce the same +text and a report can group on the column directly: + +- Keys are the variable names without the `@`, sorted by name. +- A declared list renders as a JSON array, in the order it was written. Declare + a one-member list with a trailing comma, `('SOL-USDT',)`, so that the value + stays an array. Without the comma it is a scalar. +- A `NULL` value renders as JSON `null`. + +| SQL type | JSON | +| ------------------------------------------------- | ---------------------------------------------------------------------- | +| `BOOLEAN` | `true` or `false` | +| `BYTE`, `SHORT`, `INT`, `LONG`, `FLOAT`, `DOUBLE` | Number | +| `CHAR`, `STRING`, `SYMBOL`, `VARCHAR` | String | +| `TIMESTAMP` | ISO 8601 string, in the timestamp's own precision (micro or nanosecond) | +| `DATE` | ISO 8601 string, with microseconds | +| `UUID`, `IPv4` | String | + +### Values that cannot be audited + +A read fails, rather than record a row that misstates it, when an `AUDITED` +variable: + +- Resolves to a type the table above does not list, such as an array. The error + is `audited view parameter has a type that cannot be audited`. +- Holds a sub-query anywhere in its value, such as + `(SELECT max(timestamp) FROM trades)`. The error is the same, with the type + `CURSOR`. +- Can change while the query runs, such as `rnd_int()` or `systimestamp()`. The + error is `audited view parameter has a value that can change during the query`. + A value built on `now()` is fixed for the whole query, and is recorded. + +`CREATE VIEW ... WITH AUDIT`, `ALTER VIEW` and `CREATE OR REPLACE VIEW` check the +view's own declarations the same way, so a view that no read could audit is +refused when it is defined. + +A read also fails when an `AUDITED` value cannot be evaluated, for example when +a caller's override makes a function raise an error. This holds even when the +query never uses the variable, such as one that only feeds a column the query +does not select. The read returns the error without reading any data, and +records a row with status `error` and `NULL` in `params`. The trail cannot show +what such a read was given, so the read does not run. + +### Query the trail + +```questdb-sql title="Reads per principal and parameter set, today" +SELECT principal, view_name, params, count() AS reads +FROM 'sys.view_audit' +WHERE ts IN '$today' +GROUP BY principal, view_name, params +ORDER BY reads DESC; +``` + +```questdb-sql title="Extract one parameter" +SELECT ts, principal, json_extract(params, '$.since')::timestamp AS since +FROM 'sys.view_audit' +WHERE view_name = 'trades_by_symbol' AND ts IN '$today'; +``` + +See [`json_extract()`](/docs/query/functions/json/#json_extract) for the path +syntax. + +### Retention + +The table is created with the storage policy that +[`view.audit.storage.policy`](/docs/configuration/audited-views/#viewauditstoragepolicy) +sets, `TO PARQUET 1d` by default, so older partitions of the trail move to +Parquet. The setting applies only when the server creates the table. After +that, change the policy with +[`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/). + +The table cannot be dropped, so the trail cannot be erased by whoever holds +`DROP TABLE`. That holds for every route that drops a table: `DROP TABLE`, +`DROP ALL TABLES`, and a +[CSV import](/docs/connect/compatibility/rest-api/#imp---import-data) with +`overwrite=true`. It can be truncated with +[`TRUNCATE TABLE`](/docs/query/sql/truncate/), which keeps retention the +operator's to manage. + +`ALTER TABLE` cannot change the table's structure: adding, dropping, renaming +or retyping a column, and turning deduplication on or off, are refused. Each +node keeps its own table, so such a change would stay on the node that made it. +The storage policy, partition operations and `TRUNCATE TABLE` remain available. + +## What counts as a read + +A read is one execution of a statement that reads rows through an audited view: +`SELECT`, `INSERT INTO ... SELECT`, `CREATE TABLE AS SELECT`, or an `UPDATE` of a +non-WAL table that reads the view in its `FROM` clause or in a sub-query. Each +execution records its own rows, including every execution of a cached or +prepared statement. + +When one statement mentions the same view more than once, in a join, a union, +or a sub-query, the view records one row for each distinct set of parameter +values. References that resolve to the same values are one read and share a +row. Values are compared as rendered, so `1` and `1.0` count as different +values. + +These record nothing: + +- `CREATE VIEW`, `CREATE MATERIALIZED VIEW`, `ALTER VIEW` and + `CREATE OR REPLACE VIEW` whose query reads an audited view, when the check + passes. Each opens its query only to check it, and hands no rows to anyone. A + check that fails records a row with status `error`, like any failed read, + because its error message can carry values from the rows it read. +- Reads the database runs on its own behalf: materialized view refreshes and + WAL apply. No principal is reading data there. +- An `UPDATE` of a WAL table. See [Limitations](#limitations). + +## Audited views that read other audited views + +An audited view read inside the body of another audited view records no row of +its own when the outer view's row covers it. The outer view covers the inner +one when every variable the inner view declares `OVERRIDABLE AUDITED` is also +declared `AUDITED` in the outer view, by name, overridable or not. Those are the +only values a caller can change through the outer view, so the outer row then +shows everything the caller chose. + +An inner view with no `OVERRIDABLE AUDITED` variables is always covered. So an +audited view that unions several audited views with fixed or no audited +parameters records one row, for itself. + +An inner view that is not covered keeps its row, because a caller's value can +reach it through the outer view without appearing on the outer row: + +```questdb-sql title="The inner view has a parameter the caller can set" +CREATE VIEW symbol_trades AS ( + DECLARE OVERRIDABLE AUDITED @sym := 'BTC-USDT' + SELECT timestamp, symbol, side, price, amount + FROM trades + WHERE symbol = @sym +) WITH AUDIT; + +CREATE VIEW buy_trades AS ( + SELECT * FROM symbol_trades WHERE side = 'buy' +) WITH AUDIT; +``` + +```questdb-sql title="The caller's @sym passes through buy_trades" +DECLARE @sym := 'ETH-USDT' SELECT * FROM buy_trades; +``` + +| view_name | params | +| --------------- | -------------------- | +| `buy_trades` | `{}` | +| `symbol_trades` | `{"sym":"ETH-USDT"}` | + +To record one row, re-declare the parameter in the outer view. The view stays +audited through `ALTER VIEW`, which on an audited view also needs the +[`AUDIT VIEW`](#permissions) permission: + +```questdb-sql title="The outer view records @sym itself" +ALTER VIEW buy_trades AS ( + DECLARE OVERRIDABLE AUDITED @sym := 'BTC-USDT' + SELECT * FROM symbol_trades WHERE side = 'buy' +); +``` + +The same read now records one row, `buy_trades` with `{"sym":"ETH-USDT"}`. +Declare it `AUDITED` without `OVERRIDABLE` instead to fix the value for every +caller. + +The rule in full: + +- Coverage is checked against the outermost audited view around the read, + through any views between them, audited or not. +- A view that is not audited never covers another. An audited view read + through a plain view always records its row. +- A reference to the inner view outside the outer view, in the same statement, + records as usual. If it resolves the same values as an inner read that is not + covered, the two share one row. +- When an inner view is covered, its `AUDITED` variables that are not + `OVERRIDABLE` are not recorded. Read directly, the view records them. +- Coverage depends only on the view definitions, so a given statement always + records the same set of views. + +## Delivery + +Auditing is lossy by design, and there is no lossless mode. Recording a read +never makes the read wait, and a read is never refused because its row cannot +be recorded, except on a +[read-only instance](#replication-and-read-only-instances). Treat the trail as a +best-effort record: under overload, or when a write fails, a read can go +unrecorded. + +A read puts its row on a bounded in-memory queue, and a background job writes +the queue to the table. The queue holds +[`view.audit.queue.capacity`](/docs/configuration/audited-views/#viewauditqueuecapacity) +rows, 4096 by default. + +A read goes unrecorded when: + +- **The queue is full when the read finishes**, because audited reads outpace + the job. The server logs `view audit queue is full, dropping rows` with a + running total, on the first drop and every 1024th after that. Raise the + capacity if this appears during bursts of audited reads. +- **The job fails to write a batch.** The batch, up to 1024 rows, is lost, and + the server logs `could not write view audit rows`. +- **The audit table is missing a column, or a column has the wrong type.** The + server logs the reason once and discards audit rows. The server creates the + table with the right columns and SQL cannot change them, so this happens only + to a table created some other way. See [Retention](#retention). +- **The server stops or crashes with rows still in the queue.** The queue is + held in memory only. + +Lost rows show in the server log only. No metric reports them. + +## Permissions + +| Action | Permissions | +| ------------------------------------------------------------------- | --------------------------------------------------------- | +| Create a view `WITH AUDIT` | `CREATE VIEW` and `AUDIT VIEW` | +| Drop an audited view, with `DROP VIEW` or `DROP ALL TABLES` | `DROP VIEW` on the view and `AUDIT VIEW` | +| Change an audited view with `ALTER VIEW` or `CREATE OR REPLACE VIEW` | `ALTER VIEW` on the view and `AUDIT VIEW`. The view stays audited | +| Read an audited view | `SELECT` on the view, as for any [view](/docs/concepts/views/#definer-security-model-enterprise) | +| Read or truncate the trail | `SELECT` or `TRUNCATE TABLE` on `sys.view_audit` | + +`AUDIT VIEW` is a database-level permission, included in `ALL` and +`DATABASE ADMIN`. It guards every statement that binds a view to the trail, +changes what it records, or releases it: creating, redefining and dropping an +audited view. A new body can remove the view's `AUDITED` declarations, or keep +a variable `OVERRIDABLE` while no longer recording it, so without the +permission a principal who can alter a view, or drop and recreate it, could +shed its auditing unremarked. + +A view's auditing is set when it is created. `ALTER VIEW` and +`CREATE OR REPLACE VIEW` over an existing view keep it, and do not accept +`WITH AUDIT`. To audit an existing view, or to stop auditing one, drop it and +create it again. + +`sys.view_audit` takes ordinary table permissions. Grant `SELECT` on it to the +people who review the trail, and keep write permissions such as `INSERT`, +`UPDATE` and `TRUNCATE TABLE` to the operators who manage it. + +## Replication and read-only instances + +Every node records the reads it serves, replicas included, into its own +`sys.view_audit`. The rows are not replicated between nodes, so the complete +trail is the union of the tables on all nodes. + +A replica does not create `sys.view_audit` itself. The table's definition, and +its storage policy, reach the replica from the primary, and the replica writes +the reads it serves into that table. Until the definition arrives, the replica +keeps its audit rows in the queue, and a read whose row finds the queue full +goes unrecorded, as in [Delivery](#delivery). This matters only for a replica +that starts before its primary has created the table, such as when replicas +are upgraded first. + +An instance started with `readonly=true` cannot write the trail, so it refuses +reads of audited views with +`cannot read an audited view on a read-only instance`. Views that are not +audited are unaffected. + +## Limitations + +- **A materialized view over an audited view is not audited.** Its refreshes + record nothing, reads of the materialized view are not audited, and + `CREATE MATERIALIZED VIEW` records nothing either. A principal who can read an + audited view and create materialized views can therefore make its data + readable with no audit row. Grant `CREATE MATERIALIZED VIEW` with that in + mind. A [live view](/docs/concepts/live-views/) cannot be defined over a + view, so it cannot be used this way. +- **An `UPDATE` of a WAL table records nothing** when it reads an audited view. + Its read happens during WAL apply, on every node, rather than in the session + that submitted it. A WAL table's `UPDATE` can read no other table and use no + join, so the only such statement that compiles reads, in a sub-query, an + audited view whose query reads only the updated table. The same statement on + a non-WAL table is recorded. +- **Copies are recorded once.** `INSERT INTO ... SELECT` and + `CREATE TABLE AS SELECT` record the read that made the copy. Reads of the copy + are not audited. +- **The trail is best-effort.** Auditing is lossy, and there is no lossless + mode. A read goes unrecorded when the queue is full, when a write fails, or + when the server stops with rows still queued. See [Delivery](#delivery). + +## See also + +- [Views](/docs/concepts/views/) +- [CREATE VIEW](/docs/query/sql/create-view/) +- [DECLARE](/docs/query/sql/declare/) +- [Role-based access control](/docs/security/rbac/) +- [Audited views configuration](/docs/configuration/audited-views/) +- [Storage policy](/docs/concepts/storage-policy/) diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index 62dd03447..9b40c29bc 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -816,6 +816,7 @@ SELECT * FROM all_permissions(); | ALTER COLUMN CACHE | Database | Table | Column | Enable/disable symbol caching | | ALTER COLUMN TYPE | Database | Table | Column | Change column types | | ATTACH PARTITION | Database | Table | Attach partitions | +| AUDIT VIEW | Database | Create, alter and drop [audited views](/docs/security/audited-views/) | | BACKUP DATABASE | Database | Create database backups | | CANCEL ANY COPY | Database | Cancel COPY operations | | CREATE TABLE | Database | Create tables | diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 3b01a24ab..2b77de53a 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -687,6 +687,7 @@ module.exports = { type: "doc", label: "Overview", }, + "configuration/audited-views", "configuration/cairo-engine", "configuration/cold-storage", "configuration/copy-settings", @@ -722,6 +723,11 @@ module.exports = { type: "doc", label: "Role-Based Access Control (RBAC)", }, + { + id: "security/audited-views", + type: "doc", + label: "Audited views", + }, { id: "security/oidc", type: "doc",