From 30440914f2a6aaad645f5af0bb0ee6ff850cea86 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 28 Aug 2026 12:12:46 -0600 Subject: [PATCH] docs: object-storage tiering is rejected, not deferred (#403 item 5b) Item 5's TTL-to-volume example ages cold data to object storage. Its stated prerequisites, #393 and #394, are both closed, which is what made the item live. This is the answer to whether to build on them, and it is no. ## Why, and it is not difficulty The seam is small: 4 call sites read through PgColumnarReadLogicalData and 1 writes through PgColumnarWriteLogicalData, and the object-storage API already offers exactly the ranged reads a tier would need. The obstacle is what the bytes currently are. Every columnar page is written through the buffer manager and WAL-logged as a full-page image (log_newpage_buffer, on written pages and on gap pages alike). Five properties follow from that, and a tier keeps none of them: crash recovery, physical replication, self-contained backup, PITR, and rollback of an aborted rewrite. One of those five settles it. docs/limitations.md already records that logical decoding cannot carry columnar tables, because their data reaches WAL as full-page images with no tuple structure, and directs users to physical replication. Physical replication works precisely BECAUSE the bytes are in WAL. Tiering removes them, so a standby replays the WAL and finds nothing, and the only supported way to replicate a columnar table stops working for any table that uses the feature. Keeping replication would mean putting a record in the WAL that a standby acts on by fetching remotely: a new WAL semantic in a new record type, interpreted by a replay path this extension does not own. The project's standing constraint rejects that rather than deferring it, so this is recorded as rejected. ## What to do instead The user-facing goal is reachable today by composing what exists, with none of the five invariants touched: export the cold range with export_parquet or parallel_export_parquet, drop the local rows with pgcolumnar.expire (item 5a) or DELETE plus compact, and read it back through the external Parquet reader or iceberg_scan. The cost is that the cold data is no longer part of the same table, which the document states plainly rather than dressing up. Every fact in the document was checked against the code rather than recalled, including the call-site counts, which an earlier draft had as 5 and 2 by counting the definitions along with the calls. Also records the outcome of every #403 remaining item in the plan document, including the two that changed shape once measured. Refs #403. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UaQ4vThXcmTCf3KRQpUDrE --- design/ISSUE_403_REMAINING_ITEMS.md | 17 ++++ design/OBJECT_STORAGE_TIERING.md | 126 ++++++++++++++++++++++++++++ 2 files changed, 143 insertions(+) create mode 100644 design/OBJECT_STORAGE_TIERING.md diff --git a/design/ISSUE_403_REMAINING_ITEMS.md b/design/ISSUE_403_REMAINING_ITEMS.md index c09bcc8..f0118ff 100644 --- a/design/ISSUE_403_REMAINING_ITEMS.md +++ b/design/ISSUE_403_REMAINING_ITEMS.md @@ -45,6 +45,23 @@ Two structural facts that shape the ordering: manager. That is a real seam for tiering. It is also the reason tiering is not a small change: bytes outside the storage manager are outside crash recovery. +## Outcome, 2026-08-28 + +The plan is kept as written above, and this section records where each item +actually landed. Two changed shape once measured, which is what the phases were +ordered to find out early. + +| Item | Outcome | +| --- | --- | +| 6, hash table sizing | Merged, #810. Corrected on review: sizing at Begin allocated 131,072 entries for 47 real groups, so it now happens on the first grow and a grow is bounded by 64x the live count. | +| 4, filter ordering | Merged, #811. The defect was worse than the issue described: the order was ATTRIBUTE order, and writing the selective predicate first in the query did not change it. 200 zone-map probes to 102. | +| 3, set skipping index | **Not built. Measured and not justified.** On the shape #403 names, `os` and `arch` hold every distinct value in every row group, so a set index stores the whole domain and prunes nothing; the clumped columns are contiguous in sort order, where min/max is already exact. The `<>` pushdown proposed in its place was withdrawn after measurement too: the ceiling is freq(v), and sorting already achieves 90% of that bound. | +| 7, idempotent inserts | PR #814. Whole-load fingerprint, not part hashes, because 2PC makes the load atomic. Opt-in, as decided. | +| 5a, TTL | PR #815. `pgcolumnar.expire`, explicit rather than folded into a rewrite. | +| 5b, tiering | **Rejected, not deferred.** design/OBJECT_STORAGE_TIERING.md. It removes columnar tables from physical replication, which is their only supported replication path, and keeping replication would need a new WAL record type the extension may not add. | + +Phase 3, the alpha3 cycle, was opened and merged as #812, carrying #761. + ## Phases, in dependency order ### Phase 1: item 6, size the group hash table from runtime statistics diff --git a/design/OBJECT_STORAGE_TIERING.md b/design/OBJECT_STORAGE_TIERING.md new file mode 100644 index 0000000..604d482 --- /dev/null +++ b/design/OBJECT_STORAGE_TIERING.md @@ -0,0 +1,126 @@ +# Tiering columnar data to object storage: design and recommendation + +Issue #403 item 5 records ClickHouse's TTL-to-volume feature, which ages cold +data to an object-storage volume: + +```sql +TTL (ts + INTERVAL 1 WEEK) TO VOLUME 's3' +``` + +Its stated prerequisites, #393 (object storage reads) and #394 (object storage +writes), are both closed. This document is the answer to whether we should build +the feature on top of them. + +**Recommendation: do not build native tiering.** Build the same capability as a +workflow over the pieces that already exist. The reasoning is below, and it is +not primarily about difficulty. + +Written 2026-08-28. Item 5a, `pgcolumnar.expire`, is the other half of item 5 and +is implemented separately. + +## Audience and scope + +For anyone deciding whether to move columnar bytes out of the relation's main +fork. It covers native row-group data only. External Parquet and Iceberg reads +already live in object storage and are unaffected by everything here. + +## What exists today, checked against the code + +| Fact | Evidence | +| --- | --- | +| Native bytes live in the relation's main fork | `PgColumnarWriteLogicalData` writes through `ReadBufferExtended(rel, MAIN_FORKNUM, ...)` | +| There is one seam each way | 4 call sites read through `PgColumnarReadLogicalData` (all in `columnar_reader.c`), 1 writes through `PgColumnarWriteLogicalData` (`columnar_write_state.c:2755`) | +| Every page is WAL-logged as a full-page image | `log_newpage_buffer(buffer, true)` on each written page, and on each gap page | +| Object storage supports ranged reads | `PgColumnarObjStoreApi.read(h, off, buf, n)`, plus `open`, `close`, a multipart sink, `delete_object` and `list` | +| Physical replication is the supported path for columnar tables | `docs/limitations.md`: logical decoding reads heap tuple records, columnar data reaches WAL as full-page images carrying no tuple structure | + +The seam is genuinely small, and the object-storage API already offers ranged +reads, which is exactly the shape a reader needs. **Feasibility is not the +problem.** + +## The invariants a tier would have to preserve + +A row group's bytes are, today, ordinary relation data. Five properties follow +from that, and a tier has to keep all five or say which it abandons. + +1. **Crash recovery.** After a crash, replay of WAL restores every page. The + bytes are in the WAL stream as full-page images. +2. **Physical replication.** A standby has the data because it replays the same + WAL. This is the *only* supported replication path for columnar tables. +3. **Backup.** `pg_basebackup` copies the data directory and the WAL needed to + make it consistent. A backup is complete by construction. +4. **Point-in-time recovery.** Restoring to a past point restores the bytes as + they were at that point, because the WAL says what they were. +5. **Transactional rollback.** A rewrite that aborts leaves nothing behind that a + reader can see, because the pages it wrote are never committed. + +## Why tiering breaks them, and which break is fatal + +Moving a row group's bytes to an object store removes them from the main fork. +They are then not in WAL, and each invariant fails in turn: + +- **Crash recovery** no longer covers the data. This is the failure people expect + and it is the least serious, because an object store is itself durable. +- **Physical replication silently stops carrying the data.** This is the fatal + one. A standby replays WAL and finds no bytes. It cannot fetch them, because + nothing in the WAL stream says they exist. The extension's own documented + replication story is physical replication, so tiering removes the only + supported way to replicate a columnar table, for the tables that use it. +- **Backup** stops being self-contained. `pg_basebackup` produces a backup that + restores to a database referring to objects it does not contain. +- **PITR** becomes unsound in a way that cannot be fixed inside the extension. + Restoring to a point before an object was deleted requires the object store to + hold a version that the extension deleted, and object stores do not roll back. +- **Rollback** requires an aborted tiering rewrite to leave no reachable object, + which means the abort path has to delete remote objects it may have already + written, over a network that may be unreachable at that moment. + +The second one is what settles it. The others are engineering problems with +known, expensive answers. Losing physical replication is a change to what the +product is. + +## Why this is not deferrable + +The project's standing constraint is that the extension may use core WAL +mechanisms and existing record types only. A tier that kept replication working +would have to put *something* in the WAL that a standby can act on: a record +saying "this row group now lives at this URL", which the standby then reads +remotely. That is a new WAL semantic, carried in a new record type, interpreted +by a replay path this extension does not own. + +So this is not a large feature waiting for time. It is a design that the +extension's constraints reject. Recording it as **rejected** rather than +**deferred** is the honest disposition, and it is why this document exists +instead of a phase plan. + +## What to build instead + +The user-facing goal, "old data should stop occupying expensive local storage", +is achievable today by composing pieces that already exist, with none of the +invariants above touched: + +1. Export the cold range to Parquet in object storage, with + `pgcolumnar.export_parquet` or `pgcolumnar.parallel_export_parquet` (#394). +2. Drop the local rows, with `pgcolumnar.expire` (#403 item 5a, PR #815, not yet + merged) or an ordinary `DELETE` plus `pgcolumnar.compact`. +3. Read the exported data when it is needed, through the external Parquet reader + or `iceberg_scan` (#388, #393). + +The local relation stays entirely in its main fork, so crash recovery, physical +replication, backup, PITR and rollback are exactly what they were. What is given +up is that the cold data is no longer part of the same table: a query that spans +both has to union them. + +That is a real limitation and it should be stated plainly rather than dressed up. +It is also the difference between a feature this extension can support and one it +cannot. + +## What would change this answer + +One thing: core PostgreSQL gaining a way for an extension to participate in WAL +and replay for data it stores outside the main fork. If that arrives, the seam +identified here is still the right place to start, and this document's invariant +list is the specification the work would have to satisfy. + +Nothing about object storage, and nothing about the size of the change, is what +stands in the way.