Skip to content

Kafgres read - #24

Merged
RayElg merged 19 commits into
mainfrom
sql-read-range
Nov 18, 2026
Merged

RayElg merged 19 commits into
mainfrom
sql-read-range

Conversation

@RayElg

@RayElg RayElg commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Adds kafgres_read(), allows reading the logs directly from SQL backend. Read-side counterpart to kafgres_produce().

Call kafgres_read('topic', ...) to get one row per Kafka record

Also adds an incident response demo in demo/read

@RayElg
RayElg added this pull request to stack #25 October 8, 2026 21:53
Base automatically changed from refactor-dispatch to main October 9, 2026 02:09
@RayElg
RayElg force-pushed the sql-read-range branch 3 times, most recently from 910f89f to 0764dcf Compare October 14, 2026 22:20
RayElg added 14 commits November 2, 2026 23:01
…, and bump to 0.4.0 (I4, I8)

- kafgres_read(topic, partition, from_offset, to_offset, from_time, to_time, isolation,
  max_records): records as rows with key, value, headers, timestamp type, leader epoch,
  producer id/epoch, sequence and transactional flag. Composes with WHERE, joins and LATERAL.
- An engine-agnostic cursor over LogStore::read (I4) resolves offset and time bounds, reads
  1 MiB chunks, and yields records one at a time; a record over 65,536 headers is refused
  before decoding (I8). The read_committed abort filter lives in the codec.
- FetchSlice reports where the aborted-transaction list hit its cap, so a read never judges
  a batch past it. The wire Fetch handler does not use this yet.
- The table engine copies batches out of SPI's own context: pgrx's get::<Vec<u8>> kept every
  batch in TopTransactionContext for the life of a long statement (2.9 GB on a 2 GB scan).
- SQL ACL checks take any operation; kafgres_read needs GRANT EXECUTE, plus READ on the topic
  with ACLs on. The log is read as the bootstrap superuser.
- 0.4.0, with a 0.3.0 upgrade script; upgrade-test defaults to starting from 0.3.0.
… reader cannot stall the broker's segment roll (I7)

- On the table engine every SPI read took ACCESS SHARE on kafgres_log, the topic's table and
  its segments, held until the caller's transaction ended. CREATE TABLE ... PARTITION OF and
  retention's DROP TABLE need ACCESS EXCLUSIVE there, so the broker loop waited out its 2 s
  lock_timeout on every produce that rolled a segment, for as long as the reader stayed open:
  250 records took 13.4 s behind an idle reader, 0.6 s after.
- dbtx::released runs a read in a subtransaction it rolls back, which releases what the read
  locked. Bound resolution and each 1 MiB chunk go through it; the reads write nothing.
- It restores the caller's memory context and resource owner, and rolls back before
  rethrowing, so a scan cancelled inside a PL/pgSQL exception block leaves the block's own
  subtransaction intact.
…an xid, and Kafka transaction state and table log start on a fresh snapshot (I6, I8)
…IZABLE, whose fixed snapshot would show a Kafka transaction aborted since as committed
…e what a broker read sees and assign no xid (I6, I8)
…and flush the pending error before released() rolls back (I6)
… numbers, and drop the docs' out-of-range promise for a scan racing retention
…ent scripts on stdin, skip the memory test before 14, wait out the ACL reload, and satisfy clippy's unnecessary_sort_by
…ap exists, wait for the ACL setting instead of sleeping, and document kafgres_read()'s ACL default, header key decoding and released()'s memory contract (I6, I8)
…gres.acls_enabled, not only a new session, so the next test's produce is not refused
… read that moved the offset, and refuse segment-engine time bounds on a standby (I9)
@RayElg
RayElg marked this pull request as ready for review November 17, 2026 01:28
@RayElg RayElg changed the title [wip] Kafgres read Kafgres read Nov 17, 2026
@RayElg
RayElg merged commit 3a5f1c4 into main Nov 18, 2026
56 of 57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant