Repository navigation
Kafgres read - #24
Merged
Merged
Kafgres read#24
Conversation
RayElg
force-pushed
the
sql-read-range
branch
3 times, most recently
from
October 14, 2026 22:20
910f89f to
0764dcf
Compare
…, and bump to 0.4.0 (I4, I8) - kafgres_read(topic, partition, from_offset, to_offset, from_time, to_time, isolation, max_records): records as rows with key, value, headers, timestamp type, leader epoch, producer id/epoch, sequence and transactional flag. Composes with WHERE, joins and LATERAL. - An engine-agnostic cursor over LogStore::read (I4) resolves offset and time bounds, reads 1 MiB chunks, and yields records one at a time; a record over 65,536 headers is refused before decoding (I8). The read_committed abort filter lives in the codec. - FetchSlice reports where the aborted-transaction list hit its cap, so a read never judges a batch past it. The wire Fetch handler does not use this yet. - The table engine copies batches out of SPI's own context: pgrx's get::<Vec<u8>> kept every batch in TopTransactionContext for the life of a long statement (2.9 GB on a 2 GB scan). - SQL ACL checks take any operation; kafgres_read needs GRANT EXECUTE, plus READ on the topic with ACLs on. The log is read as the bootstrap superuser. - 0.4.0, with a 0.3.0 upgrade script; upgrade-test defaults to starting from 0.3.0.
… reader cannot stall the broker's segment roll (I7) - On the table engine every SPI read took ACCESS SHARE on kafgres_log, the topic's table and its segments, held until the caller's transaction ended. CREATE TABLE ... PARTITION OF and retention's DROP TABLE need ACCESS EXCLUSIVE there, so the broker loop waited out its 2 s lock_timeout on every produce that rolled a segment, for as long as the reader stayed open: 250 records took 13.4 s behind an idle reader, 0.6 s after. - dbtx::released runs a read in a subtransaction it rolls back, which releases what the read locked. Bound resolution and each 1 MiB chunk go through it; the reads write nothing. - It restores the caller's memory context and resource owner, and rolls back before rethrowing, so a scan cancelled inside a PL/pgSQL exception block leaves the block's own subtransaction intact.
…an xid, and Kafka transaction state and table log start on a fresh snapshot (I6, I8)
…IZABLE, whose fixed snapshot would show a Kafka transaction aborted since as committed
…e what a broker read sees and assign no xid (I6, I8)
…and flush the pending error before released() rolls back (I6)
… numbers, and drop the docs' out-of-range promise for a scan racing retention
…ent scripts on stdin, skip the memory test before 14, wait out the ACL reload, and satisfy clippy's unnecessary_sort_by
…ap exists, wait for the ACL setting instead of sleeping, and document kafgres_read()'s ACL default, header key decoding and released()'s memory contract (I6, I8)
…gres.acls_enabled, not only a new session, so the next test's produce is not refused
… record reader (I6)
RayElg
force-pushed
the
sql-read-range
branch
from
November 3, 2026 01:25
0bde983 to
5cc106d
Compare
… read that moved the offset, and refuse segment-engine time bounds on a standby (I9)
RayElg
marked this pull request as ready for review
November 17, 2026 01:28
…which Python 3.13 and later require when verifying
…overtakes it, and read NUL in a header key as U+FFFD (I8)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds kafgres_read(), allows reading the logs directly from SQL backend. Read-side counterpart to kafgres_produce().
Call kafgres_read('topic', ...) to get one row per Kafka record
Also adds an incident response demo in demo/read