Skip to content

fix(pds): page listRecords without repeating or stalling - #252

Open
decoded-cipher wants to merge 2 commits into
ascorbic:mainfrom
decoded-cipher:fix/251-list-records-pagination
Open

decoded-cipher wants to merge 2 commits into
ascorbic:mainfrom
decoded-cipher:fix/251-list-records-pagination

Conversation

@decoded-cipher

@decoded-cipher decoded-cipher commented Oct 5, 2026 •

Copy link
Copy Markdown

Summary

com.atproto.repo.listRecords pagination was broken in two ways (#251):

  • Forward paging repeated records. Each page started with repo.walkRecords(cursor), and MST.walkFrom returns its start key twice when that key exists. So the record at each page boundary came back up to three times. With limit=2 a page was just that record twice, and the cursor never moved.
  • reverse=true never advanced. The walk always ran forwards and the page was reversed afterwards. The cursor ended up as the smallest key on the page, so following it returned the same page forever.

This PR:

  • Walks the collection's range of the repo tree directly, in either direction. The cursor key is excluded, and subtrees outside the range are skipped. Only the records on the returned page are read.
  • Lists records newest first by default and oldest first with reverse=true, matching the reference PDS. Cirrus had this the other way round.
  • Uses the bare rkey of the last record as the cursor, as the reference PDS does. Old collection/rkey cursors are still accepted, so clients in the middle of paging across an upgrade keep working.
  • Clamps limit to 1–100, defaulting to 50 when it's missing or not a number. Before, limit=0 or a negative limit looped forever, and limit=abc returned the whole collection in one response.
  • Stops a listing for a collection with no records from reading every record after it in the repo.

Closes #251

Commits

  1. fix(pds): clamp listRecords limit to the lexicon range.
  2. fix(pds): page listRecords without repeating or stalling, with tests, changeset and plan doc row.

Notes for review

  • Behaviour change: the default order flips to newest first, which is why the changeset is minor. This matches the reference PDS (orderBy(uri, reverse ? 'asc' : 'desc')). Apps that use limit=1 to fetch the latest record get the newest one now.
  • Last page has no cursor. The reference PDS returns a cursor on every page, and the client then fetches an empty page to find the end. Cirrus only returns a cursor when there are more records. I kept that because the dashboards show "N+" based on whether a cursor is present.
  • Out-of-range limit is clamped, not rejected with 400 as in the reference PDS. This is consistent with how spaces.ts handles listSpaces.
  • Not a bug in @atproto/repo. Upstream never passes an existing key to walkFrom: list() skips the start key and listWithPrefix() starts from a prefix that isn't a real key. Cirrus was relying on behaviour walkFrom doesn't promise.

Test plan

  • PDS unit suite passes (351, 10 new): walker tests on a 340-key multi-level tree with neighbouring collections on both sides, at many page sizes, both directions, plus cursor edge cases; endpoint tests for both orders, the cursor format and limit clamping. The new endpoint tests fail on main.
  • PDS CLI suite passes (84).
  • Each commit builds and passes tests on its own; no new type errors.
  • Fuzz test outside the repo: 6,000 random queries on 150 random trees matched a brute-force sorted list.
  • Deployed to a test instance with 340 records and checked live against a CAR export of the repo (details below).

Before / after

main / 0.19.0 (public instance, read-only requests):

Request Result
limit=3 10 follows came back as 24 records; boundary records 3 times
limit=2 never ends, 2 distinct records repeated
limit=3&reverse=true same page and cursor every time

This PR, on pds-test.arjunkrishna.dev, compared against com.atproto.sync.getRepo:

Check Result
Full paging at limits 1, 2, 3, 7, 50, 99, 100, both orders Exact order, all 340 records, no duplicates, always finishes
Order Newest first by default, oldest first with reverse=true
Cursor Bare rkey; old format, a key with no record, an empty cursor and the ends of the list all correct
limit of 0, -5, abc, 101, 1000, missing 50, 1, 50, 100, 100, 50
Neighbouring collections, collection with no records No overlap; empty result with no cursor
Records created or deleted between pages, including the cursor record No duplicates, nothing skipped

The test instance is still running with this data, in case it's useful:

https://pds-test.arjunkrishna.dev/xrpc/com.atproto.repo.listRecords?repo=did:web:pds-test.arjunkrishna.dev&collection=dev.arjunkrishna.test.pagination&limit=7

listRecords passed limit straight through, capped only at 100. limit=0
or a negative value returned an empty page whose cursor pointed back at
the start of the collection, so clients that follow the cursor looped
forever. A non-numeric limit became NaN, which disabled the limit and
returned every record in the collection in one response.

Clamp it to 1-100 and fall back to the default of 50 when it is
missing or not a number, as spaces.ts already does for listSpaces.
listRecords started each page with repo.walkRecords(cursor). The cursor
was the last record of the previous page, and MST.walkFrom yields its
start key twice when that key exists, so the boundary record came back
up to three times across pages. With limit=2 a page was just that
record twice, so the cursor never moved. reverse=true walked forwards
and reversed the page afterwards, which made the cursor the smallest
key of the page, so following it returned the same page forever.

Walk the collection's MST range directly instead, in either direction,
excluding the cursor key and skipping subtrees outside the range. Only
the records on the page are read. Records are listed newest first by
default and oldest first with reverse=true, matching the reference PDS;
the order used to be the other way round. The cursor is the bare rkey
of the last record, as in the reference PDS, and cursors in the old
collection/rkey form are still accepted. Listing a collection with no
records no longer reads every record after it in the repo.

Closes ascorbic#251
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

listRecords pagination returns duplicate records, and reverse=true cursor never advances

1 participant