Repository navigation
feat(api): page the History view with a keyset cursor (has_more, next_cursor) - #1277
Conversation
…expose the actionable cap fetchExecutionsByClass logged a listing error and skipped the class, so a failed actionable query hid every pending approval while the response kept truncated=false. The class fetch now reports failure and fetchExecutionsAsHistory raises the incomplete-results signal (including on the no-executions early return). HistoryResponse gains actionable_executions_limit (config.MaxListLimit) next to executions_limit (terminal cap, DefaultListLimit), so the UI banner can quote the cap that was actually hit instead of always quoting the terminal cap. The banner text itself lives in frontend/ and is left to the UI owner. Refs LeanerCloud#686 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_cursor) GET /api/history listed at most `limit` purchase_history rows and 100 terminal executions, and rows past those caps could not be reached: there was no way to ask for the next page. Completed purchases and terminal executions are now one stream, newest first, ordered by (timestamp DESC, source DESC, row id DESC), which is unique. Each response lists `limit` rows of it and, when more rows the caller may see follow, sets has_more and next_cursor; passing next_cursor back as `cursor` returns the next page. The cursor is a keyset position, not an offset: the account scope and the execution filters run in Go after the SQL read, so an offset would count rows the caller never sees, and every row written ahead of it would shift the page. With the keyset, a later page resumes strictly after the last row shown, so rows written between page loads never repeat or skip a row, and a page costs the same at any depth. `offset` is rejected with 400 instead of ignored, so a client written against an offset API cannot loop on the first page. Actionable executions (pending approvals, in-flight purchases) are not paged: the first page lists the whole class as before, and cursor pages leave it out, so paging can never push a pending approval off the first page. The LeanerCloud#686 signals are kept: a failed execution listing or a full actionable class still sets truncated. has_more is judged on rows the caller may see, so other tenants' rows never set it, and the cursor only ever names a row the caller was shown. The scope is applied again on every page. Store: ListPurchaseHistoryPage (GetPurchaseHistoryFiltered's predicate plus the keyset, returning each row's id) and ListHistoryExecutionsPage (the terminal statuses, clean completed rows excluded, scheduled_date bounds pushed into SQL). No migration: the existing timestamp and scheduled_date indexes serve both. Closes LeanerCloud#674 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bring the branch up to date with main. Conflicted files: - internal/api/handler_history.go: main (LeanerCloud#1200) replaced summarizePurchaseHistory(all) in the getHistory response with the scope-aware SQL summary plus SummaryComplete; the branch added ActionableExecutionsLimit to the same literal. Kept both: main's Summary/SummaryComplete and the branch's ActionableExecutionsLimit. - internal/api/types.go: both sides rewrote the Truncated doc comment. Combined them: main's note that Summary still counts rows beyond the caps while SummaryComplete is true, and the branch's note that a failed execution class listing also sets Truncated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch is stacked on fix/issue-686 (LeanerCloud#1276). That branch was brought up to date with origin/main in ee67abd, so merging it here brings in main and the LeanerCloud#686 conflict resolutions in one step. Conflicted files: - internal/api/main_nonet_test.go, internal/api/main_integration_test.go: both sides added a teardown hook list for the shared Postgres server (this branch: afterAllTests; main LeanerCloud#1400: afterTests). Took main's files unchanged; this branch only needed the hook, which main now provides. - internal/api/handler_history.go, getHistory: main (LeanerCloud#1200) loads the account scope once (loadHistoryScope), filters rows with scope.filter, defers the stale-approval sweep until after the summary aggregate, and returns the scope-aware SQL summary with SummaryComplete. This branch pages the stream (fetchHistoryPage) and lists the actionable class only on the first page. Combined: the page is fetched first, the scope is loaded once and applied to the actionable rows, staleExecs is declared outside the Cursor == nil branch so the sweep stays deferred (and does not run on cursor pages), and the response carries main's Summary/SummaryComplete plus this branch's paging fields. The summary is now whole-scope on every page. - internal/api/handler_history.go, fetchPurchaseHistory: main replaced the GetAllPurchaseHistory/GetPurchaseHistoryFiltered split with the shared historyFilters.purchaseHistoryFilter; this branch reads one keyset batch through ListPurchaseHistoryPage. Kept both: purchaseHistoryFilter is kept (the summary uses it) and the batch read now starts from it with Limit set to the batch size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main (LeanerCloud#1200) made purchaseHistoryFilterWhere the one purchase_history predicate builder and historyExecutionsPredicate the one executions predicate, shared by the History row fetch and the summary aggregate so the totals count exactly the rows the list can show. This branch's page queries carried their own copies (purchaseHistoryPageConds and an inline status/clean-completed predicate). Use the shared ones instead, so the paged list and the whole-scope summary cannot drift apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main (LeanerCloud#1200) added purchaseHistoryRowStatus so listed purchase_history rows and the summary's aggregate groups get the same Status. The paged stream set the literal "completed"; use the constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The History summary tests came in from main (LeanerCloud#1200) and mock the old purchase_history reads (GetAllPurchaseHistory, GetPurchaseHistoryFiltered). This branch reads purchases through ListPurchaseHistoryPage, so the filter-equality test panicked on a nil row filter. Mock the paged read instead; the assertions are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Brought this branch up to date with main. This branch is stacked on #1276, so I merged #1276's updated head (ee67abd, which already contains origin/main 2514c01) instead of merging main twice. New head: 2b3c8bb. A trial merge with the current main (c1f1781) is clean and passes Conflict resolution (merge commit 998ebbd):
Follow-up commits (semantic conflicts that merged without a textual conflict):
Checks on the merged head:
When merging, note that |
|
Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE Blocking:
Non-blocking:
Attacked:
Body-stale:
Reviewer: rB11. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths. |
Brings the branch up to date with main now that LeanerCloud#1276 (the base this PR was stacked on), LeanerCloud#1100 (route documentation) and their neighbours have merged. Conflicted files: - internal/api/handler_history.go, internal/api/handler_history_actionable_starvation_test.go: main's copies are byte-identical to the LeanerCloud#1276 tip (ee67abd) this branch already contained, so the branch's paged versions are kept unchanged. - internal/api/types.go: HistoryResponse keeps the branch's ExecutionsLimit comment and the HasMore/NextCursor fields; main's other type changes (RecommendationsResponse.NextCursor, UnrestrictedAccountAccess, nullable CurrentCoverage) merged cleanly. - internal/api/openapi.yaml: GET /api/history takes main's corrected parameter list from LeanerCloud#1100 (provider, account_ids, account_id, start/end dates, no stray interval parameter) and the 400 response, plus the branch's description, the paged limit (maximum 1000) and the cursor parameter. HistoryResponse keeps the branch's limit and executions_limit text and its has_more/next_cursor properties. Also, internal/api/handler_history_truncation_test.go merged without a conflict marker but declared TestHandler_getHistory_ActionableExecutionsLimitExposed twice: LeanerCloud#1276 and the branch each added it at a different offset. Main's copy equals the LeanerCloud#1276 tip, so the branch's version (one declaration) is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged current main into the branch. #1276, the base this PR was stacked on, has merged, so this PR's diff against main is now only #674's changes. New head: 2b349dd. Conflicts:
One more fix in the merge: Checks:
Note: #1254 deletes the hand-rolled analytics |
|
Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE Blocking:
Non-blocking:
Attacked:
Body-stale:
Reviewer: rB18. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths. |
|
Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE Addendum to my review above, posted after skimming the earlier reviews. The verdict is unchanged. Blocking:
Non-blocking:
Attacked:
Body-stale:
Reviewer: rB18. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths. |
Conflict: internal/analytics/collector_test.go. LeanerCloud#1254 folded the hand-rolled analytics mock onto mocks.MockConfigStore, so took main's file and dropped this branch's two stubs there; the shared mock already carries them from this branch's internal/mocks/stores.go change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged origin/main (now |
|
Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE Blocking:
Non-blocking:
Attacked:
Body-stale:
Reviewer: rB22. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths. |
|
Local verification at |
|
Merged by IAmJSD's agent at IAmJSD's request (IAmJSD is acting owner while the maintainers are away), under the bar IAmJSD set: 1 approving independent review(s) on |
Stacked on fix/issue-686 (#1276). Until that merges, this diff also shows its commit (0ad90b7). Review only the last commit.
Root cause
GET /api/historyread at mostlimitpurchase_history rows (default 100, max 1000) and at most 100 terminal executions (failed, expired, canceled, audit-gap and partially completed). It returnedtruncated(#630) when a cap was hit, but it had no way to ask for the next rows, so older rows could not be reached at all. #632's scope note moved paging to this issue.Fix
One paged stream. Completed purchases and terminal executions are merged into one list, newest first. The order is
(timestamp DESC, source DESC, row id DESC):purchase_history.timestamporpurchase_executions.scheduled_date;purchase_history.idorexecution_id.Both ids are UUIDs and unique per table, so every row has its own position.
purchase_idcan't be the tiebreaker because legacy rows repeat it (migration 000108). Each response listslimitrows of the stream.Keyset cursor, not offset.
has_more: trueandnext_cursor. Passingnext_cursorback ascursorresumes strictly after the last row shown.v1.<source>.<unix µs>.<uuid>. Anything that doesn't decode to that form is a 400, so a bad cursor never reaches SQL.offsetis rejected with 400 instead of being ignored. feat(api): page the History view past the first limit rows (offset, has_more) #674's issue text asked for anoffsetparameter. A client built to that spec would otherwise get the first page back for every offset and keep asking for more.Pending approvals are not paged. The actionable class (pending, notified, scheduled, approved, running, paused) is read and listed the same way as before:
So paging can never push a pending approval off the first page. Before this PR it stayed visible only because each source had its own cap. Now it holds by construction. #686's signals are unchanged: a failed listing or a full actionable class still sets
truncated.Scope.
has_moreis judged on rows the caller may see. The page reads one visible row pastlimit, so rows outside the caller's scope never sethas_moreand never leave a trailing empty page.next_cursoronly names a row that was in the response.filterPurchaseHistoryByAllowedAccounts, which now runs per batch. Its result is mapped back to rows bymarkOutOfScope, so the scope rule still has a single implementation.Store. Two new methods in
internal/config/store_postgres_history_page.go:ListPurchaseHistoryPage:GetPurchaseHistoryFiltered's predicate plus the keyset condition. It also returns each row's id, and still scans through the sharedscanPurchaseHistoryRow.ListHistoryExecutionsPage: the terminal statuses with clean completed rows excluded, thescheduled_datedate bounds pushed into SQL, plus the keyset condition.Each source is read in batches of
max(limit+1, 100), capped at 1000. A batch is read again only when filters or scope hide rows. There is no migration:idx_purchase_history_timestampandidx_purchase_executions_scheduled_datecover both queries.Response and docs.
HistoryResponsegainshas_moreandnext_cursor.truncatedis also true wheneverhas_moreis, so today's banner keeps showing until the UI has Load more.executions_limitnow equalslimit, because terminal executions share the page./api/historynow documents thelimitandcursorparameters, the paging contract, and the new fields.Closes #674
Interaction with #1200 (scope-aware summary totals)
This branch doesn't touch the summary. On its own,
summarystill covers the rows in the response, as it did before. Combined with #1200, the summary is whole-scope on every page.I checked this with a trial merge of this branch and #1200's head (abec6cd) in a scratch worktree:
git merge-treereports conflicts only inhandler_history.goandtypes.go. fix(history): truncation banner quotes the wrong cap and a failed class listing does not set truncated #686 alone already conflicts with feat(api): scope-aware SQL totals for the History summary cards #1200 in both files.getHistory.loadHistoryScopeonce, before the actionable fetch.scope.filter.deferthe sweep withstaleExecsdeclared outside theCursor == nilbranch.historySummary(ctx, filters, scope, all)/SummaryCompleteplus this PR's paging fields.fetchPurchaseHistory. Usefilters.purchaseHistoryFilter()withLimitoverridden by the batch size, then callListPurchaseHistoryPage.TestGetHistory_SummaryFilterMatchesRowFetchFilterandTestGetHistory_SummaryFallsBackToPageWhenAggregateFails. They needGetAllPurchaseHistory/GetPurchaseHistoryFilteredrenamed toListPurchaseHistoryPagewith a thirdmock.Anything(4 lines). Without that change,TestGetHistory_SummaryFilterMatchesRowFetchFilterpanics on a nilrowFilter.Start.go test ./internal/api/passed.TestIntegration_HistorySummary_*passed on real Postgres, and so did this PR'sTestIntegration_HistoryPaging_*.limit=7. Every one of the 18 pages had the identical whole-scope summary, withsummary_complete: true. The pages listed 129 rows in total, exactly thetotal_purchasesthe summary reports.Verification
All on the Linux VM with Go 1.26.9, with
AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true. Integration tests used real Postgres via testcontainers (the shared container). This is local mock and integration evidence only: there's no deployed-stack or real-account coverage. The change is DB and handler only.Fail before, pass after (real Postgres). A throwaway probe seeded 12 completed purchases and paged with
limit=5, followingnext_cursor:"map[p-00 … p-04]" should have 12 item(s), but has 5(nohas_more, no cursor);has_morefalse on the last one, all 12 rows reached.New tests.
internal/api/handler_history_page_integration_test.go, real Postgres:TestIntegration_HistoryPaging_EveryRowOnceInOrder: two tenants, purchases and terminal executions sharing timestamps within and across tables,limit=7. Every row comes back exactly once in stream order. Both pending approvals, older than every other row, are on page 1 and on no later page. A purchase and a failed execution written at the top after page 1 don't shift, repeat or skip anything.TestIntegration_HistoryPaging_ScopedUserNeverPagesIntoAnotherTenant: tenant-B rows are newer, interleaved and older. The tenant-A user gets exactly the tenant-A rows plus their own ambient execution, once each and in order. No cursor names a hidden row, andhas_moreis false on the last page although tenant-B rows follow it. Tenant B's pending approval never appears.internal/config/store_postgres_history_page_integration_test.go, real Postgres:ListPurchaseHistoryPageselects the same rows asGetPurchaseHistoryFilteredfor 8 filter shapes (provider, both account columns, the date edges ±1µs);scheduled_datebounds.internal/api/handler_history_page_test.go, unit/mock:offset, malformed, wrong version, unknown source, out-of-range timestamp, non-UUID, non-canonical UUID, injection, overlong;truncated;Mutation probes. Each one was applied by hand, run, then reverted:
<=→<in the cross-source keyset condition:TestIntegration_HistoryKeyset_CrossSourceTiesandTestIntegration_HistoryPaging_EveryRowOnceInOrderfail.id DESCfrom the purchase_history ORDER BY:TestIntegration_ListPurchaseHistoryPage_WalksEveryRowOncefails.has_morebefore the visibility check:TestHandler_getHistory_ScopedPagingStaysInScopefails. The first version of the test let this survive, so its fixture now ends on a full page.Existing tests. The history handler tests now mock
ListPurchaseHistoryPage/ListHistoryExecutionsPageinstead ofGetAllPurchaseHistory/GetPurchaseHistoryFiltered. For a[]PurchaseHistoryRecordfixture, the shared mock assigns row ids by position, so most of these edits are one-line renames. These assertions changed on purpose:limitrows ⇒ truncated": exactly a page of rows is no longer a false positive, so the tests now uselimit+1.executions_limitnow followslimit.TestHandler_getHistory_ScopedUserTruncationLeaksNoCounts: other tenants' rows no longer settruncated. It still asserts that no count leaks.TestHandler_getHistory_FetchesDisjointClassesWithTheirLimits: asserts one actionable read plus one terminal page read.Combined-tree verification. This branch was tested together with the other queued branches on a local, never-pushed tree:
queue/platform@936996ad= origin/maina14dc8f2plus these heads, merged in this order:fix/issue-72andfix/issue-312conflict on one line ininternal/api/handler_dashboard.go(both change thecalculateCommitmentMetricscall). The tree used #72's side with #312's validatedprovidervariable; whichever PR merges second takes that same resolution.Linux, Go 1.26.9,
AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true; Docker Postgres via testcontainers:936996ad:go build ./...,go vet ./...,go vet -tags integration ./...and the fullgo test -race -short -count=1 ./....936996ad: golangci-lint v2.10.1 found 0 issues on every changed package, with and without--build-tags integration.tests/e2eis a separate module, so it was vetted under its owne2etag.gocyclo -over 10is silent on the changed non-test files.-tags integration -count=1, 60-minute local timeout because the VM was shared):936996ad, these pass:internal/server,internal/credentials,internal/scheduler,internal/database/postgres/migrations,internal/purchaseandinternal/api.internal/configpasses exceptTestPostgresStoreDB_ResolveStaleRIExchange_ValidatesCloudCheck, which fails the same way on unmodified main (fixed by fix(config): compare resolve checked_at at the database's microsecond precision #1252).26795376, same branches at earlier heads):internal/analytics,internal/auth,internal/commitmentopts,internal/databaseandcmd/lambda/clear-rate-limitall pass. The later fixes don't touch those packages.httpToLambdaRequestsignature change;scripts/setup-git-secrets.shpatterns in a fresh repo with an isolated HOME, found nothing in the changed files, and a planted fake key was flagged.scripts/test-git-secrets-allowlist.sh: 137 PASS, 0 FAIL.Notes and things left out
Rows that change state between page loads. A row is listed in the state it has when its page is read. The common case is the stale-approval sweep:
scheduled_dateis at least 7 days old, so a later page normally lists the same execution again, now as expired.The same thing happens to any approval that fails or expires while the user pages. The stream itself never repeats a row; this is one execution appearing once as actionable and once as terminal. The frontend should de-duplicate by
purchase_idand keep the newest load (see below).Scan cost for scoped users. Scope still runs in Go, so filling a page for a user whose rows are sparse can read several batches. In the worst case that's the whole table, as feat(api): scope-aware SQL totals for the History summary cards #1200's aggregate already does, and the date window bounds it when set. Pushing the scope into SQL is fix(history): pending approvals can fall off the first page because filters are applied after the executions cap; add paging and push provider/account/date filters into the executions SQL #632's "PR2" and is not done here.
Scope checks per batch.
markOutOfScopereusesfilterPurchaseHistoryByAllowedAccounts, so a request loads the account scope once per batch instead of once overall. Usually that's 2–3 times, all cheap. With feat(api): scope-aware SQL totals for the History summary cards #1200 merged, this can become oneloadHistoryScopeplusscope.allows.truncatedwording. In types.go and openapi.yaml thetruncateddescription still says it "may be a false positive when exactlylimit… rows exist". That's no longer true for the paged stream. I left that paragraph alone because fix(history): truncation banner quotes the wrong cap and a failed class listing does not set truncated #686 and feat(api): scope-aware SQL totals for the History summary cards #1200 both rewrite it. The newhas_moredescription explains how the two flags relate.Page size semantics.
limitnow counts completed purchases and terminal executions together. Before, the first page could hold up tolimitpurchases plus 100 terminal executions, with no ordering between the two lists. The older of those rows are now on page 2 instead.Predicate copy.
purchaseHistoryPageCondsrepeatsGetPurchaseHistoryFiltered's three trivial conditions and reuses its shared account predicate.TestIntegration_ListPurchaseHistoryPage_MatchesFilteredFetchguards against drift. feat(api): scope-aware SQL totals for the History summary cards #1200 extractspurchaseHistoryFilterWhere; once both are merged, the page query can call it.Test teardown.
internal/apiintegration tests that use the shared container now get it stopped through anafterAllTestshook inmain_nonet_test.go, registered from the newmain_integration_test.go.Lint config.
.golangci.yml: the new store file is added to the existing misspell allowlist for the legacycancelled_bycolumn.Frontend follow-up for the UI owner (#676)
No change under frontend/. What "Load more" needs, in
frontend/src/api/history.ts,frontend/src/history.tsandfrontend/src/types.ts:has_more?: boolean,next_cursor?: stringandactionable_executions_limit?: numbertoHistoryResponse, and letgetHistorypasscursor(andlimitif wanted).has_moreis true, show a "Load more" button under the history table. On click, repeat the request with the same filters pluscursor=next_cursor, then append the returned rows to the history list.has_more: false.purchase_idand keep the row from the newest response. A stale approval swept to expired after page 1 shows up again on a later page in its new state (see the notes above).has_moreis true, the rows can be paged to, so replace "Narrow the date range or filters to see them" with the Load more button. Keep the banner only whentruncated && !has_more, meaning rows that paging can't reach: the actionable cap (quoteactionable_executions_limit) or a failed listing. Theexecutions_limittext in the current banner now equalslimit.Maintainers: please mirror the issue's triage labels on this PR:
priority/p2,severity/medium,urgency/this-quarter,impact/many,effort/m,type/feat,triaged.🤖 Generated with Claude Code