Add cursor pagination and cache validators to template registry

[solaawojobi00-bit]

Add cursor pagination and cache validators to the template registry

Problem

starforge template list/search always fetch and print the entire marketplace registry in one shot, and the local cache is refetched wholesale on every TTL expiry — there's no way to page through results, and no conditional-request support to avoid re-downloading an unchanged registry.

Scenario	Before
Browsing a large registry	Every match printed at once, no way to page
Registry unchanged since last fetch	Full body re-downloaded every 24h regardless
Resuming a browse after the registry changed	No cursor concept existed

Solution

• Cursor pagination: templates::paginate() / Page<T> split a result set into pages of at most --limit entries. Cursors are opaque, base64-encoded, and anchored to a template's name rather than its list position, so a page stays valid even if entries are added elsewhere in the registry between calls. A cursor pointing at a since-removed entry is rejected with a clear error instead of silently skipping ahead.
• Cache validators: fetch_and_cache_remote() now sends If-None-Match with the last-seen ETag; an unchanged registry replies 304 Not Modified and the local cache is reused instead of re-downloading the body. The ETag is persisted in a sidecar registry.etag file.
• Both are opt-in: template list/search with neither --limit nor --cursor behave exactly as before (show everything), so this is non-breaking.

Changes

• src/utils/templates.rs
  • paginate(), Page<T>, encode_cursor/decode_cursor: stable, name-anchored cursor pagination over any slice.
  • fetch_and_cache_remote() reworked to send a conditional If-None-Match request and return a FetchOutcome::{Fetched, NotModified}; load_registry() handles the 304 path by reusing the cache and resetting its TTL clock instead of rewriting it.
  • registry_dir() now honors STARFORGE_TEMPLATE_REGISTRY_DIR (mirrors the existing STARFORGE_TEMPLATE_REGISTRY_URL override) so tests don't depend on HOME — dirs::home_dir() ignores HOME/USERPROFILE overrides on Windows, and my first test run actually wrote fixtures into a real ~/.starforge/templates/ before I caught this and fixed it at the root.
• src/commands/template.rs
  • --limit/--cursor added to List and Search (including the --json output path on list, which now also reports shown_count/next_cursor).
  • Shared print_pagination_footer() prints Shown X of Y and the next cursor when a page was requested.
• templates/registry.json: security_review.findings was stored as an integer (0, 1) in several entries but the struct expects Option<String> — parsing this bundled fallback registry failed outright. Caught by my own "fetch fails, fall back to bundled registry" test; fixed the data to match the schema.
• Cargo.toml/Cargo.lock, src/utils/database.rs, src/commands/mod.rs, src/utils/mod.rs: see Notes for Reviewers — three small, unrelated pre-existing bugs that were blocking cargo build entirely on master before this PR could even be built against it.

Regression Tests

Test	Scenario	Acceptance criterion
paginate_walks_all_pages_in_order	Primary flow: walk a 5-item set 2 at a time via successive cursors, in order	Primary flow
paginate_cursor_past_last_item_returns_empty_page	Boundary: cursor for the last item returns an empty final page, no error	Boundary case
paginate_rejects_zero_limit	Invalid input: --limit 0	Clear invalid-input handling
paginate_rejects_malformed_cursor	Invalid input: non-base64 --cursor value	Clear invalid-input handling
paginate_rejects_cursor_for_unknown_entry	Failure path: cursor for a since-removed entry	Failure path
fetch_and_cache_remote_stores_and_sends_etag	Primary flow: 200 response stores ETag; follow-up conditional request with that ETag gets 304	Primary flow
load_registry_reuses_cache_on_304_after_forced_refresh	Primary flow (end-to-end via public API): forced refresh short-circuits on 304 and reuses cache	Primary flow
load_registry_falls_back_to_bundled_default_when_remote_unreachable	Unsupported environment: remote unreachable, no cache yet → bundled fallback	Unsupported environments

Testing

cargo build --lib --bin starforge — clean, no errors.

cargo test --lib templates:: (68 tests, run against a locally-patched build — see Notes for Reviewers on why):

test utils::templates::tests::paginate_walks_all_pages_in_order ... ok
test utils::templates::tests::paginate_cursor_past_last_item_returns_empty_page ... ok
test utils::templates::tests::paginate_rejects_zero_limit ... ok
test utils::templates::tests::paginate_rejects_cursor_for_unknown_entry ... ok
test utils::templates::tests::paginate_rejects_malformed_cursor ... ok
test utils::templates::tests::fetch_and_cache_remote_stores_and_sends_etag ... ok
test utils::templates::tests::load_registry_reuses_cache_on_304_after_forced_refresh ... ok
test utils::templates::tests::load_registry_falls_back_to_bundled_default_when_remote_unreachable ... ok

(64 other pre-existing templates:: tests unaffected and passing; one pre-existing, unrelated test — test_publish_template_versioned_stores_by_version — fails via futures::executor::block_on not providing a Tokio runtime context; not touched by this PR.)

Manual CLI verification (template list --limit N, --cursor, search --limit, invalid --cursor, --limit 0) was done earlier against an equivalent implementation before I discovered the branch/base issue described below; I was not able to re-run it interactively against this exact commit — see the stack-overflow note below.

Notes for Reviewers

This PR ended up touching a few things beyond #681 itself, all pre-existing and discovered while getting a clean base to build against:

1. master doesn't currently build. Before any of my changes, cargo build fails with a missing thiserror dependency (used via #[derive(thiserror::Error)] in database.rs but never declared in Cargo.toml) and a rusqlite::Transaction mutability bug in the same file (Migration::up/down took &mut Connection, but Transaction only implements Deref, not DerefMut, so &mut tx never actually worked — narrowed the trait to &Connection, which is all the one real implementation needs). Also, two modules (commands/ai_doc_qa.rs, utils/ai_doc_qa.rs) exist on disk but were never wired up via pub mod ai_doc_qa;, so main.rs's reference to them doesn't resolve. All three are fixed here since [2026 Registry] Add cursor pagination and cache validators #681 couldn't otherwise be built/tested at all.
2. cargo test --lib still doesn't compile even after those fixes — 22 further pre-existing errors across ~10 unrelated files (plugins/manifest.rs, plugins/registry.rs, utils/compliance.rs, utils/bindings.rs, commands/audit.rs, utils/ai.rs, utils/template_analytics.rs, utils/template_recommender.rs), mostly test fixtures that fell out of sync with struct changes from other PRs. I did not fix these — out of scope for [2026 Registry] Add cursor pagination and cache validators #681 — but I did apply throwaway local patches to unblock compilation just long enough to run and confirm the tests listed above pass, then reverted those files before this commit. Worth a maintainer pass on its own.
3. Unrelated stack overflow on this machine: starforge --version crashes with STATUS_STACK_OVERFLOW on Windows — reproduces identically with zero of this PR's changes present (confirmed by reverting templates.rs/commands/template.rs/registry.json and rebuilding), so it predates this PR. Best guess is Windows' 1MB default main-thread stack being exceeded by clap building the full ~90-module command tree at startup; very likely doesn't reproduce on Linux CI's 8MB default stack, but I couldn't confirm that (no WSL distro available in this environment) and couldn't fix it without a substantial, unrelated change (custom linker/thread-stack setup). Flagging so it isn't a surprise if CI's smoke-test job behaves differently than my local testing did.
4. Given feat: Add starforge wallet rename command #2 and fix: Validate Stellar public key format before saving a wallet #3, I could not get a fully green local cargo test/clippy/smoke run on this branch — only cargo build and the pagination/caching-specific tests above are verified directly. Apologies for the scope of this note; happy to split the database.rs/ai_doc_qa fixes into a separate PR if preferred.

Closes #681

[drips-wave]

@solaawojobi00-bit Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits