Repository navigation
Lazy ingest on first query + watch mode (incremental ingest and freshness already shipped) #175
Description
Summary
Make ingest incremental / automatic so users don't have to manually rebuild graph.db.
Motivation
Today the workflow requires the user (or agent) to remember to run cgis_ingest before any query, and to re-run it after edits or the graph is stale. In practice this is the main day-to-day friction:
- An agent that calls
trace_flowbeforeingestgets nothing. - After editing files, every query silently runs against a stale graph until a full re-ingest.
- Full re-ingest of a large repo on every change is wasteful (the FastAPI backend was 10k nodes / 38k edges).
Proposal
Two parts, either independently useful:
-
Incremental ingest —
cgis_ingestdetects changed files (mtime/hash vs a stored manifest) and re-extracts only those, patching the graph. Full rebuild only when the schema/version changes. -
Auto/lazy ingest — query tools (
trace_flow,analyze_impact,drift, …) transparently ensure the graph exists and is fresh before running:- if
db_pathmissing → ingest first (with a one-line notice) - if stale (changed files since last ingest) → incremental update first
- optional
watchmode that keeps the graph live as files change
- if
Bonus: have query tools return a small graph_age / stale_files: N field so callers know freshness.
Impact
Removes the "did you ingest first?" footgun and the stale-graph trap — the biggest barriers to making cgis a default, always-on tool rather than a manual step. Critical for distribution (MCP server, CI action, IDE).
Activity
zaebee commentedon Sep 9, 2026
Part 1 shipped; this issue now tracks part 2 only. Checked against
mainat52e0de1.cgis ingest --incremental / -iexists:_process_fileskips a file whose content hash still matches (files_state),_persist_incrementalwrites back only re-parsed files and deletes nodes of files that disappeared, and_is_noop_incrementalexits early when nothing changed.Part 2 has not shipped, and there is now direct evidence it should
stale_filesis computed inside the pipeline during ingest (pipeline.py:201, 229) and never reaches a query. No tool returns a freshness field, so every caller still guesses.The sharper evidence is that the same need has been solved twice, locally, in the last two merged PRs.
OrphanReportnow carries two hand-rolled staleness signals, each documented as "zero here means re-ingest":test_sources(feat(audit): orphan-class query — nodes nothing builds or extends (needs an annotation edge to be precise) #415) — zero in a repo that has tests means the graph predates theis_testcolumn;generated_excluded(cgis orphans: repo-declared allowlist + nested-class scope (generated-stub filter shipped) #432/feat(orphans): generated classes are noise, not findings — hide them by default (#432) #441) — zero in a repo with generated code means it predatesis_generated.Both exist because a query cannot ask "is this graph current?". A third column would mean a third such counter. That is the argument for a general
graph_age/stale_filesfield rather than per-report proxies — and the proxies can then say what they actually mean instead of doubling as freshness probes.Related friction from the same sessions: a graph built before a schema migration reads a silently-false column until re-ingest, and the
is_generatedmigration had to blankfiles_statehashes precisely so that "re-ingest" was advice a user could follow.Scope for part 2
(1) and (2) are independently useful and are what the two ad-hoc counters are standing in for; (3) is the ergonomics win.
feat(freshness): every query says when the graph is out of date (#175 …
zaebee commentedon Sep 9, 2026
Part 2 shipped in #443 (
0d6d197); this issue now tracks lazy ingest and watch mode only.ingest_staterecords the root and ingest time;SQLiteStore.freshness()stats the tracked files and their directories — never walking — and returnsFRESH/STALE/UNKNOWN. Every CLI read command and the graph-reading MCP tools carry it, and say nothing when the graph is fresh.Two things this turned out to need that the issue did not anticipate
Three states, not two.
UNKNOWN— no ingest record, or a root that has moved — is reported distinctly rather than folded intoSTALEorFRESH. A signal that reads the same when all is well and when it could not look is the failure #441 had just found ingenerated_excluded.The signal's channel matters as much as its content. On stdout the CLI note broke thirteen tests at once (
--format jsonis piped); it goes to stderr. In MCP the same hazard splits by return type — afreshnesskey inside JSON objects, a prose note for text tools, a YAML comment forcgis_init_ontology.cgis_find_symbolreturns a JSON list with nowhere to put a key, so it deliberately carries no signal; giving it one means changing its documented shape.Measured
os.walk+ stat over the same treeNot walking is a correctness choice, not a performance one — the two cost the same. A walk would have to reproduce
IngestionPipeline's directory exclusions, so that set now lives once incore/paths.EXCLUDED_DIRS, shared by both.Still open here
And a follow-up this enables
OrphanReport.test_sourcesand.generated_excludedwere only ever freshness proxies — that is the argument in this issue's earlier comment. They can now say what they actually mean, but changing what two shipped fields report is its own reviewable diff.[-]Incremental / auto ingest: keep graph.db fresh without manual re-ingest[/-][+]Lazy ingest on first query + watch mode (incremental ingest and freshness already shipped)[/+]zaebee commentedon Sep 16, 2026
Retitled to the remainder:
--incrementaland the freshness signal (#443) shipped; open are lazy/auto ingest on first query and watch mode. Note #38: incremental ingest currently leaves stale edges, which matters before making it automatic.fix(freshness): report a write made during ingest as stale (#535)
zaebee commentedon Oct 2, 2026
Decision and what shipped. This will close with #541.
Measured on this repo (6.5k nodes / 28.6k edges): the freshness probe costs about 3 ms. An incremental ingest takes 0.7 s after a body edit and 5.6 s after a signature edit, which forces a full rebuild.
CGIS_AUTO_REFRESH=1, every graph-reading MCP tool runs an incremental ingest first when the graph is STALE. One refresh at a time, with freshness re-read under the lock. The refresh reuses the recorded--source-root/--domains(otherwise it would rename every node). A missing database is never created.Stophook, gitpost-checkout/post-merge/post-rewrite).Generated by Claude Code