[alex-rawlings-yyc]

Part of #426; closes #442. Stacked on #486.

Summary

Derived indexes, built from the records on first use and kept current by apply, and a debug command that rebuilds them and reports differences.

AnalysisRepository
+  getTokenAnalyses / getPhraseAnalyses / getSegmentAnalyses(ids)
+  findLinksByToken(tokenRefs)            # token links, and phrase links touching them
+  findLinksBySegment(segmentIds)
+  findLinksToAnalyses(analysisIds)
+  findLinksInBooks(bookCodes, statuses?)
+  findTokenAnalysesByForm / findPhraseAnalysesByForm(normalizedForms)
+  checkIndexes()
interlinearizer.checkIndexes(sessionId) -> [{ index, key, missing, extra }]

• AnalysisIndex answers the lookups for both DraftRepository and Pt9ImportRepository, through a shared IndexedRepository base that hands out copies.
• Keys: token ref, segment id, analysis id, normalized form, book, and book plus status.
• The conformance suite checks every lookup, comparing without regard to order, after applies, undo and redo.

Evidence

The done-when, checked in Node with npm run perf:indexes over the #407 tiers. It applies random change sets with undos, compares lookups with a scan of what storage holds, and runs checkIndexes. In the app the PAPI command reports nothing until edits go through a session, which #443's commands make possible.

tier             links   steps (undos)   differences: at load / after   lookup mismatches
small            2,436     300 (74)                0 / 0                       0
medium          45,252     110 (32)                0 / 0                       0
large-loadable 297,064      17 (6)                 0 / 0                       0
large          495,824      15 (2)                 0 / 0                       0

Decisions with a strong alternative

• Reads by id are here, not in 3d, which needs them.
• findLinksInBooks takes several books, since the contract has lookups take keys in batches.
• checkIndexes is part of the seam; a backend keeping no derived indexes reports nothing. It reports nothing until a lookup or apply has built an index.
• A Save keeps the draft and index, so the next edit reads nothing again; exportAll and a whole save drop them. Copying the draft on every load instead cost about 1.3 s per load on large-loadable. A session's first edit still re-reads the draft, which 4a can move to the session's open.
• A sync or a deletion drops an import's kept records, so an open import session answers from what storage holds.
• A replaced record keeps its place in the index, so a one-book lookup returns links in stored order, which reanchorAnalysisToBook's tie-breaks need in 3j. The seam still promises no order.

Follow-up

#505 scopes each apply's write to the books it reaches.

Merge Danger

Door: two-way

Nothing calls the lookups or the command yet.

Blast Radius: DraftRepository and Pt9ImportRepository, which keep a draft and its index after their first lookup or apply; savePt9Import and deleteProject, which drop an open import repository's records

🤖 Generated with Claude Code

---

This change is 

Summary by CodeRabbit

• New Features
  • Added a command to check analysis data in an open session and return any differences as JSON. The command reports an empty result when no differences are found.
  • Added analysis lookups across saved drafts and Paratext 9 imports, including searches by analysis ID, linked token or segment, book, assignment status, and normalized form.
• Documentation
  • Clarified how analysis data is loaded, kept current, and checked for differences.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: e213136f-1fa2-49dc-8ad4-a3321270d4f1

📝 Walkthrough

The change adds derived indexes for analysis and link lookups, integrates them with draft and Paratext 9 import repositories, and adds index-checking and performance-checking tools.

Changes: Analysis index and repository lookup

Layer / File(s)	Summary
Analysis index and record tables src/types/analysis-repository.ts, src/services/recordTable.ts, src/services/analysisIndex.ts, src/__tests__/services/recordTable.test.ts, src/__tests__/services/analysisIndex.test.ts	The repository contract adds analysis retrieval, link searches, normalized-form searches, and index differences. RecordTable and AnalysisIndex implement these lookups and compare indexes with rebuilt indexes. Tests cover lookups, updates, and differences.
Repository lookup and index caching src/services/indexedRepository.ts, src/services/draftRepository.ts, src/services/pt9ImportRepository.ts, src/services/projectStorage.ts, src/utils/change-set.ts, src/utils/json-copy.ts, src/__tests__/analysis-repository-conformance.ts, src/__tests__/services/{draftRepository,indexedRepository,projectStorage,pt9ImportRepository,analysisSessions,draftRepository.conformance}.test.ts, src/__tests__/utils/{change-set,json-copy}.test.ts, AGENTS.md	IndexedRepository delegates lookups to AnalysisIndex and returns copies. Draft and Paratext 9 import repositories retain indexes, with import revisions invalidating stale caches. Project storage forwards draft lookup calls. Change-set results share supplied put records and scope records; jsonCopy provides a JSON-based copy function. Tests cover repository lookups, cache behavior, index consistency, and copy behavior.
Index check command src/main.ts, src/types/interlinearizer.d.ts, src/__tests__/main.test.ts	The interlinearizer.checkIndexes command checks the open session’s repository and returns differences as JSON. It logs and rethrows errors. Tests cover registration and command behavior.
Index consistency performance harness perf/indexes/cli.ts, package.json, perf/README.md, AGENTS.md	The perf:indexes command applies random edits and occasional undo operations, then checks indexes and compares lookups with scans. It reports unknown tiers or mismatches with a nonzero exit status.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant VSCode
  participant CheckIndexesHandler
  participant AnalysisRepository
  VSCode->>CheckIndexesHandler: Invoke command with session ID
  CheckIndexesHandler->>AnalysisRepository: Call checkIndexes
  AnalysisRepository-->>CheckIndexesHandler: Return index differences
  CheckIndexesHandler-->>VSCode: Return JSON string

Loading

Possibly related issues

• sillsdev/interlinearizer-extension#443 — The issue’s token commands use apply; this change adds repository index maintenance through apply.

Possibly related PRs

• sillsdev/interlinearizer-extension#485 — This change extends the Pt9ImportRepository introduced in that PR with indexed lookups and revision-based cache invalidation.
• sillsdev/interlinearizer-extension#469 — This change extends the repository contract and conformance suite introduced in that PR.

Merge Risk: 🔵 Low · up to 8bdea

A lookup racing with import deletion can keep returning the deleted import until the repository is reopened. Fix the revision ordering before merge if deleted imports must stop answering immediately.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 72.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 40 functions across 23 files. (3 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main changes: adding derived indexes and a rebuild-and-diff command.
Linked Issues check		Issue #442 requires derived indexes and a debug rebuild-and-diff PAPI command. AnalysisIndex builds indexes from repository records, supports all declared lookups, and updates them through apply. …
Out of Scope Changes check		The changes stay within issue #442. AnalysisIndex, RecordTable, IndexedRepository, repository cache and apply changes implement derived indexes and their lookups. The command, public types, conf…

Full details: Linked Issues check

Explanation

Issue #442 requires derived indexes and a debug rebuild-and-diff PAPI command. AnalysisIndex builds indexes from repository records, supports all declared lookups, and updates them through apply. DraftRepository and Pt9ImportRepository retain the indexed records as designed. IndexedRepository.checkIndexes() compares the kept index with a rebuilt index. interlinearizer.checkIndexes(sessionId) exposes this result as JSON and has handler tests. The perf:indexes harness exercises the lookup and rebuild checks on the #407 tiers, including edits and undos, and the PR reports zero differences and lookup mismatches. The implementation builds on first use rather than eagerly at repository construction, but this is the documented implementation choice and does not omit the required behavior.

Full details: Out of Scope Changes check

Explanation

The changes stay within issue #442. AnalysisIndex, RecordTable, IndexedRepository, repository cache and apply changes implement derived indexes and their lookups. The command, public types, conformance tests, service tests, and perf:indexes harness support or verify that behavior. The jsonCopy and change-set updates protect repository and index ownership at the repository boundary. Import revision tracking invalidates cached indexed records after sync or deletion, which supports index correctness. Documentation changes describe the new behavior. No unrelated change is demonstrated.

Full details: Docstring Coverage

Explanation

Docstring coverage is 72.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 40 functions across 23 files. (3 skipped: 3 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[alex-rawlings-yyc]

On the follow-ups in the first review's summary:

• [4a] Wire the WebView to the host behind a feature flag #453, the apply cost: filed as Scope apply's draft write to the books an edit reaches #505, which blocks [4a] Wire the WebView to the host behind a feature flag #453, and noted on [4a] Wire the WebView to the host behind a feature flag #453.
• [4a] Wire the WebView to the host behind a feature flag #453, getFullAnalysis: noted on [4a] Wire the WebView to the host behind a feature flag #453 that it should answer from a copy of the kept draft.
• [3k] Add analysis to segment pages #450: noted that the first session-form getSegments pays for re-reading the draft and building the index, which 3k's checks should time.
• 3j, 3k and 3m: no comment. Their done-whens compare different things, and the part they could share, loading a tier in Node, is already in perf/storage.ts.
• Build a host C# program in the extension to prepare for Harmony integration #462: no comment. The seam's doc says a repository keeping no indexes reports nothing, and the conformance suite expects that.

[imnasnainaec]

@imnasnainaec reviewed 9 files and all commit messages, made 2 comments, and resolved 26 discussions.
Reviewable status: 9 of 27 files reviewed, 2 unresolved discussions (waiting on alex-rawlings-yyc).

---

src/__tests__/utils/json-copy.test.ts line 7 at r3 (raw file):

describe('jsonCopy', () => {
  it('copies a JSON-plain value deeply, sharing nothing with it', () => {
    const value = { gloss: { en: 'and' }, morphemes: [{ form: 'and' }] };

⛏️ Could we add some other atomic types in this test value?

---

src/services/recordTable.ts line 107 at r3 (raw file):

  private put(record: T): void {
    this.remove(record.id);

❓ From 3j (#449): this line changes the order lookups return records in, and the re-anchor pass depends on that order.

• Removing before re-filing moves a replaced record to the end of records and of each index set. applyChangeSet keeps a replaced record in place.
• 3j's pass will read a book through findLinksInBooks([book]).
• reanchorAnalysisToBook breaks ties by list order: of two stale links contending for one target, only the earlier revives. So it matches the WebView only when it gets a book's links in stored order.

Keeping a replaced record's place would give a one-book lookup stored order. Map.set already keeps it, so put would only need to re-file the index keys that change.

Is that worth doing here? If not, 3j can sort by a key of its own, accepting tie-breaks that differ from main's where two links contend for one target.

[alex-rawlings-yyc]

@alex-rawlings-yyc made 2 comments and resolved 2 discussions.
Reviewable status: 8 of 27 files reviewed, all discussions resolved (waiting on alex-rawlings-yyc and imnasnainaec).

---

src/__tests__/utils/json-copy.test.ts line 7 at r3 (raw file):

Previously, imnasnainaec (D. Ror.) wrote…

⛏️ Could we add some other atomic types in this test value?

Done; the value now also carries a number, a boolean and null.

---

src/services/recordTable.ts line 107 at r3 (raw file):

Previously, imnasnainaec (D. Ror.) wrote…

❓ From 3j (#449): this line changes the order lookups return records in, and the re-anchor pass depends on that order.

• Removing before re-filing moves a replaced record to the end of records and of each index set. applyChangeSet keeps a replaced record in place.
• 3j's pass will read a book through findLinksInBooks([book]).
• reanchorAnalysisToBook breaks ties by list order: of two stale links contending for one target, only the earlier revives. So it matches the WebView only when it gets a book's links in stored order.

Keeping a replaced record's place would give a one-book lookup stored order. Map.set already keeps it, so put would only need to re-file the index keys that change.

Is that worth doing here? If not, 3j can sort by a key of its own, accepting tie-breaks that differ from main's where two links contend for one target.

Done. put keeps a replaced record's place and re-files only the keys that change, so a one-book lookup returns that book's links in stored order, which DraftRepository's own tests pin. The seam still promises no order, so a later backend needn't keep it.

[imnasnainaec]

@imnasnainaec partially reviewed 18 files.
Reviewable status: 24 of 27 files reviewed, 1 unresolved discussion (waiting on alex-rawlings-yyc).

[imnasnainaec]

@imnasnainaec reviewed 4 files and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on alex-rawlings-yyc).