Add phrase spans, page-built token maps and undo step locations

AGENTS.md

@@ -95,7 +95,7 @@ Saved projects are listed through [ProjectSummaries](src/services/projectSummari
 
 [AnalysisSessions](src/services/analysisSessions.ts) keeps the host's sessions, behind the `openSession`, `closeSession` and `setAnalysisLanguage` commands. A session is a handle on either one source project's draft, paired with the interlinear project it saves to, or a read-only Paratext 9 import. A source has at most one session: opening another closes it. Opening a session starts building its repository's indexes ahead of the session's first command. Opening also starts a fresh whole-text read of the source, which lasts until the session closes. Each session runs its commands one at a time through `run`. A command that changes stored analysis goes through `runEdit` instead, which a read-only session refuses. An edit reaches the repository as a `ChangeSet` of whole-record puts and deletes, applied through the session's [EditHistory](src/services/editHistory.ts) so that undo can revert it; boundary changes ride in the change set's scope record. Re-anchor passes are the exception. As the read delivers each book, `reanchorBook` re-anchors the draft to it through `run`; `reanchorMissingBooks` does the same for each linked book the text lacks. Both skip a read-only session. A pass to a book's edited text reports nothing itself; `reportScriptureEdit` reports the edit, with what its passes changed, as one change that names the book whole. A pass is no undo step and leaves the unsaved mark as it was; the history records it, so that an undo or redo re-anchors what it restores. The WebView doesn't use sessions yet.
 
-An edit, undo or redo replies with a `CommandReply`: the `ChangeSet` it applied, and the pool buckets of the forms it touched, read by [sessionPages](src/services/sessionPages.ts). A command sent again under a `changeId` the session has applied gets its first reply and applies nothing. Each change a session applies, re-anchor passes included, is described by [describeChange](src/services/analysisChanges.ts) and goes out as the session form of `onAnalysisChanged`: its segment ranges by book, analysis ids and cross-book queries. A book the host keeps no tokenized copy of is named whole. `getBookOutline`, `getSegments` and `getTextReadStatus` also take a `{ sessionId }` address; that form segments the text by the draft's boundaries and adds the analysis that `sessionPages` reads.
+An edit, undo or redo replies with a `CommandReply`: the `ChangeSet` it applied, and the pool buckets of the forms it touched, read by [sessionPages](src/services/sessionPages.ts). A command sent again under a `changeId` the session has applied gets its first reply and applies nothing. Each change a session applies, re-anchor passes included, is described by [describeChange](src/services/analysisChanges.ts) and goes out as the session form of `onAnalysisChanged`: its segment ranges by book, analysis ids and cross-book queries. A book the host keeps no tokenized copy of is named whole. `getBookOutline`, `getSegments` and `getTextReadStatus` also take a `{ sessionId }` address; that form segments the text by the draft's boundaries and adds the analysis that `sessionPages` reads. Its pages also span each phrase they touch with every member token, wherever it sits. Each undo step carries the place it was made at, or a summary of a step made at no one place, as [editSteps](src/services/editSteps.ts) derives them, and an undo or redo hands its step back. On the WebView side, [page-indexes](src/utils/page-indexes.ts) builds the token-level lookups and the boundary guard from the pages held and their spans, and finds the segment of a target whose page hasn't loaded; nothing calls it until Phase 4.
 
 Host commands live in [src/services/commands/](src/services/commands/), one module per family. A command stages its changes in a [ChangeBuffer](src/services/commands/changeBuffer.ts), whose reads answer as the repository will once those changes apply, and `AnalysisSessions.edit` applies what it staged as one undo step after the approval rule passes. Each command copies the reducer it replaces, step for step, and its parity tests run one input through both with [runParity](src/__tests__/services/commands/parity.ts). A boundary command replaces no reducer; it copies the WebView's boundary dispatch, phrase breaks included, and `runBoundaryParity` runs one edit through both. The session then re-anchors the book's analysis to the new segments in the same undo step, as the WebView's pass does after a boundary edit. An unsaved-change discard copies what the WebView's discard saves, and `runDiscardParity` runs one discard through both. The discard re-anchors each book it rewrites in the same undo step. A reducer or dispatch change in Phase 3 updates its command and parity tests in the same PR.
 

perf/pages/cli.ts

@@ -37,11 +37,14 @@ import type {
   RecordChanges,
   WarningLogger,
 } from '../../src/types/analysis-repository';
-import type { SegmentPage } from '../../src/types/segment-pages';
+import type { OutlineSegment, SegmentPage } from '../../src/types/segment-pages';
 import { isWordToken } from '../../src/types/type-guards';
 import { bookOfRef } from '../../src/utils/analysis-book';
 import { normalizeSurfaceForm } from '../../src/utils/analysis-identity';
-import { bookIndexesOf } from '../../src/utils/book-indexes';
+import { bookIndexesOf, buildBookIndexes } from '../../src/utils/book-indexes';
+import { buildPageIndexes, outlineRangeOf } from '../../src/utils/page-indexes';
+import { computeSplitFreeRefs, sortByDocOrder } from '../../src/utils/phrase-arc';
+import { phraseSurfaceForm } from '../../src/utils/phrase-text';
 import { reanchorSegmentation, segmentationForBook } from '../../src/utils/segmentation';
 import {
   orphanedCardCountAt,
@@ -249,9 +252,12 @@ async function checkPages(
   repository: DraftRepository,
   analysis: TextAnalysis,
   books: readonly Book[],
-): Promise<{ pages: number; mismatches: string[] }> {
+  viewBook: string,
+): Promise<{ pages: number; mismatches: string[]; phrasesCrossing: number; jumps: number }> {
   const mismatches: string[] = [];
   let pages = 0;
+  let phrasesCrossing = 0;
+  let jumps = 0;
   // eslint-disable-next-line no-restricted-syntax -- books are read one at a time to bound memory
   for (const book of books) {
     const expectedFor = expectedPages(analysis, book);
@@ -314,8 +320,112 @@ async function checkPages(
       if (differing.length > 0)
         mismatches.push(`${book.bookRef} ${start}: ${differing.map(([name]) => name).join(', ')}`);
     }
+    if (book.bookRef === viewBook) {
+      const held: SegmentPage[] = [];
+      for (let index = 0; index < book.segments.length; index += 2) {
+        // eslint-disable-next-line no-await-in-loop -- calls for one draft must not overlap
+        held.push(await readSegmentPage(repository, book, index, index + 1));
+      }
+      const bookPhrases = analysis.phraseAnalysisLinks.filter(
+        ({ tokens }) => bookOfRef(tokens[0].tokenRef) === book.bookRef,
+      );
+      phrasesCrossing += checkPageIndexes(book, outline.segments, held, bookPhrases, mismatches);
+      // A jump to a token or segment whose page is not held loads the page that holds it.
+      for (let index = 1; index < book.segments.length; index += 20) {
+        const segment = book.segments[index];
+        const [token] = segment.tokens.filter(isWordToken);
+        const targets = token ? [token.ref, segment.id] : [segment.id];
+        // eslint-disable-next-line no-restricted-syntax -- calls for one draft must not overlap
+        for (const target of targets) {
+          const range = outlineRangeOf(outline.segments, target);
+          // eslint-disable-next-line no-await-in-loop -- calls for one draft must not overlap
+          const page = range && (await readSegmentPage(repository, book, range.start, range.end));
+          const landed = page && buildPageIndexes(outline.segments, [...held, page]);
+          const holds =
+            target === segment.id
+              ? landed?.segmentById.has(target)
+              : landed?.tokenSegmentMap.get(target) === segment.id;
+          if (!holds) mismatches.push(`${book.bookRef} jump to ${target}`);
+          jumps += 1;
+        }
+      }
+    }
   }
-  return { pages, mismatches };
+  return { pages, mismatches, phrasesCrossing, jumps };
+}
+
+/**
+ * Checks that lookups built from the pages `held` read as the whole book's lookups do. The check
+ * covers each phrase's text, arc order, split previews and segments. It also covers the boundary
+ * guard on the held tokens and where each token's segment lies in the outline.
+ *
+ * @returns How many phrases that the held pages name reach a segment that no held page holds.
+ */
+function checkPageIndexes(
+  book: Book,
+  outline: readonly OutlineSegment[],
+  held: readonly SegmentPage[],
+  bookPhrases: readonly PhraseAnalysisLink[],
+  mismatches: string[],
+): number {
+  const whole = buildBookIndexes(book.segments);
+  const paged = buildPageIndexes(outline, held);
+  const links = [
+    ...new Map(
+      held.flatMap((page) => page.analysis?.phraseAnalysisLinks ?? []).map((l) => [l.id, l]),
+    ).values(),
+  ];
+  let crossing = 0;
+  links.forEach((link) => {
+    const refs = link.tokens.map(({ tokenRef }) => tokenRef);
+    if (refs.some((tokenRef) => !paged.segmentById.has(whole.tokenSegmentMap.get(tokenRef) ?? '')))
+      crossing += 1;
+    const differs = (name: string, a: unknown, b: unknown) => {
+      if (JSON.stringify(a) !== JSON.stringify(b))
+        mismatches.push(`${book.bookRef} phrase ${link.id}: ${name}`);
+    };
+    differs('text', phraseSurfaceForm(link.tokens, paged), phraseSurfaceForm(link.tokens, whole));
+    differs(
+      'arc order',
+      sortByDocOrder(link.tokens, paged.tokenDocOrder),
+      sortByDocOrder(link.tokens, whole.tokenDocOrder),
+    );
+    differs(
+      'split previews',
+      refs.map((tokenRef) => computeSplitFreeRefs(link, tokenRef, paged.tokenDocOrder)),
+      refs.map((tokenRef) => computeSplitFreeRefs(link, tokenRef, whole.tokenDocOrder)),
+    );
+    differs(
+      'segments',
+      refs.map((tokenRef) => paged.tokenSegmentMap.get(tokenRef)),
+      refs.map((tokenRef) => whole.tokenSegmentMap.get(tokenRef)),
+    );
+  });
+  const guarded = new Set<string>();
+  bookPhrases.forEach(({ tokens }) => {
+    const orders = tokens.flatMap(({ tokenRef }) => whole.tokenDocOrder.get(tokenRef) ?? []);
+    for (let i = Math.min(...orders) + 1; i <= Math.max(...orders); i += 1)
+      guarded.add(whole.wordRefByOrder[i]);
+  });
+  const heldRefs = held.flatMap((page) =>
+    page.segments.flatMap((segment) => segment.tokens.map((token) => token.ref)),
+  );
+  if (
+    JSON.stringify(heldRefs.filter((tokenRef) => guarded.has(tokenRef))) !==
+    JSON.stringify(heldRefs.filter((tokenRef) => paged.straddledBoundaryRefs.has(tokenRef)))
+  )
+    mismatches.push(`${book.bookRef} boundary guard`);
+  const misplaced = book.segments.flatMap((segment, index) =>
+    segment.tokens.flatMap(({ ref }) => {
+      const range = outlineRangeOf(outline, ref);
+      return range?.start === index && range.end === index + 1 ? [] : [ref];
+    }),
+  );
+  if (misplaced.length > 0)
+    mismatches.push(
+      `${book.bookRef} outline places ${misplaced.length} tokens elsewhere, ${misplaced[0]} first`,
+    );
+  return crossing;
 }
 
 /** The book segmented by `delta`, its segments kept per delta so indexes over them are built once. */
@@ -739,15 +849,15 @@ async function checkTier(tier: TierSpec): Promise<boolean> {
   const storage = memoryStorage();
   await new DraftRepository(storage, draft.sourceProjectId, SILENT).replaceAll(draft);
   const repository = new DraftRepository(storage, draft.sourceProjectId, SILENT);
-  const pages = await checkPages(repository, draft.analysis, books);
+  const pages = await checkPages(repository, draft.analysis, books, tier.viewBook);
 
   const links = draft.analysis.tokenAnalysisLinks.length;
   // Each change rereads the whole draft twice, so larger tiers get fewer.
   const steps = Math.min(200, Math.max(10, Math.round(2e6 / links)));
   const events = await checkEvents(draft, books, viewBook, steps);
 
   console.log(
-    `${tier.name.padEnd(15)} ${pages.pages} pages, ${pages.mismatches.length} differ  ${events.changes} changes, ${events.mismatches.length} differ  open ${timing.open.toFixed(0)} ms, first page ${timing.first.toFixed(0)} ms, next ${timing.next.toFixed(1)} ms`,
+    `${tier.name.padEnd(15)} ${pages.pages} pages, ${pages.mismatches.length} differ (${pages.phrasesCrossing} phrases reach unheld segments, ${pages.jumps} jumps)  ${events.changes} changes, ${events.mismatches.length} differ  open ${timing.open.toFixed(0)} ms, first page ${timing.first.toFixed(0)} ms, next ${timing.next.toFixed(1)} ms`,
   );
   [...pages.mismatches, ...events.mismatches].slice(0, 20).forEach((m) => console.log(`  ${m}`));
   return pages.mismatches.length === 0 && events.mismatches.length === 0;

src/__tests__/components/AnalysisCatalogPanel.test.tsx

@@ -22,7 +22,7 @@ import { AnalysisStoreProvider, useGlossDispatch } from '../../components/Analys
 import { SourceTextContext } from '../../components/SourceTextContext';
 import { InterlinearNavProvider, useInterlinearNav } from '../../components/InterlinearNavContext';
 import type { SourceText } from '../../hooks/useSourceTextReader';
-import type { StepSummary } from '../../hooks/useDraftProject';
+import type { StepSummary } from '../../types/edit-step';
 import { emptyAnalysis } from '../../types/empty-factories';
 import type { HeadingPlacement } from '../../utils/analysis-query';
 import { defaultScrRef, FIXTURE_STAMPS, makeScrollGroupHook } from '../test-helpers';
@@ -2457,6 +2457,24 @@ describe('AnalysisCatalogPanel', () => {
         expect(saved.tokenAnalyses[0]).not.toHaveProperty('morphemesStale');
         expect(saved.tokenAnalyses[0].morphemes?.map((m) => m.form)).toEqual(['λογγ', 'ος']);
       });
+
+      it('summarizes keeping the breakdown as an edit to its analysis', async () => {
+        const { asOneStep, summaries } = spyOnSteps();
+        renderPanel({ analysis: STALE_BREAKDOWN, asOneStep });
+
+        const row = await expandRow('ta-1');
+        await openBreakdown(row);
+        await userEvent.click(within(rowFor('ta-1')).getByTestId('morpheme-breakdown-save'));
+
+        expect(summaries).toStrictEqual([
+          {
+            kind: 'catalogEdit',
+            form: SHARED.tokenAnalyses[0].surfaceText,
+            analysisId: 'ta-1',
+            survivingAnalysisId: 'ta-1',
+          },
+        ]);
+      });
     });
 
     it('leaves the breakdown alone when the editor is canceled', async () => {

src/__tests__/components/AnalysisStore.test.tsx

@@ -1462,6 +1462,18 @@ describe('useConfirmMorphemesDispatch', () => {
     const saved: TextAnalysis = onSave.mock.lastCall[0];
     expect(saved.tokenAnalyses[0]).not.toHaveProperty('morphemesStale');
   });
+
+  it('saves the kept breakdown at the token it was kept from', () => {
+    const onSave = jest.fn();
+    const { result } = renderStoreHook(() => useConfirmMorphemesDispatch(), {
+      initialAnalysis: STALE_BREAKDOWN,
+      onSave,
+    });
+
+    act(() => result.current('ta-1', 'tok-1'));
+
+    expect(onSave).toHaveBeenLastCalledWith(expect.anything(), 'tok-1');
+  });
 });
 
 describe('useUnapprovedTokenAnalysis', () => {

src/__tests__/components/TokenChip.test.tsx

@@ -861,7 +861,7 @@ describe('TokenChip', () => {
       await userEvent.click(screen.getByRole('button', { name: 'mock-edit-breakdown' }));
       await userEvent.click(screen.getByRole('button', { name: 'mock-confirm' }));
 
-      expect(confirm).toHaveBeenCalledWith('ta-1');
+      expect(confirm).toHaveBeenCalledWith('ta-1', 'GEN 1:1:0');
     });
 
     it('offers no keeping for a breakdown that is not stale', async () => {

src/__tests__/hooks/useDraftProject.test.ts

@@ -6,9 +6,9 @@ import type { DraftProject, TextAnalysis } from 'interlinearizer';
 import { FIXTURE_STAMPS, makeTokenLink } from '../test-helpers';
 import useDraftProject, {
   type DraftContent,
-  type StepSummary,
   type UseDraftProjectResult,
 } from '../../hooks/useDraftProject';
+import type { StepSummary } from '../../types/edit-step';
 import { emptyAnalysis } from '../../types/empty-factories';
 import { CURRENT_MODEL_VERSION } from '../../types/model-version';