diff --git a/AGENTS.md b/AGENTS.md index bfb75a00b..92a1a7ca5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -290,9 +290,9 @@ exists. while the config readers do light I/O, yet both use named exports because neither is an operation surface. - **`links.ts` is the deliberate edge case** — a pure parser that nonetheless - exports a single `links` namespace, _not_ for the service-grouping reason above - but to wall off its `/g` grammar regexes (shared `lastIndex` footgun) behind - position-safe methods. + exports a single `links` namespace to keep link recognition private behind + position-safe methods. Its scanners own their cursors per call; the remaining + `/g` inline-code regex is used through `matchAll` and `replace`. `vault-filesystem.ts` illustrates the nuance within one module: its high-level data API is grouped under `vaultFs`, but its low-level shared primitives @@ -976,6 +976,11 @@ Two naming layers — MCP (JSON wire format) and TypeScript (internal): suite (`src/__tests__/docker/`, see "Remote image boot tests — when to add"); the branches inside one script stay in that script's test file. +- New test support files go in subfolders of their `__tests__/` folder: + a module that a test imports or runs (a harness, an input generator, + an oracle, a script the test spawns) in `helpers/`, and data that the + code under test reads (a sample vault, JSON vectors, a stub binary) in + `fixtures/`. - Separate `it()` blocks over callback-pattern `it.each` when assertions are structurally different. `it.each` is for identical assertion shapes, with labeled case objects and `$label` in the title, never diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 40e6b375d..f3cd0960f 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -121,6 +121,13 @@ graph TB Embeddings persist across rebuilds to reuse unchanged content and can be regenerated from the vault. +### Note-content parsers + +The `obsidian-markdown/` parsers read text straight from vault files, so one crafted note must not be able to stall the server. Their work grows linearly with input size (sorting canvas nodes adds a log factor), and adversarial tests check that bound at doubling input sizes. Two exceptions remain: + +- **Large canvases:** groups nest only while `content nodes × groups + groups²` stays at most 16,000,000 (2,800 groups with 2,800 nodes fit). A larger canvas renders flat, still listing every node, group label and connection. +- **Body tags:** the tag parser can still slow quadratically, so it runs in a worker with a time limit (see **Body tags column** → Worker under [Property discovery + daily notes](#property-discovery--daily-notes)). + ## MCP Tools Common metadata on all discovery tools (`vault_search`, `vault_search_by_tag`, `vault_search_by_folder`, `vault_recent_notes`, `vault_search_by_property`, `vault_find_orphans` — introduced in the subsections below): @@ -348,7 +355,7 @@ Link queries use a `links` table populated during indexing: `vault_read_file` reads non-markdown vault files, dispatching on the file's kind to the most useful representation per type. `obsidian-markdown/file-kinds.ts` maps the lowercased extension to a kind, so `Report.PDF` is a PDF. Reads, indexing, the note mover and the link tools take every non-markdown file's kind from it: 1. **Images** (`.png`/`.jpg`/`.jpeg`/`.gif`/`.webp`) return an MCP `image` content block plus a one-line metadata text block: the path, the delivered format, dimensions and bytes, and, when the image was recompressed, its original dimensions and bytes. A shared fit-to-byte-budget pipeline (`utils/fit-image-to-byte-budget.ts`, built on sharp) makes oversized images deliverable: EXIF auto-orient → resize long edge to ≤1568px → walk a fixed quality ladder (JPEG via mozjpeg for opaque images, WebP for alpha — PNG has no quality knob) → shrink dimensions by √(budget/actual) if the ladder floor still exceeds the budget. Deterministic and terminating (bounded attempts, 64px floor); sharp's default `limitInputPixels` stays active as the decompression-bomb guard. The budget (`MAX_IMAGE_OUTPUT_BYTES`, default 48 KiB binary) is sized for the tightest mainstream client cap. -2. **Canvas** (`.canvas`) linearizes to markdown via the pure `obsidian-markdown/canvas.ts` parser ([JSON Canvas 1.0](https://jsoncanvas.org)): group membership by spatial rect containment (innermost group wins; equal rects tiebreak deterministically by id), nodes in reading order (y, then x), and an edge list with node ids resolved to display names. Lenient parsing — unknown properties ignored, malformed entries skipped. `raw: true` skips the linearizer and returns the JSON source verbatim for full structural fidelity. Canvas content is also FTS-indexed (linearized text in `file_content` + `file_content_fts` tables, gated behind `FILE_TOOLS_ENABLED`) so canvas files appear in `vault_search` results with `kind: "file"`. Canvas `file`-type node references are extracted unconditionally into the `links` table for graph queries regardless of the feature gate. +2. **Canvas** (`.canvas`) linearizes to markdown via the pure `obsidian-markdown/canvas.ts` parser ([JSON Canvas 1.0](https://jsoncanvas.org)): group membership by spatial rect containment within the [nesting limit](#note-content-parsers) (innermost group wins; equal rects tiebreak deterministically by id), nodes in reading order (y, then x), and an edge list with node ids resolved to display names. Lenient parsing — unknown properties ignored, malformed entries skipped. `raw: true` skips the linearizer and returns the JSON source verbatim for full structural fidelity. Canvas content is also FTS-indexed (linearized text in `file_content` + `file_content_fts` tables, gated behind `FILE_TOOLS_ENABLED`) so canvas files appear in `vault_search` results with `kind: "file"`. Canvas `file`-type node references are extracted unconditionally into the `links` table for graph queries regardless of the feature gate. 3. **Text formats** (`.svg`/`.json`/`.txt`/`.csv`/`.xml`/`.log`/`.yaml`/`.yml`/`.base`) pass through as text, without a leading byte-order mark, capped at a fixed 100 KiB output size (explicit error over silent truncation). They skip the NUL-byte check in item 5, so a NUL-padded log still reads. `start_line`/`limit` page any text result — text files, canvas renditions, PDF-extracted text — as a 1-based line window preceded by a window-metadata block stating the range, total line count, and next `start_line`; the cap applies to the window, and an unpaged read is byte-exact apart from that mark. 4. **PDFs** (`.pdf`) return structured markdown reconstructed from layout-aware extraction (`unpdf`, based on Mozilla's PDF.js): - A document metadata header (title, page count, link count) diff --git a/src/vault-mcp/obsidian-markdown/__tests__/canvas.test.ts b/src/vault-mcp/obsidian-markdown/__tests__/canvas.test.ts index 4325741c8..03e72e0bf 100644 --- a/src/vault-mcp/obsidian-markdown/__tests__/canvas.test.ts +++ b/src/vault-mcp/obsidian-markdown/__tests__/canvas.test.ts @@ -1,9 +1,11 @@ -import { describe, it, expect } from "vitest" +import { describe, it, expect, vi } from "vitest" +import { readParserComplexityReport } from "./helpers/frontmatter-canvas-adversary.js" import { linearizeCanvas, extractCanvasFileLinks, extractCanvasCardTexts, rewriteCanvasLinks, + canNestCanvasGroups, } from "../canvas.js" /** Minimal node factories — geometry defaults keep tests focused on the @@ -633,3 +635,238 @@ describe("rewriteCanvasLinks", () => { expect(rewriteInvalidCanvas).toThrow(/^invalid \.canvas JSON: /) }) }) + +const groupNode = (index: number): Record => ({ + id: `g${String(index).padStart(5, "0")}`, + type: "group", + label: `Group ${index}`, + x: index * 30, + y: 0, + width: 20, + height: 20, +}) + +const separatedCanvas = (counts: { + groups: number + contentNodes: number +}): { + json: string + nested: string + flat: string +} => { + const groups = Array.from({ length: counts.groups }, (_, index) => groupNode(index)) + const contentNodes = Array.from({ length: counts.contentNodes }, (_, index) => + textNode({ + id: `n${index}`, + text: `Text ${index}`, + x: index * 30 + 1, + y: 1, + width: 1, + height: 1, + }), + ) + const overview = `# Canvas: ${counts.groups + counts.contentNodes} nodes, 0 edges` + const contentSections = contentNodes.map((_, index) => `[text]\nText ${index}`) + const groupSections = groups.map((_, index) => `## Group: Group ${index}`) + const nestedGroupSections = groupSections.flatMap((heading, index) => { + const content = contentSections[index] + return content ? [heading, content] : [heading] + }) + return { + json: canvasJson([...contentNodes, ...groups]), + nested: [overview, ...contentSections.slice(counts.groups), ...nestedGroupSections].join( + "\n\n", + ), + flat: [overview, ...groupSections, ...contentSections].join("\n\n"), + } +} + +describe("canvas ownership work and rendering depth", () => { + it("renders nested duplicate IDs once in a parent-killed child", () => { + expect(readParserComplexityReport("canvas-cycle")).toEqual({ + fixture: "canvas-cycle", + node: process.version, + semantics: "exact", + }) + }, 35_000) + + it("renders each child only under its owning node when parent IDs repeat", () => { + const nodes = [ + { ...groupNode(0), id: "a", label: "First" }, + { ...groupNode(0), id: "a", label: "Second", x: 100 }, + { ...groupNode(0), id: "b", label: "Child", x: 1, y: 1, width: 10, height: 10 }, + textNode({ id: "text", text: "Body", x: 2, y: 2, width: 1, height: 1 }), + ] + expect(linearizeCanvas(canvasJson(nodes))).toBe( + [ + "# Canvas: 4 nodes, 0 edges", + "## Group: First", + "### Group: Child", + "[text]\nBody", + "## Group: Second", + ].join("\n\n"), + ) + }) + + it("renders 28 duplicate-ID groups once within the parent watchdog", () => { + expect(readParserComplexityReport("canvas-duplicate-ids")).toEqual({ + fixture: "canvas-duplicate-ids", + node: process.version, + inputBytes: 2590, + nodeCount: 28, + headingCount: 28, + medianMs: expect.any(Number), + semantics: "exact", + }) + }, 35_000) + + it.each([ + { fixture: "canvas-cross", copies: 1 }, + { fixture: "canvas-multiple", copies: 5 }, + ])( + "bounds $fixture at the ownership budget with complete output checks", + ({ fixture, copies }) => { + expect(readParserComplexityReport(fixture)).toEqual({ + fixture, + node: process.version, + checksPerCanvas: 16_000_000, + copies, + singleMedianMs: expect.any(Number), + medianMs: expect.any(Number), + semantics: "exact", + }) + }, + 35_000, + ) + + it.each([ + { label: "identical-rectangle depth", fixture: "canvas-depth" }, + { label: "content-count ownership", fixture: "canvas-flat" }, + ])( + "bounds the $label adversary in a parent-killed child", + ({ fixture }) => { + expect(readParserComplexityReport(fixture)).toEqual({ + fixture, + node: process.version, + medianMs: expect.any(Number), + semantics: "exact", + }) + }, + 35_000, + ) + + it.each([ + { label: "zero groups", groups: 0, contentNodes: 100_000, nested: true }, + { label: "one group at budget", groups: 1, contentNodes: 15_999_999, nested: true }, + { + label: "one group one check above budget", + groups: 1, + contentNodes: 16_000_000, + nested: false, + }, + { label: "both axes at budget", groups: 2000, contentNodes: 6000, nested: true }, + { label: "both axes above budget", groups: 2000, contentNodes: 6001, nested: false }, + { label: "group axis at budget", groups: 4000, contentNodes: 0, nested: true }, + { label: "group axis above budget", groups: 4001, contentNodes: 0, nested: false }, + ])("uses both axes for $label without allocating nodes", ({ groups, contentNodes, nested }) => { + expect(canNestCanvasGroups({ groups, contentNodes })).toBe(nested) + }) + + it.each([ + { groups: 2000, contentNodes: 2000, nested: true }, + { groups: 2000, contentNodes: 6000, nested: true }, + { groups: 2000, contentNodes: 6001, nested: false }, + { groups: 4000, contentNodes: 4000, nested: false }, + { groups: 8000, contentNodes: 8000, nested: false }, + { groups: 8000, contentNodes: 8001, nested: false }, + { groups: 2000, contentNodes: 100_000, nested: false }, + { groups: 4001, contentNodes: 1, nested: false }, + { groups: 0, contentNodes: 4, nested: true }, + ])( + "renders every section with $groups groups and $contentNodes content nodes", + ({ groups, contentNodes, nested }) => { + const fixture = separatedCanvas({ groups, contentNodes }) + expect(linearizeCanvas(fixture.json)).toBe(nested ? fixture.nested : fixture.flat) + }, + ) + + it("skips every ownership scan above budget before constructing group maps", () => { + const fixture = separatedCanvas({ groups: 2000, contentNodes: 6001 }) + const nativeFilter = Array.prototype.filter + // Ownership filters scan the 2,000-group array; node parsing filters scan all 8,001 nodes. + let ownershipScans = 0 + const filter = vi.spyOn(Array.prototype, "filter").mockImplementation(function ( + this: unknown[], + predicate, + thisArg, + ) { + if (this.length === 2000) ownershipScans++ + return nativeFilter.call(this, predicate, thisArg) + }) + try { + expect(linearizeCanvas(fixture.json)).toBe(fixture.flat) + expect(ownershipScans).toBe(0) + } finally { + filter.mockRestore() + } + }) + + it("renders a 2,000-group identical-rectangle chain once with H6 capping", () => { + const groups = Array.from({ length: 2000 }, (_, index) => ({ ...groupNode(index), x: 0 })) + const expectedGroups = groups + .toReversed() + .map((_, index) => `${"#".repeat(Math.min(2 + index, 6))} Group: Group ${1999 - index}`) + expect(linearizeCanvas(canvasJson(groups))).toBe( + ["# Canvas: 2000 nodes, 0 edges", ...expectedGroups].join("\n\n"), + ) + }) + + it("preserves flat labels, reading order and all connection names and labels", () => { + const padding = Array.from({ length: 4001 }, (_, index) => ({ + ...groupNode(index), + label: undefined, + y: 10, + })) + const nodes = [ + { ...groupNode(0), id: "group", label: "Named", x: 0, y: -2 }, + textNode({ id: "text", x: 0, y: -1, text: "## Title\nBody" }), + { + id: "file", + type: "file", + x: 0, + y: 0, + width: 1, + height: 1, + file: "Folder/Note.md", + subpath: "#Heading", + }, + { id: "link", type: "link", x: 0, y: 1, width: 1, height: 1, url: "https://example.com" }, + { id: "unknown", type: "custom", x: 0, y: 2, width: 1, height: 1 }, + { id: "invalid", type: "text", text: "Dropped" }, + ...padding, + ] + const edges = [ + { fromNode: "group", toNode: "text", label: "names" }, + { fromNode: "file", toNode: "link", label: "" }, + { fromNode: "unknown", toNode: "absent", label: "missing" }, + { fromNode: "g00000", toNode: "group" }, + { fromNode: "invalid" }, + ] + expect(linearizeCanvas(canvasJson(nodes, edges))).toBe( + [ + "# Canvas: 4006 nodes, 4 edges", + "## Group: Named", + "[text]\n## Title\nBody", + "[file] → Folder/Note.md#Heading", + "[link] → https://example.com", + "[custom]", + ...padding.map(() => "## Group: (unlabeled)"), + "## Connections", + "Named → Title (names)", + "Note.md → https://example.com", + '(custom node) → (missing node "absent") (missing)', + "(unlabeled group) → Named", + ].join("\n\n"), + ) + }) +}) diff --git a/src/vault-mcp/obsidian-markdown/__tests__/frontmatter.test.ts b/src/vault-mcp/obsidian-markdown/__tests__/frontmatter.test.ts index bb24cb5d4..fc641aa13 100644 --- a/src/vault-mcp/obsidian-markdown/__tests__/frontmatter.test.ts +++ b/src/vault-mcp/obsidian-markdown/__tests__/frontmatter.test.ts @@ -2,6 +2,7 @@ import matter from "gray-matter" import { DateTime } from "luxon" import { parseDocument, stringify as stringifyYaml } from "yaml" import { describe, it, expect, vi } from "vitest" +import { readParserComplexityReport } from "./helpers/frontmatter-canvas-adversary.js" import { findBodyStartLine, findWrittenBodyStartLine, @@ -1078,3 +1079,234 @@ describe("mergeFrontmatter", () => { }) }) }) + +describe("linear duplicate-key checking", () => { + const snapshot = (note: string): unknown => { + const strictReading = (read: (source: string) => unknown): unknown => { + try { + return { value: read(note) } + } catch (error) { + if (!(error instanceof UnsupportedPropertiesBlockError)) throw error + return { error: { kind: error.kind, message: error.message } } + } + } + const tolerant = parseNoteSkippingUnreadableBlock(note) + return { + strict: strictReading(parseNote), + rewrite: strictReading(parseNoteForRewrite), + tolerant: { + data: tolerant.data, + content: tolerant.content, + error: tolerant.unreadableBlockError + ? { + kind: tolerant.unreadableBlockError.kind, + message: tolerant.unreadableBlockError.message, + } + : null, + }, + } + } + + it.each([ + { label: "unique keys", fixture: "yaml-unique" }, + { label: "late duplicates", fixture: "yaml-duplicate" }, + ])( + "bounds 100,000 $label with a parent watchdog and exact semantic checks", + ({ fixture }) => { + expect(readParserComplexityReport(fixture)).toEqual({ + fixture, + node: process.version, + sizes: [50_000, 100_000], + mediansMs: [expect.any(Number), expect.any(Number)], + semantics: "exact", + }) + }, + 35_000, + ) + + it.each([ + { + label: "malformed flow", + yaml: "{a: 1, a: [unclosed}", + reason: + "at line 2, column 20: Flow sequence in block collection must be sufficiently indented and end with a ]", + }, + { + label: "unclosed quote", + yaml: 'a: 1\na: "unclosed', + reason: 'at line 3, column 13: Missing closing "quote', + }, + { + label: "tab indentation", + yaml: "a: 1\na:\n\tb: 2", + reason: "at line 4, column 1: Tabs are not allowed as indentation", + }, + { + label: "missing value", + yaml: "a: 1\na: 2\nb", + reason: "at line 4, column 1: Implicit map keys need to be followed by map values", + }, + ])( + "refuses mixed duplicate and $label errors with the syntax-first message", + ({ yaml, reason }) => { + const error = { + kind: "invalid-yaml", + message: `properties block is not valid YAML ${reason}`, + } + expect(snapshot(`---\n${yaml}\n---\nbody`)).toEqual({ + strict: { error }, + rewrite: { error }, + tolerant: { data: {}, content: "body", error }, + }) + }, + ) + + it.each( + [ + { label: "ordinary", yaml: "a: 1\na: 2", line: 3, column: 1 }, + { label: "quoted", yaml: 'a: 1\n"a": 2', line: 3, column: 1 }, + { label: "tagged", yaml: "a: 1\n!!str a: 2", line: 3, column: 7 }, + { label: "anchored", yaml: "a: 1\n&key a: 2", line: 3, column: 6 }, + { label: "explicit empty", yaml: "?\n: a\n?\n: b", line: 5, column: 1 }, + { + label: "nested before root", + yaml: "x: 1\nouter:\n a: 1\n a: 2\nx: 2", + line: 5, + column: 3, + }, + { label: "block before value", yaml: "a: 1\na: {b: 1, b: 2}", line: 3, column: 1 }, + { + label: "duplicate pair with a nested block duplicate", + yaml: "a: 1\na:\n b: 1\n b: 2", + line: 3, + column: 1, + }, + { label: "flow after value", yaml: "{a: 1, a: {b: 1, b: 2}}", line: 2, column: 18 }, + { label: "collection key first", yaml: "? {a: 1, a: 2}\n: value", line: 2, column: 10 }, + { label: "numeric equivalents", yaml: "1: first\n1.0: second", line: 3, column: 1 }, + { label: "null equivalents", yaml: "null: first\n~: second", line: 3, column: 1 }, + { label: "boolean equivalents", yaml: "true: first\nTrue: second", line: 3, column: 1 }, + { label: "default merge keys", yaml: "<<: first\n<<: second", line: 3, column: 1 }, + { label: "duplicate before alias conversion", yaml: "a: 1\na: *absent", line: 3, column: 1 }, + { label: "duplicate before shape", yaml: "- {a: 1, a: 2}", line: 2, column: 10 }, + ].flatMap((testCase) => [ + { ...testCase, newline: "LF", separator: "\n" }, + { ...testCase, newline: "CRLF", separator: "\r\n" }, + ]), + )("keeps the exact $label refusal with $newline", ({ yaml, line, column, separator }) => { + const note = `---\n${yaml}\n---\nbody`.replaceAll("\n", separator) + const error = { + kind: "invalid-yaml", + message: `properties block is not valid YAML at line ${line}, column ${column}: Map keys must be unique`, + } + expect(snapshot(note)).toEqual({ + strict: { error }, + rewrite: { error }, + tolerant: { data: {}, content: "body", error }, + }) + }) + + it("retains distinct numeric/string, NaN, alias-node and collection-node keys", () => { + const note = + '---\n1: numeric\n"1": string\n.nan: first\n.nan: second\nsource: &key value\n*key : first alias\n*key : second alias\n? [a, b]\n: first collection\n? [a, b]\n: second collection\n---\nbody' + expect(parseNote(note)).toEqual({ + data: { + "1": "string", + NaN: "second", + source: "value", + value: "second alias", + "[ a, b ]": "second collection", + }, + content: "body", + }) + }) + + it.each([ + { label: "LF", separator: "\n", offset: 9 }, + { label: "CRLF", separator: "\r\n", offset: 13 }, + ])("retains the native explicit-empty-key offset with $label", ({ separator, offset }) => { + const note = ["---", "?", ": a", "?", ": b", "---", "body"].join(separator) + const thrown = catchThrown(() => parseNote(note)) + + if (!(thrown instanceof UnsupportedPropertiesBlockError)) + throw new Error("missing properties refusal") + const { cause } = thrown + + if (!(cause instanceof Error) || !("pos" in cause)) + throw new Error("missing YAML error position") + expect(cause.pos).toEqual([offset, offset + 1]) + }) + + it("matches the native duplicate checker on 10,000 bounded fixed-seed documents", async () => { + const nativeYaml = await vi.importActual("yaml") + const nativeParseDocument = nativeYaml.parseDocument + // Seed 0x739afb02; each map has at most 12 pairs, and nesting is at most three maps. + let seed = 0x739afb02 + const random = (bound: number): number => { + seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0 + return (seed >>> 8) % bound + } + const spellings = [ + "a", + '"a"', + "b", + "null", + "~", + "true", + "True", + "1", + "1.0", + '"1"', + ".nan", + "!!str a", + "&key a", + ] + const warning = vi.spyOn(process, "emitWarning").mockImplementation(() => undefined) + try { + for (let i = 0; i < 10_000; i++) { + const pairs = Array.from( + { length: 1 + random(12) }, + () => `${spellings[random(spellings.length)]}: ${random(10)}`, + ) + // Keep anchors unique while retaining tagged and anchored key positions. + const anchoredPairs = pairs.map((pair, index) => pair.replace("&key ", `&key${index} `)) + const block = anchoredPairs.join("\n") + const flow = `{${anchoredPairs.join(", ")}}` + const templates = [ + block, + flow, + `outer:\n ${block.replaceAll("\n", "\n ")}\nx: 1\nx: 2`, + `{x: 1, outer: ${flow}, x: 2}`, + `outer: 1\nouter:\n ${block.replaceAll("\n", "\n ")}`, + `{outer: 1, outer: ${flow}}`, + `? ${flow}\n: value`, + `?\n: a\n?\n: b`, + `source: &source value\n*source : 1\n*source : 2`, + `? [a, b]\n: first\n? [a, b]\n: second`, + ] + const yaml = templates[random(templates.length)] ?? "" + const note = `---\n${yaml}\n---\nbody`.replaceAll("\n", random(2) ? "\r\n" : "\n") + const { blockText } = splitPropertiesBlock(note) + const nativeErrors = nativeParseDocument(blockText ?? "").errors + expect( + nativeErrors.filter((error) => error.code !== "DUPLICATE_KEY"), + note, + ).toEqual([]) + const actual = snapshot(note) + vi.mocked(parseDocument).mockImplementation((source, options) => + nativeParseDocument(source, { ...options, uniqueKeys: true }), + ) + try { + expect(actual, note).toEqual(snapshot(note)) + } finally { + vi.mocked(parseDocument).mockRestore() + // Parser call histories retain entire documents; the differential needs only this case. + vi.clearAllMocks() + } + } + } finally { + warning.mockRestore() + } + // 10,000 documents take about 10 s on a laptop and over 30 s on a loaded CI runner. + }, 120_000) +}) diff --git a/src/vault-mcp/obsidian-markdown/__tests__/headings-linear.test.ts b/src/vault-mcp/obsidian-markdown/__tests__/headings-linear.test.ts new file mode 100644 index 000000000..2cb6d679c --- /dev/null +++ b/src/vault-mcp/obsidian-markdown/__tests__/headings-linear.test.ts @@ -0,0 +1,219 @@ +import { execFileSync } from "node:child_process" +import { readFileSync } from "node:fs" +import { runInNewContext } from "node:vm" +import ts from "typescript" +import { describe, expect, it } from "vitest" +import { parseHeadings, type HeadingInfo } from "../headings.js" +import * as lineParsers from "../lines.js" +import { tasks } from "../tasks.js" + +/** Allows scheduling noise across the full input range. */ +const LINEAR_GROWTH_ALLOWANCE = 2 + +type HeadingParser = (lines: readonly string[]) => HeadingInfo[] + +/** Keeps the former suffix and boundary rules as a small-input oracle. */ +const readLegacyHeadings = ( + source = readFileSync(new URL("../headings.ts", import.meta.url), "utf8"), +): HeadingParser => { + const boundaryStart = source.indexOf(" const completedHeadings =") + const boundaryEnd = source.indexOf("\n}", boundaryStart) + + if (boundaryStart < 0 || boundaryEnd < 0) + throw new Error("heading boundary oracle anchor missing") + const legacyPass = `return collectedHeadings.map((heading, index) => { + const nextSameOrHigher = collectedHeadings.slice(index + 1).find(next => next.level <= heading.level); + return { ...heading, bodyEndLine: nextSameOrHigher?.startLine ?? Math.max(heading.bodyStartLine, trailingCommentBlockStart) }; + });` + const legacySource = source.slice(0, boundaryStart) + legacyPass + source.slice(boundaryEnd) + const closingSource = legacySource.replace( + "stripClosingHashes(matchedText)", + 'matchedText.replace(/(?:^|[ \\t]+)#+[ \\t]*$/, "")', + ) + + if (closingSource === legacySource) throw new Error("closing-hash oracle anchor missing") + const compiled = ts.transpileModule(closingSource, { + compilerOptions: { target: ts.ScriptTarget.ESNext, module: ts.ModuleKind.CommonJS }, + }).outputText + const result: unknown = runInNewContext(`${compiled}\n;exports.parseHeadings`, { + exports: {}, + require: () => lineParsers, + }) + + if (typeof result !== "function") throw new Error("legacy heading parser could not be loaded") + return (lines) => result(lines) +} + +describe("heading traversal compatibility", () => { + it("refuses to compare closing hashes when the frozen oracle substitution misses", () => { + const source = readFileSync(new URL("../headings.ts", import.meta.url), "utf8") + const missingAnchor = source.replace("stripClosingHashes(matchedText)", "matchedText") + + expect(missingAnchor).not.toBe(source) + expect(() => readLegacyHeadings(missingAnchor)).toThrow( + new Error("closing-hash oracle anchor missing"), + ) + }) + + it("preserves full heading ranges and task attribution over fixed-seed documents", () => { + const legacy = readLegacyHeadings() + // The generator advances one deterministic state for every choice. + let state = 0x73a82f19 + const draw = (bound: number): number => { + state = (Math.imul(state, 1664525) + 1013904223) >>> 0 + return state % bound + } + const choices = [ + "# A", + "## B ##", + "###", + "#### C #", + "## Equal", + "Title", + "===", + "---", + "", + "```", + "~~~", + "%%", + "%% inline %%", + "- [ ] Work", + "- [ ] Other 📅 2026-01-02", + "text", + "# x\rtext", + "# x\u2028text", + "# 😀", + "## v2#", + "# a#b", + "# a #b#", + "# ##", + "### ###", + ] + const count = Number(process.env.PARSER_CORPUS_CASES ?? 10_000) + + for (let index = 0; index < count; index += 1) { + const lines = Array.from({ length: draw(24) }, () => choices[draw(choices.length)] ?? "") + const expectedHeadings = legacy(lines) + expect(parseHeadings(lines)).toEqual(expectedHeadings) + const note = lines.join(index % 2 === 0 ? "\n" : "\r\n") + const extracted = tasks.extractTasksFromBody(note) + + for (const task of extracted) { + const expectedHeading = expectedHeadings.findLast( + (heading) => heading.startLine < task.line - 1, + ) + const sourceLine = lines[task.line - 1] ?? "" + const standaloneTask = tasks.extractTasksFromBody(sourceLine)[0] + + expect(task).toEqual({ + ...standaloneTask, + line: task.line, + heading: expectedHeading?.text ?? null, + }) + } + } + }) + + it.each([ + { newline: "\n", label: "LF" }, + { newline: "\r\n", label: "CRLF" }, + ])("keeps word-attached hashes and strips whole hash texts in $label headings", ({ newline }) => { + const lines = ["## v2#", "# a#b", "# a #b#", "# ##", "### ###"] + const expectedHeadings = [ + { text: "v2#", level: 2, startLine: 0, bodyStartLine: 1, bodyEndLine: 1 }, + { text: "a#b", level: 1, startLine: 1, bodyStartLine: 2, bodyEndLine: 2 }, + { text: "a #b#", level: 1, startLine: 2, bodyStartLine: 3, bodyEndLine: 3 }, + { text: "", level: 1, startLine: 3, bodyStartLine: 4, bodyEndLine: 5 }, + { text: "", level: 3, startLine: 4, bodyStartLine: 5, bodyEndLine: 5 }, + ] + const normalizedLines = lineParsers.splitIntoLines(lines.join(newline)) + + expect(readLegacyHeadings()(normalizedLines)).toEqual(expectedHeadings) + expect(parseHeadings(normalizedLines)).toEqual(expectedHeadings) + }) + + it("preserves empty, nested, setext, fence and comment boundaries exactly", () => { + const lines = [ + "- [ ] Before", + "# A", + "- [ ] Under A", + "Child", + "---", + "- [ ] Under Child", + "```", + "# hidden", + "```", + "##", + "- [ ] Under empty", + "# Last ##", + "- [ ] Last task", + "", + "%% settings", + "config", + "%%", + "", + ] + + expect(parseHeadings(lines)).toEqual([ + { text: "A", level: 1, startLine: 1, bodyStartLine: 2, bodyEndLine: 11 }, + { text: "Child", level: 2, startLine: 3, bodyStartLine: 5, bodyEndLine: 9 }, + { text: "", level: 2, startLine: 9, bodyStartLine: 10, bodyEndLine: 11 }, + { text: "Last", level: 1, startLine: 11, bodyStartLine: 12, bodyEndLine: 13 }, + ]) + expect( + tasks.extractTasksFromBody(lines.join("\n")).map(({ line, heading }) => ({ line, heading })), + ).toEqual([ + { line: 1, heading: null }, + { line: 3, heading: "A" }, + { line: 6, heading: "Child" }, + { line: 11, heading: "" }, + { line: 13, heading: "Last" }, + ]) + }) + + it("bounds closing-hash and complete heading-task traversal adversaries in a child", () => { + const headingsUrl = new URL("../headings.ts", import.meta.url).href + const tasksUrl = new URL("../tasks.ts", import.meta.url).href + const output = execFileSync( + process.execPath, + [ + "--import", + "tsx", + "--input-type=module", + "--eval", + ` + import assert from 'node:assert/strict'; + import { parseHeadings } from ${JSON.stringify(headingsUrl)}; + import { tasks } from ${JSON.stringify(tasksUrl)}; + const measurements=[]; + for (const variant of ['closing','traversal']) { + const sizes = variant === 'closing' ? [25000,50000,100000] : [16000,32000,64000]; + const medians=[]; + for (const size of sizes) { + const input = variant === 'closing' ? '# x' + ' '.repeat(size) : '# a\\n'.repeat(size); + const check = () => { + const headings=parseHeadings(input.split('\\n')); + if(variant === 'closing') assert.deepEqual(headings,[{text:'x',level:1,startLine:0,bodyStartLine:1,bodyEndLine:1}]); + else assert.deepEqual(headings,Array.from({length:size},(_,index)=>({text:'a',level:1,startLine:index,bodyStartLine:index+1,bodyEndLine:index+1 === size ? size+1 : index+1}))); + assert.deepEqual(tasks.extractTasksFromBody(input), []); + }; + check(); + const samples=Array.from({length:5},()=>{const start=performance.now();check();return performance.now()-start}).toSorted((a,b)=>a-b); + medians.push(samples[2]); + } + assert.ok(medians.every(value=>value<1000),variant+' exceeded ceiling '+medians); + assert.ok(medians.at(-1)<=Math.max(10,medians[0])*(sizes.at(-1)/sizes[0])*${LINEAR_GROWTH_ALLOWANCE},variant+' scaled superlinearly '+medians); + measurements.push({variant,sizes,medians}); + } + process.stdout.write(JSON.stringify(measurements)); + `, + ], + { encoding: "utf8", timeout: 30_000, killSignal: "SIGKILL" }, + ) + + expect( + JSON.parse(output).map((measurement: { variant: string }) => measurement.variant), + ).toEqual(["closing", "traversal"]) + process.stdout.write(`Heading parser timings ${output}\n`) + }, 35_000) +}) diff --git a/src/vault-mcp/obsidian-markdown/__tests__/helpers/frontmatter-canvas-adversary.ts b/src/vault-mcp/obsidian-markdown/__tests__/helpers/frontmatter-canvas-adversary.ts new file mode 100644 index 000000000..352afb39d --- /dev/null +++ b/src/vault-mcp/obsidian-markdown/__tests__/helpers/frontmatter-canvas-adversary.ts @@ -0,0 +1,266 @@ +import { fileURLToPath } from "node:url" +import assert from "node:assert/strict" +import { performance } from "node:perf_hooks" +import { execFileSync } from "node:child_process" +import { linearizeCanvas } from "../../canvas.js" +import { parseNote, UnsupportedPropertiesBlockError } from "../../frontmatter.js" + +const parserWorkerPath = fileURLToPath(import.meta.url) + +export const readParserComplexityReport = (fixture: string): unknown => { + // Node's parent watchdog kills and reaps a stuck synchronous parser before returning. + const output = execFileSync( + process.execPath, + ["--expose-gc", "--import", "tsx", parserWorkerPath, fixture], + { + encoding: "utf8", + timeout: 30_000, + killSignal: "SIGKILL", + }, + ) + return JSON.parse(output) +} + +const yamlFixture = (size: number, duplicate: boolean): string => { + const entries = Array.from({ length: size }, (_, index) => `k${index}: ${index}`) + return ["---", ...entries, ...(duplicate ? ["k0: duplicate"] : []), "---", "body"].join("\n") +} + +const canvasFixture = (counts: { groups: number; contentNodes: number; shape: string }): string => { + const groups = Array.from({ length: counts.groups }, (_, index) => { + const horizontal = index % 2 === 0 + const crossedGeometry = { + x: horizontal ? -10 : 2, + y: horizontal ? 2 : -10, + width: horizontal ? 100 : 1, + height: horizontal ? 1 : 100, + } + const geometry = (() => { + if (counts.shape === "depth") return { x: 0, y: 0, width: 20, height: 20 } + if (counts.shape === "separated") return { x: index * 30, y: 0, width: 20, height: 20 } + return crossedGeometry + })() + return { + id: `g${String(index).padStart(5, "0")}`, + type: "group", + label: `Group ${index}`, + ...geometry, + } + }) + const nodes = Array.from({ length: counts.contentNodes }, (_, index) => ({ + id: `n${index}`, + type: "text", + text: `Text ${index}`, + x: counts.shape === "separated" ? index * 30 + 1 : 0, + y: counts.shape === "separated" ? 1 : 0, + width: 1, + height: 1, + })) + return JSON.stringify({ nodes: [...groups, ...nodes], edges: [] }) +} + +const measure = (run: () => void): number => { + run() + const samples = Array.from({ length: 5 }, () => { + // Release prior documents before timing the next parse rather than charging their GC to it. + global.gc?.() + const start = performance.now() + run() + return performance.now() - start + }).toSorted((a, b) => a - b) + const median = samples[2] + + if (median === undefined) throw new Error("missing timing median") + return median +} + +const yamlCase = (size: number, duplicate: boolean): (() => void) => { + const note = yamlFixture(size, duplicate) + + if (!duplicate) { + const data = Object.fromEntries( + Array.from({ length: size }, (_, index) => [`k${index}`, index]), + ) + + return () => { + assert.deepEqual(parseNote(note), { data, content: "body" }) + } + } + return () => { + try { + parseNote(note) + } catch (error) { + const expected = `properties block is not valid YAML at line ${size + 2}, column 1: Map keys must be unique` + + if ( + error instanceof UnsupportedPropertiesBlockError && + error.kind === "invalid-yaml" && + error.message === expected + ) + return + throw error + } + throw new Error("late duplicate was accepted") + } +} + +const canvasCase = (counts: { + groups: number + contentNodes: number + shape: string +}): (() => void) => { + const json = canvasFixture(counts) + const groups = Array.from({ length: counts.groups }, (_, index) => `## Group: Group ${index}`) + const content = Array.from({ length: counts.contentNodes }, (_, index) => `[text]\nText ${index}`) + const overview = `# Canvas: ${counts.groups + counts.contentNodes} nodes, 0 edges` + const depthGroups = groups + .toReversed() + .map( + (_, index) => + `${"#".repeat(Math.min(2 + index, 6))} Group: Group ${counts.groups - index - 1}`, + ) + const chainHeadings = (parity: number): string[] => { + return groups + .flatMap((_, index) => (index % 2 === parity ? [index] : [])) + .toReversed() + .map((index, depth) => `${"#".repeat(Math.min(2 + depth, 6))} Group: Group ${index}`) + } + const readingGroups = [...chainHeadings(1), ...chainHeadings(0)] + const nested = [ + overview, + ...content, + ...(counts.shape === "depth" ? depthGroups : readingGroups), + ].join("\n\n") + const expected = + counts.shape === "separated" ? [overview, ...groups, ...content].join("\n\n") : nested + return () => { + const actual = linearizeCanvas(json) + + if (actual !== expected) + throw new Error("canvas rendition differs from the complete expected output") + } +} + +const runFixture = (fixture: string): unknown => { + if (fixture === "canvas-cycle") { + const nodes = [ + { id: "a", type: "group", label: "Outer A", x: 0, y: 0, width: 30, height: 30 }, + { id: "b", type: "group", label: "B", x: 1, y: 1, width: 20, height: 20 }, + { id: "a", type: "group", label: "Inner A", x: 2, y: 2, width: 10, height: 10 }, + ] + assert.equal( + linearizeCanvas(JSON.stringify({ nodes, edges: [] })), + [ + "# Canvas: 3 nodes, 0 edges", + "## Group: Outer A", + "### Group: B", + "#### Group: Inner A", + ].join("\n\n"), + ) + return { fixture, node: process.version, semantics: "exact" } + } + if (fixture === "canvas-duplicate-ids") { + const levels = 14 + const nodes = Array.from({ length: levels }, (_, level) => { + return [0, 1].map((copy) => ({ + id: `g${String(level).padStart(3, "0")}`, + type: "group", + label: `level-${level}-copy-${copy}`, + x: -level, + y: -level, + width: 10 + level * 2, + height: 10 + level * 2, + })) + }).flat() + const json = JSON.stringify({ nodes, edges: [] }) + const firstCopies = Array.from({ length: levels }, (_, depth) => { + return `${"#".repeat(Math.min(2 + depth, 6))} Group: level-${levels - depth - 1}-copy-0` + }) + const secondCopies = Array.from({ length: levels }, (_, level) => { + return `${"#".repeat(Math.min(2 + levels - level - 1, 6))} Group: level-${level}-copy-1` + }) + const expected = ["# Canvas: 28 nodes, 0 edges", ...firstCopies, ...secondCopies].join("\n\n") + const render = (): number => { + const actual = linearizeCanvas(json) + const headingCount = actual.split("\n").filter((line) => line.includes("Group:")).length + assert.equal(headingCount, nodes.length) + assert.equal(actual, expected) + return headingCount + } + const headingCount = render() + const medianMs = measure(() => { + render() + }) + return { + fixture, + node: process.version, + inputBytes: Buffer.byteLength(json), + nodeCount: nodes.length, + headingCount, + medianMs, + semantics: "exact", + } + } + if (fixture === "yaml-unique" || fixture === "yaml-duplicate") { + const duplicate = fixture === "yaml-duplicate" + const smallerMs = measure(yamlCase(50_000, duplicate)) + const largerMs = measure(yamlCase(100_000, duplicate)) + + if (smallerMs > 10 && largerMs > smallerMs * 3) + throw new Error(`YAML scaling exceeded 3x: ${smallerMs} -> ${largerMs}`) + return { + fixture, + node: process.version, + sizes: [50_000, 100_000], + mediansMs: [smallerMs, largerMs], + semantics: "exact", + } + } + if (fixture === "canvas-depth") { + const medianMs = measure(canvasCase({ groups: 2000, contentNodes: 0, shape: "depth" })) + assert.ok(medianMs < 1000, `canvas-depth exceeded ceiling ${medianMs}`) + return { fixture, node: process.version, medianMs, semantics: "exact" } + } + if (fixture === "canvas-flat") { + const medianMs = measure( + canvasCase({ groups: 2000, contentNodes: 100_000, shape: "separated" }), + ) + assert.ok(medianMs < 1000, `canvas-flat exceeded ceiling ${medianMs}`) + return { fixture, node: process.version, medianMs, semantics: "exact" } + } + if (fixture === "canvas-cross" || fixture === "canvas-multiple") { + const run = canvasCase({ groups: 2000, contentNodes: 6000, shape: "cross" }) + const copies = fixture === "canvas-multiple" ? 5 : 1 + const singleMedianMs = measure(run) + const medianMs = + copies === 1 + ? singleMedianMs + : measure(() => { + for (let index = 0; index < copies; index++) run() + }) + + assert.ok(singleMedianMs < 1000, `${fixture} exceeded single-canvas ceiling ${singleMedianMs}`) + assert.ok(medianMs < copies * 1000, `${fixture} exceeded ceiling ${medianMs}`) + assert.ok( + medianMs <= singleMedianMs * copies * 1.5, + `${fixture} scaled superlinearly ${singleMedianMs}, ${medianMs}`, + ) + return { + fixture, + node: process.version, + checksPerCanvas: 16_000_000, + copies, + singleMedianMs, + medianMs, + semantics: "exact", + } + } + throw new Error(`unknown adversarial fixture: ${fixture}`) +} + +if (process.argv[1] === parserWorkerPath) { + const fixture = process.argv[2] + + if (!fixture) throw new Error("missing adversarial fixture") + process.stdout.write(`${JSON.stringify(runFixture(fixture))}\n`) +} diff --git a/src/vault-mcp/obsidian-markdown/__tests__/helpers/link-scanner-adversary.ts b/src/vault-mcp/obsidian-markdown/__tests__/helpers/link-scanner-adversary.ts new file mode 100644 index 000000000..8739f99ba --- /dev/null +++ b/src/vault-mcp/obsidian-markdown/__tests__/helpers/link-scanner-adversary.ts @@ -0,0 +1,63 @@ +import assert from "node:assert/strict" +import { links } from "../../links.js" +import { stripMarkdownSyntax } from "../../plaintext.js" + +const variant = process.argv[2] +const sizes = [20_000, 40_000, 80_000] + +const fixture = (size: number): { text: string; expected: unknown } => { + if (variant === "wiki") { + return { + text: `[[Keep]] ${"[".repeat(size)}`, + expected: [{ text: "[[Keep]]", start: 0, end: 8, kind: "wikilink" }], + } + } + if (variant === "markdown") { + return { + text: `[Keep](Keep) ${"[".repeat(size)}](a`, + expected: [{ text: "[Keep](Keep)", start: 0, end: 12, kind: "markdown" }], + } + } + if (variant === "templater") { + return { text: `[[Keep]] ${"<%".repeat(size / 2)}`, expected: ["Keep"] } + } + if (variant === "callout") { + return { + text: `[[Keep]] ${">[!\n".repeat(size / 4)}`, + expected: `Keep >[!\n${"[!\n".repeat(size / 4 - 1)}`, + } + } + if (variant === "plaintext-markdown") { + const tail = `${"[".repeat(size)}](a` + return { text: `[[Keep]] ${tail}`, expected: `Keep ${tail}` } + } + if (variant === "plaintext-alias") { + const tail = "[[x|".repeat(size / 4) + return { text: `[[Keep|alias]] ${tail}`, expected: `alias ${tail}` } + } + return { text: `[[Keep]] ${"[".repeat(size)}`, expected: `Keep ${"[".repeat(size)}` } +} + +const parse = (text: string): unknown => { + if (variant === "wiki" || variant === "markdown") return links.matchLinksInLine(text) + if (variant === "templater") return links.extractFromBody(text) + return stripMarkdownSyntax(text) +} + +const measurements = sizes.map((size) => { + const { text, expected } = fixture(size) + assert.deepEqual(parse(text), expected) + const samples = Array.from({ length: 5 }, () => { + const started = performance.now() + const result = parse(text) + const elapsed = performance.now() - started + assert.deepEqual(result, expected) + return elapsed + }) + const median = samples.toSorted((a, b) => a - b)[2] + + assert.notEqual(median, undefined) + return { bytes: Buffer.byteLength(text), median } +}) + +process.stdout.write(JSON.stringify({ variant, measurements })) diff --git a/src/vault-mcp/obsidian-markdown/__tests__/helpers/link-scanner-corpus.ts b/src/vault-mcp/obsidian-markdown/__tests__/helpers/link-scanner-corpus.ts new file mode 100644 index 000000000..535943bf8 --- /dev/null +++ b/src/vault-mcp/obsidian-markdown/__tests__/helpers/link-scanner-corpus.ts @@ -0,0 +1,79 @@ +/** Frozen recognizers are independent compatibility oracles for the scanner rewrites. */ +export const legacyWiki = /!?\[\[([^\]#|]+)(?:#[^\]|]*)?(?:\|[^\]]+)?\]\]/g +export const legacyMarkdown = /\[[^\]]*\]\((?![a-zA-Z][a-zA-Z0-9+.-]*:|#)([^)#\s]+)(?:#[^)\s]*)?\)/g +export const legacyTemplater = /<%[-+*_~]?.*?%>/g + +export const legacyPlaintext = (text: string): string => { + return text + .replace(/\[\[([^|\]]+)\|([^\]]+)\]\]/g, "$2") + .replace(/\[\[([^\]]+)\]\]/g, "$1") + .replace(/\[([^\]]+)\]\([^)]+\)/g, "$1") + .replace(/%%[^]*?%%/g, "") + .replace(/^ {0,3}#{1,6}(?:[ \t]+|$)/gm, "") + .replace(/^ {0,3}(?:=+|-+)[ \t]*$/gm, "") + .replace(/\*{1,3}([^*]+)\*{1,3}/g, "$1") + .replace(/_{1,3}([^_]+)_{1,3}/g, "$1") + .replace(/^>\s*\[![^\]]*\]\s*/gm, "") + .replace(/^>\s?/gm, "") +} + +export const scannerCorpora = [ + { label: "wikilinks", alphabet: ["[", "[", "]", "#", "|", "!", "a", " ", "\\", "😀", "\n"] }, + { + label: "Markdown links", + alphabet: ["[", "]", "(", ")", "#", ":", ".", "+", "a", "Z", "0", " ", "\u00a0", "😀", "\n"], + }, + { + label: "Templater", + alphabet: [ + "<", + "%", + "%", + ">", + "-", + "+", + "*", + "_", + "~", + "[", + "]", + "a", + "\r", + "\u2028", + "\u2029", + "\n", + ], + }, + { label: "plaintext aliases", alphabet: ["[", "[", "]", "|", "|", "a", "#", " ", "😀", "\n"] }, + { label: "plaintext wikilinks", alphabet: ["[", "[", "]", "a", "#", " ", "😀", "\n"] }, + { label: "plaintext Markdown", alphabet: ["[", "[", "]", "(", ")", "a", " ", "\n"] }, + { + label: "plaintext callouts", + alphabet: [">", ">", "[", "!", "]", " ", "\t", "\n", "\r", "\u2028", "\u2029", "a"], + }, +] + +export const createScannerDocuments = ( + alphabet: ReadonlyArray, + count: number, +): string[] => { + // A fixed LCG seed makes every failing document reproducible across runtimes. + let state = 0x51ca77e2 + const draw = (limit: number): number => { + state = (Math.imul(state, 1664525) + 1013904223) >>> 0 + return Math.floor((state / 0x1_0000_0000) * limit) + } + const documents: string[] = [] + + for (let index = 0; index < count; index++) { + const length = draw(160) + const characters = Array.from({ length }, () => { + const character = alphabet[draw(alphabet.length)] + + if (character === undefined) throw new Error("random character is out of bounds") + return character + }) + documents.push(characters.join("")) + } + return documents +} diff --git a/src/vault-mcp/obsidian-markdown/__tests__/links.test.ts b/src/vault-mcp/obsidian-markdown/__tests__/links.test.ts index 23f5356d1..d9675d4df 100644 --- a/src/vault-mcp/obsidian-markdown/__tests__/links.test.ts +++ b/src/vault-mcp/obsidian-markdown/__tests__/links.test.ts @@ -1,9 +1,97 @@ import { describe, it, expect } from "vitest" +import { execFileSync } from "node:child_process" import { links } from "../links.js" +import { classifyLines } from "../lines.js" +import { + createScannerDocuments, + scannerCorpora, + legacyWiki, + legacyMarkdown, + legacyTemplater, +} from "./helpers/link-scanner-corpus.js" // ── matchLinksInLine ───────────────────────────────────────────── describe("matchLinksInLine", () => { + it.each([{ variant: "wiki" }, { variant: "markdown" }, { variant: "templater" }])( + "bounds repeated failing $variant openers in an externally killed child", + ({ variant }) => { + const fixturePath = new URL("./helpers/link-scanner-adversary.ts", import.meta.url) + // The parent deadline interrupts synchronous parser stalls, including legacy regex regressions. + const output = execFileSync( + process.execPath, + ["--import", "tsx", fixturePath.pathname, variant], + { + encoding: "utf8", + timeout: 30_000, + killSignal: "SIGKILL", + }, + ) + const result = JSON.parse(output) + + expect(result.variant).toBe(variant) + expect( + result.measurements.map((measurement: { bytes: number }) => measurement.bytes), + ).toEqual(variant === "markdown" ? [20_016, 40_016, 80_016] : [20_009, 40_009, 80_009]) + for (const [index, measurement] of result.measurements.entries()) { + // Ratios below 10 ms are noise; the ceiling still bounds those samples. + expect(measurement.median).toBeLessThan(1_000) + const previous = result.measurements[index - 1] + + if (previous?.median > 10) expect(measurement.median / previous.median).toBeLessThan(3) + } + }, + 35_000, + ) + + it.each(scannerCorpora.slice(0, 3))( + "preserves every legacy match, capture and UTF-16 offset in 10,000 $label documents", + ({ alphabet }) => { + for (const document of createScannerDocuments(alphabet, 10_000)) { + const expected = [ + ...[...document.matchAll(legacyWiki)].map((match) => ({ + text: match[0], + start: match.index, + end: match.index + match[0].length, + kind: "wikilink", + })), + ...[...document.matchAll(legacyMarkdown)].map((match) => ({ + text: match[0], + start: match.index, + end: match.index + match[0].length, + kind: "markdown", + })), + ] + expect(links.matchLinksInLine(document)).toEqual(expected) + const expectedTargets = new Set() + + for (const { text, inCode } of classifyLines(document)) { + if (inCode) continue + const masked = text + .replace(/`+[^`\n]*`+/g, (match) => " ".repeat(match.length)) + .replace(legacyTemplater, (match) => " ".repeat(match.length)) + + for (const match of masked.matchAll(legacyWiki)) { + const target = (match[1] ?? "").trim().replace(/\\$/, "") + + if (target) expectedTargets.add(target) + } + for (const match of masked.matchAll(legacyMarkdown)) { + const target = (match[1] ?? "").trim() + + if (!target) continue + try { + expectedTargets.add(decodeURIComponent(target)) + } catch { + expectedTargets.add(target) + } + } + } + expect(links.extractFromBody(document)).toEqual([...expectedTargets]) + } + }, + ) + it("finds a wikilink with its offsets and kind", () => { expect(links.matchLinksInLine("see [[Note A]] end")).toEqual([ { text: "[[Note A]]", start: 4, end: 14, kind: "wikilink" }, diff --git a/src/vault-mcp/obsidian-markdown/__tests__/plaintext.test.ts b/src/vault-mcp/obsidian-markdown/__tests__/plaintext.test.ts index 2403deeda..4651908d5 100644 --- a/src/vault-mcp/obsidian-markdown/__tests__/plaintext.test.ts +++ b/src/vault-mcp/obsidian-markdown/__tests__/plaintext.test.ts @@ -1,7 +1,57 @@ import { describe, it, expect } from "vitest" +import { execFileSync } from "node:child_process" import { stripMarkdownSyntax } from "../plaintext.js" +import { + createScannerDocuments, + scannerCorpora, + legacyPlaintext, +} from "./helpers/link-scanner-corpus.js" describe("stripMarkdownSyntax", () => { + it.each([ + { variant: "plaintext-alias", expectedBytes: [20_015, 40_015, 80_015] }, + { variant: "plaintext-wiki", expectedBytes: [20_009, 40_009, 80_009] }, + { variant: "plaintext-markdown", expectedBytes: [20_012, 40_012, 80_012] }, + { variant: "callout", expectedBytes: [20_009, 40_009, 80_009] }, + ])( + "bounds repeated failing $variant openers in an externally killed child", + ({ variant, expectedBytes }) => { + const fixturePath = new URL("./helpers/link-scanner-adversary.ts", import.meta.url) + // SIGKILL bounds a synchronous regex even when the child's event loop cannot run timers. + const output = execFileSync( + process.execPath, + ["--import", "tsx", fixturePath.pathname, variant], + { + encoding: "utf8", + timeout: 30_000, + killSignal: "SIGKILL", + }, + ) + const result = JSON.parse(output) + + expect(result.variant).toBe(variant) + expect( + result.measurements.map((measurement: { bytes: number }) => measurement.bytes), + ).toEqual(expectedBytes) + for (const [index, measurement] of result.measurements.entries()) { + expect(measurement.median).toBeLessThan(1_000) + const previous = result.measurements[index - 1] + + if (previous?.median > 10) expect(measurement.median / previous.median).toBeLessThan(3) + } + }, + 35_000, + ) + + it.each(scannerCorpora.slice(3))( + "preserves the complete legacy text for 10,000 $label documents", + ({ alphabet }) => { + for (const document of createScannerDocuments(alphabet, 10_000)) { + expect(stripMarkdownSyntax(document)).toBe(legacyPlaintext(document)) + } + }, + ) + const scenarios = [ { name: "wikilink with alias → display text", diff --git a/src/vault-mcp/obsidian-markdown/__tests__/tasks-clear-linear.test.ts b/src/vault-mcp/obsidian-markdown/__tests__/tasks-clear-linear.test.ts new file mode 100644 index 000000000..e1780355f --- /dev/null +++ b/src/vault-mcp/obsidian-markdown/__tests__/tasks-clear-linear.test.ts @@ -0,0 +1,276 @@ +import { execFileSync } from "node:child_process" +import { readFileSync } from "node:fs" +import { runInNewContext } from "node:vm" +import ts from "typescript" +import { describe, expect, it } from "vitest" + +/** Allows scheduling noise across the full input range. */ +const LINEAR_GROWTH_ALLOWANCE = 2 +/** Below this many milliseconds a median is mostly scheduling noise, so it sets no tighter growth bound. */ +const TIMING_FLOOR_MS = 25 + +type RemovalGrammar = { + clear: (text: string, index: number) => string + legacy: (text: string, index: number) => string + count: number +} + +const isRemovalGrammar = (value: unknown): value is RemovalGrammar => { + return ( + typeof value === "object" && + value !== null && + "clear" in value && + typeof value.clear === "function" && + "legacy" in value && + typeof value.legacy === "function" && + "count" in value && + typeof value.count === "number" + ) +} + +/** Matches the export modifier on a top-level constant declaration. */ +const EXPORT_MODIFIER_RE = /^export /u + +/** Keeps private declarations accessible without adding production test seams. */ +const compileRemovalGrammar = (): string => { + const source = readFileSync(new URL("../tasks.ts", import.meta.url), "utf8") + const syntax = ts.createSourceFile("tasks.ts", source, ts.ScriptTarget.Latest, true) + const program = syntax.statements + .filter(ts.isVariableStatement) + .map((statement) => statement.getText(syntax).replace(EXPORT_MODIFIER_RE, "")) + .join("\n") + return ts.transpileModule( + `${program} + const legacy = (metadata, regex) => { + const flags = regex.flags.includes("g") ? regex.flags : regex.flags + "g"; + const matches = [...metadata.matchAll(new RegExp(regex.source, flags))]; + const last = matches.at(-1); + if (!last) return metadata; + const stripped = (metadata.slice(0,last.index) + metadata.slice(last.index + last[0].length)).replace(/ {2,}/g," ").trim(); + return stripped === metadata ? metadata : legacy(stripped, regex); + }; + ({ + clear: (text,index) => removeAllMetadataMatches(text,METADATA_REMOVAL_GRAMMARS[index].regex), + legacy: (text,index) => legacy(text,METADATA_REMOVAL_GRAMMARS[index].regex), + count: METADATA_REMOVAL_GRAMMARS.length + })`, + { compilerOptions: { target: ts.ScriptTarget.ESNext } }, + ).outputText +} + +/** Freezes the original recursive removal as the compatibility oracle. */ +const readRemovalGrammar = (): RemovalGrammar => { + const grammar: unknown = runInNewContext(compileRemovalGrammar()) + + if (!isRemovalGrammar(grammar)) throw new Error("private metadata removers could not be loaded") + return grammar +} + +describe("task metadata clear compatibility", () => { + it("preserves recursive removal over generated mixed, malformed and newly exposed fields", () => { + const grammar = readRemovalGrammar() + const fragments = [ + "a", + " ", + " ", + "\t", + "\n", + "\u00a0", + "#", + "[", + "(", + "]", + ")", + ",", + "️", + "📅2026-01-02", + "📆2026-01-02", + "🗓2026-01-02", + "✅2026-01-02", + "❌2026-01-02", + "➕2026-01-02", + "🛫2026-01-02", + "⏳2026-01-02", + "⌛2026-01-02", + "⏫", + "🔺️", + "🔁every day", + "🏁keep", + "🆔abc", + "⛔abc , def", + "[due::2026-01-02]", + "(priority::high)", + "[repeat::every day]", + "[onCompletion::keep)", + "[id::abc]", + "(dependsOn::abc , def)", + ] + // Fixed state makes every generated failure reproducible. + let state = 0x753c1ea + const draw = (bound: number): number => { + state = (Math.imul(state, 1664525) + 1013904223) >>> 0 + return state % bound + } + const generated = Array.from({ length: 6000 }, () => { + return Array.from({ length: 1 + draw(14) }, () => fragments[draw(fragments.length)]).join("") + }) + const exposed = [ + "\uD83D⏫\uDD3A", + "\uD83D📅2026-01-02\uDCC52026-01-02", + "📅20📅2026-01-0226-01-02", + "[due::[due::2026-01-02]2026-01-02]", + "[priority::[priority::high]high]", + "🔁 [repeat::a] ", + "🔁 [repeat::a]\t", + "🆔a🆔b c", + "⛔a,⛔b c", + " plain text ", + "🔁 ", + "🔁\t⏫ ", + ] + + for (let index = 0; index < grammar.count; index += 1) { + for (const text of [...exposed, ...generated]) { + expect({ text, index, result: grammar.clear(text, index) }).toEqual({ + text, + index, + result: grammar.legacy(text, index), + }) + } + } + }, 20_000) + + it("clears twenty to eighty thousand priority fields within a child deadline", () => { + const parserUrl = new URL("../tasks.ts", import.meta.url).href + const output = execFileSync( + process.execPath, + [ + "--import", + "tsx", + "--input-type=module", + "--eval", + ` + import assert from 'node:assert/strict'; + import { tasks, DEFAULT_STATUS_REGISTRY } from ${JSON.stringify(parserUrl)}; + const config={taskFormat:'emoji',setDoneDate:true,setCancelledDate:true,setCreatedDate:false,recurrenceOnNextLine:false,removeScheduledDateOnRecurrence:false,statusRegistry:DEFAULT_STATUS_REGISTRY}; + const sizes=[20000,40000,80000]; + const medians=[]; + for (const size of sizes) { + const taskLine='- [ ] ⏫ #'+ '⏫'.repeat(size); + const check=()=>assert.equal(tasks.updateTaskLinePriority({taskLine,newPriority:null,config}),'- [ ] #'); + check(); + const samples=Array.from({length:5},()=>{const start=performance.now();check();return performance.now()-start;}).toSorted((a,b)=>a-b); + medians.push(samples[2]); + } + assert.ok(medians.every(value=>value<1000),'metadata clear exceeded ceiling '+medians); + assert.ok(medians.at(-1)<=Math.max(${TIMING_FLOOR_MS},medians[0])*(sizes.at(-1)/sizes[0])*${LINEAR_GROWTH_ALLOWANCE},'metadata clear scaled superlinearly '+medians); + process.stdout.write(JSON.stringify({sizes,medians})); + `, + ], + { encoding: "utf8", timeout: 15_000, killSignal: "SIGKILL" }, + ) + + expect(JSON.parse(output)).toEqual({ + sizes: [20000, 40000, 80000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }) + process.stdout.write(`Metadata clear timings ${output}\n`) + }, 20_000) + + it("bounds every field grammar and newly exposed nested fields at increasing sizes", () => { + const compiled = compileRemovalGrammar() + const output = execFileSync( + process.execPath, + [ + "--input-type=module", + "--eval", + ` + import assert from 'node:assert/strict'; + import { runInThisContext } from 'node:vm'; + const grammar=runInThisContext(${JSON.stringify(compiled)}); + const fields=[ + ['✅2026-01-02','[completion::2026-01-02]'], + ['❌2026-01-02','[cancelled::2026-01-02]'], + ['📅2026-01-02','[due::2026-01-02]'], + ['⏳2026-01-02','[scheduled::2026-01-02]'], + ['🛫2026-01-02','[start::2026-01-02]'], + ['➕2026-01-02','[created::2026-01-02]'], + ['⏫','[priority::high]'], + ['🆔abc','[id::abc]'], + ['⛔abc,def','[dependsOn::abc,def]'], + ['🔁every day','[repeat::every day]'], + ['🏁keep','[onCompletion::keep]'] + ]; + assert.equal(fields.length,grammar.count); + const sizes=[10000,20000,40000]; + const timings=[]; + const checkFixture=(name,index,makeText)=>{ + const medians=[]; + for(const size of sizes) { + const text=makeText(size); + const check=()=>assert.equal(grammar.clear(text,index),'#',name+' output'); + check(); + const samples=Array.from({length:5},()=>{const start=performance.now();check();return performance.now()-start;}).toSorted((a,b)=>a-b); + medians.push(samples[2]); + } + assert.ok(medians.every(value=>value<2000),name+' exceeded ceiling '+medians); + assert.ok(medians.at(-1)<=Math.max(${TIMING_FLOOR_MS},medians[0])*(sizes.at(-1)/sizes[0])*${LINEAR_GROWTH_ALLOWANCE},name+' scaled superlinearly '+medians); + timings.push({name,medians}); + }; + fields.forEach((formats,index)=>formats.forEach((field,format)=>checkFixture(index+'-'+format,index,size=>'#'+field.repeat(size)))); + checkFixture('nested-emoji-date',2,size=>'#'+'📅20'.repeat(size)+'📅2026-01-02'+'26-01-02'.repeat(size)); + checkFixture('nested-dataview-date',2,size=>'#'+'[due::'.repeat(size)+'2026-01-02'+']2026-01-02'.repeat(size-1)+']'); + process.stdout.write(JSON.stringify({sizes,timings})); + `, + ], + { encoding: "utf8", timeout: 30_000, killSignal: "SIGKILL" }, + ) + + const result: unknown = JSON.parse(output) + + expect(result).toEqual({ + sizes: [10000, 20000, 40000], + timings: Array.from({ length: 24 }, () => ({ + name: expect.any(String), + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + })), + }) + process.stdout.write(`All metadata grammar timings ${output}\n`) + }, 35_000) + + it("clears duplicate completion stamps through status mutations within a child deadline", () => { + const parserUrl = new URL("../tasks.ts", import.meta.url).href + const output = execFileSync( + process.execPath, + [ + "--import", + "tsx", + "--input-type=module", + "--eval", + ` + import assert from 'node:assert/strict'; + import { tasks, DEFAULT_STATUS_REGISTRY } from ${JSON.stringify(parserUrl)}; + const config={taskFormat:'emoji',setDoneDate:true,setCancelledDate:true,setCreatedDate:false,recurrenceOnNextLine:false,removeScheduledDateOnRecurrence:false,statusRegistry:DEFAULT_STATUS_REGISTRY}; + const sizes=[20000,40000,80000]; + const medians=[]; + for(const size of sizes) { + const taskLine='- [x] ✅2026-01-02 ❌2026-01-03 #'+ '✅2026-01-02❌2026-01-03'.repeat(size); + const check=()=>assert.equal(tasks.updateTaskLineStatus({taskLine,newStatus:'todo',today:'2026-01-04',config}),'- [ ] #'); + check(); + const samples=Array.from({length:5},()=>{const start=performance.now();check();return performance.now()-start;}).toSorted((a,b)=>a-b); + medians.push(samples[2]); + } + assert.ok(medians.every(value=>value<1000),'status stamp clear exceeded ceiling '+medians); + assert.ok(medians.at(-1)<=Math.max(${TIMING_FLOOR_MS},medians[0])*(sizes.at(-1)/sizes[0])*${LINEAR_GROWTH_ALLOWANCE},'status stamp clear scaled superlinearly '+medians); + process.stdout.write(JSON.stringify({sizes,medians})); + `, + ], + { encoding: "utf8", timeout: 15_000, killSignal: "SIGKILL" }, + ) + + expect(JSON.parse(output)).toEqual({ + sizes: [20000, 40000, 80000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }) + }, 20_000) +}) diff --git a/src/vault-mcp/obsidian-markdown/__tests__/tasks-linear.test.ts b/src/vault-mcp/obsidian-markdown/__tests__/tasks-linear.test.ts new file mode 100644 index 000000000..2b9872c27 --- /dev/null +++ b/src/vault-mcp/obsidian-markdown/__tests__/tasks-linear.test.ts @@ -0,0 +1,682 @@ +import { execFileSync } from "node:child_process" +import { globSync, readFileSync } from "node:fs" +import { runInNewContext } from "node:vm" +import { DateTime } from "luxon" +import ts from "typescript" +import { describe, expect, it } from "vitest" +import { tasks } from "../tasks.js" +import { createScannerDocuments } from "./helpers/link-scanner-corpus.js" + +/** Allows scheduling noise across the full input range. */ +const LINEAR_GROWTH_ALLOWANCE = 2 +/** Below this many milliseconds a median is mostly scheduling noise, so it sets no tighter growth bound. */ +const TIMING_FLOOR_MS = 25 + +type Matcher = { exec: (input: string) => RegExpExecArray | null } +type Grammar = { + taskLine: Matcher + dataview: (params: { key: string; valuePattern: string }) => Matcher + emoji: Matcher + repeat: Matcher + dependencies: Matcher + metadataStart: (body: string) => number + legacyMetadataStart: (body: string) => number + withoutBlockLink: (body: string) => string + trailingTags: (text: string) => readonly string[] + legacyTrailingTags: (text: string) => readonly string[] +} + +const isGrammar = (value: unknown): value is Grammar => { + return ( + typeof value === "object" && + value !== null && + "taskLine" in value && + "dataview" in value && + typeof value.dataview === "function" && + "emoji" in value && + "repeat" in value && + "dependencies" in value && + "metadataStart" in value && + typeof value.metadataStart === "function" && + "legacyMetadataStart" in value && + typeof value.legacyMetadataStart === "function" && + "withoutBlockLink" in value && + typeof value.withoutBlockLink === "function" && + "trailingTags" in value && + typeof value.trailingTags === "function" && + "legacyTrailingTags" in value && + typeof value.legacyTrailingTags === "function" + ) +} + +/** Loads the private recognizers without adding test-only production exports. */ +const readGrammar = (): Grammar => { + const source = readFileSync(new URL("../tasks.ts", import.meta.url), "utf8") + const syntax = ts.createSourceFile("tasks.ts", source, ts.ScriptTarget.Latest, true) + const declarations = syntax.statements.filter(ts.isVariableStatement) + /** Matches the export modifier on a top-level constant declaration. */ + const exportModifier = /^export /u + const program = declarations + .map((statement) => statement.getText(syntax).replace(exportModifier, "")) + .join("\n") + const compiled = ts.transpileModule( + `${program}\n;({ + taskLine: TASK_LINE_RE, + dataview: dataviewField, + emoji: EMOJI_RECURRENCE_RE, + repeat: DATAVIEW_RECURRENCE_RE, + dependencies: DATAVIEW_DEPENDS_ON_RE, + metadataStart: findMetadataStart, + withoutBlockLink: (body) => splitTrailingBlockLink(body).body, + trailingTags: extractTrailingTags, + legacyTrailingTags: (text) => { + const tags = []; + let remaining = text; + let match = HASHTAG_FROM_END_RE.exec(remaining); + while (match) { + tags.unshift(match[0].trim()); + remaining = remaining.slice(0,match.index).trim(); + match = HASHTAG_FROM_END_RE.exec(remaining); + } + return tags; + }, + // Freeze the unbounded search while sharing the unchanged metadata parser. + legacyMetadataStart: (taskBody) => { + for (const candidate of taskBody.matchAll(new RegExp(FIRST_METADATA_SIGNIFIER_RE.source, "gu"))) { + const tail = parseTaskMetadata(taskBody.slice(candidate.index)); + const tailDescription = tail.description.replace(HASHTAG_RE, "").trim(); + if (tailDescription === "") return candidate.index; + } + return -1; + } + })`, + { compilerOptions: { target: ts.ScriptTarget.ESNext } }, + ).outputText + const grammar: unknown = runInNewContext(compiled, { DateTime }) + + if (!isGrammar(grammar)) throw new Error("private task grammar could not be loaded") + return grammar +} + +/** Freezes the suffix grammar used before the delimiter scan. */ +const legacyDataview = (inner: string): RegExp => + new RegExp(`(?:(?=[^\\]]+\\])\\[|(?=[^)]+\\))\\() *${inner} *[)\\]](?: *,)?$`) + +const consumedMatch = (match: RegExpExecArray | null): unknown => { + if (!match) return null + return { index: match.index, text: match[0], value: (match[1] ?? match[2])?.trim() } +} + +describe("task recognizer compatibility", () => { + it("preserves trailing tags over fixed-seed punctuation and whitespace variants", () => { + const grammar = readGrammar() + const alphabet = [ + "#", + "a", + "b", + " ", + "\t", + "\r", + "\n", + "\u2028", + "\u2029", + "\u00a0", + "[", + "]", + "(", + ")", + "!", + "_", + "/", + ":", + ".", + "😀", + ] + const documents = createScannerDocuments(alphabet, 10_000) + const boundaryCases = [ + "#a\t#b", + "#a\n#b", + "#a ", + "#a\t", + "#a\tcontent", + "before #a\t#b", + "##a", + "#a ! #b", + "#a #a", + "#😀/nested", + ] + + for (const text of [...boundaryCases, ...documents]) { + expect({ text, tags: grammar.trailingTags(text) }).toEqual({ + text, + tags: grammar.legacyTrailingTags(text), + }) + } + expect(grammar.trailingTags("#a\t#b #a")).toEqual(["#a", "#b", "#a"]) + }) + + it("bounds deduplication with tens of thousands of trailing description tags", () => { + const parserUrl = new URL("../tasks.ts", import.meta.url).href + const output = execFileSync( + process.execPath, + [ + "--import", + "tsx", + "--input-type=module", + "--eval", + ` + import assert from 'node:assert/strict'; + import { tasks } from ${JSON.stringify(parserUrl)}; + const sizes=[20000,40000,80000]; + const medians=[]; + for (const size of sizes) { + const taskLine='- [ ] '+'#a '.repeat(size)+'📅 2026-01-02'; + const check=()=>assert.deepEqual(tasks.deduplicateDescriptionTags(taskLine),{taskLine,deduplicatedTags:[]}); + check(); + const samples=Array.from({length:5},()=>{const start=performance.now();check();return performance.now()-start;}).toSorted((a,b)=>a-b); + medians.push(samples[2]); + } + assert.ok(medians.every(value=>value<500),'trailing tag deduplication exceeded ceiling '+medians); + assert.ok(medians.at(-1)<=Math.max(${TIMING_FLOOR_MS},medians[0])*(sizes.at(-1)/sizes[0])*${LINEAR_GROWTH_ALLOWANCE},'trailing tag deduplication scaled superlinearly '+medians); + process.stdout.write(JSON.stringify({sizes,medians})); + `, + ], + { encoding: "utf8", timeout: 10_000, killSignal: "SIGKILL" }, + ) + + expect(JSON.parse(output)).toEqual({ + sizes: [20000, 40000, 80000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }) + process.stdout.write(`Trailing tag deduplication timings ${output}\n`) + }, 15_000) + + it("preserves metadata boundaries over fixed-seed bodies and every literal task fixture", () => { + const grammar = readGrammar() + // The generator advances once per draw so a failure has a reproducible body. + let state = 0x7535eed + const draw = (bound: number): number => { + state = (Math.imul(state, 1664525) + 1013904223) >>> 0 + return state % bound + } + const fragments = [ + "Work", + "Prefer ⏫ arrows in prose", + "📅 tomorrow", + "#tag", + "#⏫⏫", + "#📅📅📅", + "#🆔a", + "#⛔a", + "#📅2026-01-02", + "#⏫\t🆔abc", + "#x[priority::high]", + "[ priority:: high]", + "⏫", + "🔺️", + "🔁 every week", + "🏁 keep", + "🆔 abc", + "⛔ a,b", + "➕2026-01-02", + "🛫2026-01-02", + "⏳2026-01-02", + "📅2026-01-02", + "✅2026-01-02", + "❌2026-99-99", + "[priority:: high]", + "(repeat:: every day)", + "[onCompletion:: delete]", + "[id:: abc]", + "(dependsOn:: a,b)", + "[created:: 2026-01-02]", + "(start:: 2026-01-02)", + "[scheduled:: 2026-01-02]", + "[due:: 2026-01-02]", + "[completion:: 2026-01-02]", + "(cancelled:: 2026-01-02)", + "[due:: bad)", + "⏫".repeat(40), + ] + const randomBodies = Array.from({ length: 1000 }, () => { + const pieces = Array.from({ length: 1 + draw(16) }, () => fragments[draw(fragments.length)]) + return pieces.join(draw(2) === 0 ? " " : "") + }) + const adjacentDataviewBodies = [ + "work #tag[id:: a]", + "work #tag(id:: a)", + "work #tag[due:: 2026-01-02]", + "work #tag(due:: 2026-01-02)", + "work #tag[priority:: high]", + "work #tag(priority:: high)", + ] + const denseBodies = [20, 21, 22, 482, 483, 484, 600].flatMap((size) => [ + "⏫".repeat(size), + `⏫ #${"⏫".repeat(size)}`, + `#${"⏫".repeat(size)}`, + `⏫ #${"⏫".repeat(size)} [due:: 2026-01-02]`, + `🆔abc ${"#⏫ ".repeat(size)}`, + ]) + const packedHashtagBodies = [1, 2, 20, 21, 22].flatMap((size) => [ + Array.from({ length: size }, () => `#${"📅".repeat(size)}`).join(" "), + Array.from({ length: size }, () => `#${"⏫".repeat(size)}`).join(" "), + `⏫ ${"#📅 ".repeat(size)}`, + `#📅2026-01-02 ${"(repeat:: every day) ".repeat(size)}[id:: abc]`, + `#⏫\t🆔abc ${"[ priority:: high] ".repeat(size)}`, + ]) + // Reverse extraction order permits every matcher to consume a field in each pass. + const fullPass = [ + "[dependsOn:: a]", + "⛔ a", + "[id:: a]", + "🆔 a", + "#tag", + "[onCompletion:: keep]", + "🏁 keep", + "[repeat:: daily]", + "🔁 daily", + "[created:: 2026-01-02]", + "➕2026-01-02", + "[start:: 2026-01-02]", + "🛫2026-01-02", + "[scheduled:: 2026-01-02]", + "⏳2026-01-02", + "[due:: 2026-01-02]", + "📅2026-01-02", + "[cancelled:: 2026-01-02]", + "❌2026-01-02", + "[completion:: 2026-01-02]", + "✅2026-01-02", + "[priority:: high]", + "⏫", + ].join(" ") + const fullTail = Array.from({ length: 21 }, () => fullPass).join(" ") + + expect(grammar.legacyMetadataStart(fullTail)).toBe(0) + expect(grammar.metadataStart(fullTail)).toBe(0) + // AST traversal accumulates fixtures from all task-related parser and mutation tests. + const fixtureBodies: string[] = [] + const fixtureFolders = [ + new URL("./", import.meta.url), + new URL("../../vault-operations/__tests__/", import.meta.url), + ] + const fixtureUrls = fixtureFolders.flatMap((folder) => { + return globSync("*task*.test.ts", { cwd: folder }).map( + (filename) => new URL(filename, folder), + ) + }) + + for (const fixtureUrl of fixtureUrls) { + const source = readFileSync(fixtureUrl, "utf8") + const syntax = ts.createSourceFile(fixtureUrl.pathname, source, ts.ScriptTarget.Latest, true) + const collectTaskLines = (node: ts.Node): void => { + if (ts.isStringLiteralLike(node)) { + for (const line of node.text.split("\n")) { + const body = grammar.taskLine.exec(line)?.[2] + + if (body !== undefined) fixtureBodies.push(grammar.withoutBlockLink(body)) + } + } + ts.forEachChild(node, collectTaskLines) + } + collectTaskLines(syntax) + } + + expect(fixtureBodies.length).toBeGreaterThan(300) + const bodies = [ + ...randomBodies, + ...denseBodies, + ...packedHashtagBodies, + ...adjacentDataviewBodies, + fullTail, + ...fixtureBodies, + ] + + for (const body of bodies) { + expect({ body, boundary: grammar.metadataStart(body) }).toEqual({ + body, + boundary: grammar.legacyMetadataStart(body), + }) + } + process.stdout.write( + `Metadata boundary corpus ${JSON.stringify({ random: randomBodies.length, dense: denseBodies.length, fixtures: fixtureBodies.length })}\n`, + ) + }, 30_000) + + it("keeps a valid boundary before an invalid signifier inside a later hashtag", () => { + const grammar = readGrammar() + + expect(grammar.metadataStart("⏫ #📅")).toBe(0) + expect(grammar.metadataStart("📅")).toBe(-1) + expect(grammar.metadataStart("#📅2026-01-02 (repeat:: every day)")).toBe(1) + }) + + it("bounds task edits whose hashtags each pack hundreds of signifiers", () => { + const parserUrl = new URL("../tasks.ts", import.meta.url).href + const output = execFileSync( + process.execPath, + [ + "--import", + "tsx", + "--input-type=module", + "--eval", + ` + import assert from 'node:assert/strict'; + import { tasks } from ${JSON.stringify(parserUrl)}; + const config = {taskFormat:'emoji',setDoneDate:true,setCancelledDate:true}; + const sizes = [120,240,480]; + const medians = []; + for (const size of sizes) { + const taskLine = '- [ ] ' + Array.from({length:size},()=> '#'+ '📅'.repeat(size+2)).join(' '); + const check = () => { + assert.equal(tasks.updateTaskLineTaskId({taskLine,taskId:'probe',config}),taskLine+' 🆔 probe'); + assert.deepEqual(tasks.diffTaskRoundTrip({taskLine,priorTaskLine:taskLine,submitted:{}}),[]); + }; + check(); + const samples = Array.from({length:3},()=>{const start=performance.now();check();return performance.now()-start;}).toSorted((a,b)=>a-b); + medians.push(samples[1]); + } + assert.ok(medians.every(value=>value<1000),'packed hashtag edits exceeded ceiling '+medians); + process.stdout.write(JSON.stringify({sizes,medians})); + `, + ], + { encoding: "utf8", timeout: 10_000, killSignal: "SIGKILL" }, + ) + + expect(JSON.parse(output)).toEqual({ + sizes: [120, 240, 480], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }) + process.stdout.write(`Packed hashtag edit timings ${output}\n`) + }, 15_000) + + it("bounds signifier-dense task edits and their growth in a child process", () => { + const parserUrl = new URL("../tasks.ts", import.meta.url).href + const output = execFileSync( + process.execPath, + [ + "--import", + "tsx", + "--input-type=module", + "--eval", + ` + import assert from 'node:assert/strict'; + import { tasks } from ${JSON.stringify(parserUrl)}; + const config = {taskFormat:'emoji',setDoneDate:true,setCancelledDate:true}; + const sizes = [1000,2000,4000,5000]; + const medians = []; + for (const size of sizes) { + const taskLine = '- [ ] ' + '⏫'.repeat(size); + const expected = '- [ ] ' + '⏫'.repeat(size-21) + ' ' + '⏫'.repeat(21) + ' 🆔 probe'; + const check = () => { + assert.equal(tasks.updateTaskLineTaskId({taskLine,taskId:'probe',config}), expected); + assert.deepEqual(tasks.diffTaskRoundTrip({taskLine,priorTaskLine:taskLine,submitted:{}}), []); + }; + check(); + const samples = Array.from({length:5}, () => {const start=performance.now();check();return performance.now()-start;}).toSorted((first,second)=>first-second); + medians.push(samples[2]); + } + assert.ok(medians.every(value => value < 250), 'dense edits exceeded 250ms ' + medians); + assert.ok(medians.at(-1) <= Math.max(${TIMING_FLOOR_MS},medians[0])*(sizes.at(-1)/sizes[0])*${LINEAR_GROWTH_ALLOWANCE}, 'dense edits scaled superlinearly ' + medians); + process.stdout.write(JSON.stringify({sizes,medians})); + `, + ], + { encoding: "utf8", timeout: 10_000, killSignal: "SIGKILL" }, + ) + const measurements: unknown = JSON.parse(output) + + expect(measurements).toEqual({ + sizes: [1000, 2000, 4000, 5000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number), expect.any(Number)], + }) + process.stdout.write(`Task metadata edit timings ${output}\n`) + }, 15_000) + + it("preserves spans and consumed captures over fixed-seed field variants", () => { + const grammar = readGrammar() + /** The generator's state advances once per draw, making failures reproducible. */ + let state = 0x739afb02 + const draw = (bound: number): number => { + state = (Math.imul(state, 1664525) + 1013904223) >>> 0 + return state % bound + } + const variants = [ + { inner: "completion:: *(\\d{4}-\\d{2}-\\d{2})", value: "2026-01-02" }, + { inner: "cancelled:: *(\\d{4}-\\d{2}-\\d{2})", value: "2026-01-02" }, + { inner: "due:: *(\\d{4}-\\d{2}-\\d{2})", value: "2026-01-02" }, + { inner: "scheduled:: *(\\d{4}-\\d{2}-\\d{2})", value: "2026-01-02" }, + { inner: "start:: *(\\d{4}-\\d{2}-\\d{2})", value: "2026-01-02" }, + { inner: "created:: *(\\d{4}-\\d{2}-\\d{2})", value: "2026-01-02" }, + { inner: "priority:: *(highest|high|medium|low|lowest)", value: "highest" }, + { inner: "repeat:: *([a-zA-Z0-9, !]+)", value: "every week !" }, + { inner: "onCompletion:: *([a-zA-Z]+)", value: "delete" }, + { inner: "id:: *([a-zA-Z0-9_-]+)", value: "task_01" }, + { + inner: "dependsOn:: *([a-zA-Z0-9_-]+(?:(?: *, *[a-zA-Z0-9_-]+)+ *)?)", + value: "a , b , c ", + }, + ] + const alphabet = [ + "[", + "(", + "]", + ")", + " ", + "!", + "a", + ":", + "0", + ",", + "\n", + "\r", + "\u2028", + "\u2029", + "😀", + "\\", + "\t", + ] + const count = Number(process.env.PARSER_CORPUS_CASES ?? 10_000) + + for (const variant of variants) { + const oldPattern = legacyDataview(variant.inner) + const keyEnd = variant.inner.indexOf("::") + const key = variant.inner.slice(0, keyEnd + 2) + const fieldGrammar = { + key: variant.inner.slice(0, keyEnd), + valuePattern: variant.inner.slice(keyEnd + ":: *".length), + } + const matcher = variant.inner.startsWith("repeat::") + ? grammar.repeat + : grammar.dataview(fieldGrammar) + + for (let index = 0; index < count; index += 1) { + const payload = Array.from( + { length: draw(70) }, + () => alphabet[draw(alphabet.length)], + ).join("") + const open = index % 2 === 0 ? "[" : "(" + const close = open === "[" ? "]" : ")" + const value = index % 3 === 0 ? variant.value : payload + const input = + index % 5 === 0 + ? payload + : `${payload}${open} ${key}${" ".repeat(draw(4))}${value}${close}${index % 4 === 0 ? " ," : ""}` + + expect({ input, match: consumedMatch(matcher.exec(input)) }).toEqual({ + input, + match: consumedMatch(oldPattern.exec(input)), + }) + } + } + /** Matches the emoji recurrence suffix, with the whitespace outside its value capture. */ + const oldEmoji = /🔁\uFE0F? *([a-zA-Z0-9, !]+)$/ + + for (let index = 0; index < count; index += 1) { + const payload = Array.from({ length: draw(90) }, () => alphabet[draw(alphabet.length)]).join( + "", + ) + const input = `text 🔁${index % 2 === 0 ? "\uFE0F" : ""}${" ".repeat(draw(8))}${index % 3 === 0 ? "every week" : payload}` + expect(consumedMatch(grammar.emoji.exec(input))).toEqual(consumedMatch(oldEmoji.exec(input))) + } + }) + + it("compares real Dataview dependency fields with the frozen suffix grammar", () => { + const grammar = readGrammar() + const legacy = legacyDataview("dependsOn:: *([a-zA-Z0-9_-]+( *, *[a-zA-Z0-9_-]+ *)*)") + const values = ["a", "a,b", "a , b , c ", "db-setup, cache_setup", "", "a!", "a,,b"] + + for (const value of values) { + for (const delimiters of [ + { open: "[", close: "]" }, + { open: "(", close: ")" }, + ]) { + for (const suffix of ["", ",", " ,", ", ", ",]", ")"]) { + const input = `- [ ] x ${delimiters.open}dependsOn:: ${value}${delimiters.close}${suffix}` + + expect({ input, match: consumedMatch(grammar.dependencies.exec(input)) }).toEqual({ + input, + match: consumedMatch(legacy.exec(input)), + }) + } + } + } + expect(consumedMatch(grammar.dependencies.exec("- [ ] x (dependsOn:: a),"))).toEqual({ + index: 8, + text: "(dependsOn:: a),", + value: "a", + }) + }) + + it("preserves the body capture after removing only its leading ASCII spaces", () => { + const grammar = readGrammar() + /** Matches the complete task line and consumes ASCII spaces before capturing its body. */ + const legacy = /^[\s>]*(?:[-*+]|[0-9]+[.)]) +\[(.)\] *(.*)$/u + const bodies = [ + "", + " ", + " text", + "\ttext", + " text", + "😀", + "x\rtext", + "x\u2028text", + "x\u2029text", + "text\n", + "text\r\n", + ] + + for (const prefix of ["- [ ]", " > 1) [x]", "+ [/]", "* [😀]"]) { + for (const body of bodies) { + const input = prefix + body + const oldMatch = legacy.exec(input) + const match = grammar.taskLine.exec(input) + const actual = match && { + text: match[0], + status: match[1], + body: match[2]?.replace(/^ +/, ""), + } + const expected = oldMatch && { text: oldMatch[0], status: oldMatch[1], body: oldMatch[2] } + expect(actual).toEqual(expected) + } + } + }) + + it("keeps fields left of recurrence and reads both Dataview captures", () => { + for (const recurrence of [ + "🔁 every week", + "[repeat:: every week ]", + "(repeat:: every week )", + ]) { + const input = `- [ ] Work 📅 2026-01-02 ${recurrence} ^work` + expect(tasks.describeTaskLine(input)).toBe("Work") + expect( + tasks.extractTasks(input).map(({ description, recurrence, dueDate, blockId }) => ({ + description, + recurrence, + dueDate, + blockId, + })), + ).toEqual([ + { description: "Work", recurrence: "every week", dueDate: "2026-01-02", blockId: "work" }, + ]) + expect( + tasks.diffTaskRoundTrip({ taskLine: input, priorTaskLine: input, submitted: {} }), + ).toEqual([]) + } + }) + + it("bounds task, recurrence and Dataview adversaries in a child process", () => { + const parserUrl = new URL("../tasks.ts", import.meta.url).href + const output = execFileSync( + process.execPath, + [ + "--import", + "tsx", + "--input-type=module", + "--eval", + ` + import assert from 'node:assert/strict'; + import { tasks } from ${JSON.stringify(parserUrl)}; + const fixtures = [ + { name: 'task-u2028', sizes: [10000,20000,40000], input: n => '- [ ]' + ' '.repeat(n) + '\\u2028x', rejected: true }, + { name: 'task-u2029', sizes: [10000,20000,40000], input: n => '- [ ]' + ' '.repeat(n) + '\\u2029x', rejected: true }, + { name: 'emoji-recurrence', sizes: [20000,40000,80000], input: n => '- [ ] x 🔁' + ' '.repeat(n) + '?', rejected: false }, + { name: 'dataview-recurrence', sizes: [1000,2000,4000], input: n => '- [ ] x [repeat::' + ' '.repeat(n) + '?]', rejected: false }, + { name: 'dataview-lookahead', sizes: [25000,50000,100000], input: n => '- [ ] ' + '('.repeat(n), rejected: false } + ]; + const measurements = []; + for (const fixture of fixtures) { + const medians = []; + for (const size of fixture.sizes) { + const input = fixture.input(size); + const check = () => { + const result = tasks.extractTasks(input); + assert.deepEqual(result.map(task => task.description), fixture.rejected ? [] : [input.slice(6)]); + assert.equal(tasks.describeTaskLine(input), fixture.rejected ? input : input.slice(6)); + assert.deepEqual(tasks.diffTaskRoundTrip({taskLine:input, priorTaskLine:input, submitted:{}}), []); + }; + check(); + const samples = Array.from({length:5}, () => { const start=performance.now(); check(); return performance.now()-start; }).sort((a,b)=>a-b); + medians.push(samples[2]); + } + assert.ok(medians.every(value => value < 250), fixture.name + ' exceeded 250ms ' + medians); + assert.ok(medians.at(-1) <= Math.max(${TIMING_FLOOR_MS},medians[0])*(fixture.sizes.at(-1)/fixture.sizes[0])*${LINEAR_GROWTH_ALLOWANCE}, fixture.name + ' scaled superlinearly ' + medians); + measurements.push({name:fixture.name,sizes:fixture.sizes,medians}); + } + process.stdout.write(JSON.stringify(measurements)); + `, + ], + { encoding: "utf8", timeout: 30_000, killSignal: "SIGKILL" }, + ) + const measurements: unknown = JSON.parse(output) + + expect(measurements).toEqual([ + { + name: "task-u2028", + sizes: [10_000, 20_000, 40_000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }, + { + name: "task-u2029", + sizes: [10_000, 20_000, 40_000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }, + { + name: "emoji-recurrence", + sizes: [20_000, 40_000, 80_000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }, + { + name: "dataview-recurrence", + sizes: [1_000, 2_000, 4_000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }, + { + name: "dataview-lookahead", + sizes: [25_000, 50_000, 100_000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }, + ]) + process.stdout.write(`Task parser timings ${output}\n`) + }, 35_000) +}) diff --git a/src/vault-mcp/obsidian-markdown/__tests__/tasks-tag-dedup.test.ts b/src/vault-mcp/obsidian-markdown/__tests__/tasks-tag-dedup.test.ts new file mode 100644 index 000000000..bd75dd52c --- /dev/null +++ b/src/vault-mcp/obsidian-markdown/__tests__/tasks-tag-dedup.test.ts @@ -0,0 +1,121 @@ +import { execFileSync } from "node:child_process" +import { readFileSync } from "node:fs" +import { runInNewContext } from "node:vm" +import { DateTime } from "luxon" +import ts from "typescript" +import { describe, expect, it } from "vitest" +import { tasks } from "../tasks.js" + +/** Allows scheduling noise across the full input range. */ +const LINEAR_GROWTH_ALLOWANCE = 2 + +type Deduplication = ReturnType + +/** Freezes the original per-tag search while sharing the separately verified boundary parser. */ +const readLegacyDeduplication = (): ((taskLine: string) => Deduplication) => { + const source = readFileSync(new URL("../tasks.ts", import.meta.url), "utf8") + const syntax = ts.createSourceFile("tasks.ts", source, ts.ScriptTarget.Latest, true) + /** Matches the export modifier on top-level declarations. */ + const exportModifier = /^export /u + const declarations = syntax.statements + .filter(ts.isVariableStatement) + .map((statement) => statement.getText(syntax).replace(exportModifier, "")) + .join("\n") + const legacy = `(taskLine) => { + const parts = splitTaskLine(taskLine); + if (!parts || !parts.metadata) return {taskLine, deduplicatedTags: []}; + const descriptionTrailingTags = extractTrailingTags(parts.description); + let metadata = parts.metadata; + const deduplicatedTags = []; + for (const tag of descriptionTrailingTags) { + const pattern = new RegExp('(^|\\\\s)' + RegExp.escape(tag) + '(?=\\\\s|$)'); + if (pattern.test(metadata)) { + metadata = metadata.replace(pattern, '').trim(); + deduplicatedTags.push(tag); + } + } + if (deduplicatedTags.length === 0) return {taskLine, deduplicatedTags}; + return {taskLine: joinTaskLine({...parts, metadata}), deduplicatedTags}; + }` + const compiled = ts.transpileModule(`${declarations}\n;(${legacy})`, { + compilerOptions: { target: ts.ScriptTarget.ESNext }, + }).outputText + const loaded: unknown = runInNewContext(compiled, { DateTime }) + + if (typeof loaded !== "function") throw new Error("legacy tag deduplication did not load") + return (taskLine) => loaded(taskLine) +} + +describe("task description tag deduplication", () => { + it("preserves sequential removals that expose a tag containing whitespace", () => { + expect(tasks.deduplicateDescriptionTags("- [ ] Work #x #a\tb 📅 2026-01-01 #a\t#x\tb")).toEqual( + { + taskLine: "- [ ] Work #x #a\tb 📅 2026-01-01", + deduplicatedTags: ["#x", "#a\tb"], + }, + ) + }) + + it("matches the original removal order over generated tag and whitespace variants", () => { + const legacyDeduplication = readLegacyDeduplication() + const tags = ["#a", "#b", "#a/b", "#a\tb", "#a\u00a0b", "#😀", "#a-b"] + const separators = [" ", " ", "\t", "\u00a0", " \t"] + // The generator advances once per draw so each mismatch is reproducible. + let state = 753 + const draw = (bound: number): number => { + state = (Math.imul(state, 1664525) + 1013904223) >>> 0 + return state % bound + } + const tagRun = (): string => { + return Array.from({ length: draw(40) }, () => { + return `${tags[draw(tags.length)]}${separators[draw(separators.length)]}` + }).join("") + } + + for (let index = 0; index < 2000; index++) { + const taskLine = `- [ ] Work ${tagRun()}📅 2026-01-02 ${tagRun()}🆔 abc` + + expect({ taskLine, result: tasks.deduplicateDescriptionTags(taskLine) }).toEqual({ + taskLine, + result: legacyDeduplication(taskLine), + }) + } + }) + + it("bounds distinct description tags against a long metadata tail in a child process", () => { + const parserUrl = new URL("../tasks.ts", import.meta.url).href + const output = execFileSync( + process.execPath, + [ + "--import", + "tsx", + "--input-type=module", + "--eval", + ` + import assert from 'node:assert/strict'; + import { tasks } from ${JSON.stringify(parserUrl)}; + const sizes = [20000,40000,80000]; + const medians = []; + for (const size of sizes) { + const tags = Array.from({length:size},(_,index)=>'#t'+index).join(' '); + const taskLine = '- [ ] '+tags+' 📅2026-01-02 🆔'+'x'.repeat(size); + const check = () => assert.deepEqual(tasks.deduplicateDescriptionTags(taskLine),{taskLine,deduplicatedTags:[]}); + check(); + const samples = Array.from({length:5},()=>{const start=performance.now();check();return performance.now()-start;}).toSorted((a,b)=>a-b); + medians.push(samples[2]); + } + assert.ok(medians.every(value=>value<350),'description tag searches exceeded ceiling '+medians); + assert.ok(medians.at(-1)<=Math.max(10,medians[0])*(sizes.at(-1)/sizes[0])*${LINEAR_GROWTH_ALLOWANCE},'description tag searches scaled superlinearly '+medians); + process.stdout.write(JSON.stringify({sizes,medians})); + `, + ], + { encoding: "utf8", timeout: 10_000, killSignal: "SIGKILL" }, + ) + + expect(JSON.parse(output)).toEqual({ + sizes: [20000, 40000, 80000], + medians: [expect.any(Number), expect.any(Number), expect.any(Number)], + }) + process.stdout.write(`Distinct description tag timings ${output}\n`) + }, 15_000) +}) diff --git a/src/vault-mcp/obsidian-markdown/__tests__/tasks.test.ts b/src/vault-mcp/obsidian-markdown/__tests__/tasks.test.ts index c14d55fbb..804d49dc5 100644 --- a/src/vault-mcp/obsidian-markdown/__tests__/tasks.test.ts +++ b/src/vault-mcp/obsidian-markdown/__tests__/tasks.test.ts @@ -634,7 +634,7 @@ describe("tasks.extractTasks", () => { const elapsed = performance.now() - started; process.stdout.write(JSON.stringify({ extracted, elapsed }));`, ], - { encoding: "utf8", timeout: 10000 }, + { encoding: "utf8", timeout: 10000, killSignal: "SIGKILL" }, ) const result = JSON.parse(output) @@ -645,7 +645,7 @@ describe("tasks.extractTasks", () => { }, ) - it("preserves dependency matches and group 1 across fixed-seed lists and existing fixtures", () => { + it("preserves emoji dependency matches and group 1 across fixed-seed lists and existing fixtures", () => { /** The former dependency grammar, retained as the compatibility oracle. */ const oldSequence = /[a-zA-Z0-9_-]+( *, *[a-zA-Z0-9_-]+ *)*/ const observedPatterns: RegExp[] = [] @@ -655,22 +655,22 @@ describe("tasks.extractTasks", () => { this: RegExp, text, ) { - if (this.source.includes("⛔") || this.source.includes("dependsOn::")) { + if (this.source.includes("⛔")) { observedPatterns.push(this) } return originalExec.call(this, text) }) try { - tasks.extractTasks("- [ ] x") + tasks.extractTasks("- [ ] x [dependsOn:: a]") } finally { execSpy.mockRestore() } - const dependencyPatterns = observedPatterns.filter((pattern) => + const dependencyPatterns = [...new Set(observedPatterns)].filter((pattern) => pattern.source.includes("[a-zA-Z0-9_-]+"), ) - expect(dependencyPatterns.length).toBe(2) + expect(dependencyPatterns.length).toBe(1) // State advances once per draw so this corpus is reproducible. let randomState = 0x5eed1234 @@ -702,13 +702,7 @@ describe("tasks.extractTasks", () => { const lines = [...fixtureLines, ...generatedLines] for (const pattern of dependencyPatterns) { - const oldPattern = new RegExp( - pattern.source.replace( - "[a-zA-Z0-9_-]+(?:(?: *, *[a-zA-Z0-9_-]+)+ *(?! ))?", - oldSequence.source, - ), - pattern.flags, - ) + const oldPattern = new RegExp(`⛔\\uFE0F? *(${oldSequence.source})$`) expect(oldPattern.source).not.toBe(pattern.source) for (const line of lines) { @@ -1419,7 +1413,7 @@ describe("task line mutations", () => { expect(result).toBe("- [ ] Task ➕ 2026-07-01") }) - it("replacing a duplicated priority keeps the first and swaps the last", () => { + it("inserts a replacement priority before the surviving duplicate", () => { const result = tasks.updateTaskLinePriority({ taskLine: "- [ ] Task ⏫ ➕ 2026-07-01 ⏫", newPriority: "highest", @@ -1625,6 +1619,57 @@ describe("task line mutations", () => { // ── replaceTaskLineDescription ──────────────────────────────── + describe("Dataview fields adjacent to hashtags", () => { + const fields = [ + { label: "square ID", field: "[id:: a]" }, + { label: "parenthesized ID", field: "(id:: a)" }, + { label: "square due date", field: "[due:: 2026-01-02]" }, + { label: "parenthesized due date", field: "(due:: 2026-01-02)" }, + { label: "square priority", field: "[priority:: high]" }, + { label: "parenthesized priority", field: "(priority:: high)" }, + ] + + it.each(fields)("preserves a $label when replacing the description", ({ field }) => { + expect( + tasks.replaceTaskLineDescription({ + taskLine: `- [ ] work #tag${field}`, + newDescription: "New", + }), + ).toBe(`- [ ] New ${field}`) + }) + + it.each(fields.slice(0, 2))("clears a $label after a hashtag", ({ field }) => { + expect( + tasks.updateTaskLineTaskId({ + taskLine: `- [ ] work #tag${field}`, + taskId: null, + config: DATAVIEW_CONFIG, + }), + ).toBe("- [ ] work #tag") + }) + + it.each(fields.slice(2, 4))("clears a $label after a hashtag", ({ field }) => { + expect( + tasks.updateTaskLineDate({ + taskLine: `- [ ] work #tag${field}`, + field: "due", + date: null, + config: DATAVIEW_CONFIG, + }), + ).toBe("- [ ] work #tag") + }) + + it.each(fields.slice(4))("clears a $label after a hashtag", ({ field }) => { + expect( + tasks.updateTaskLinePriority({ + taskLine: `- [ ] work #tag${field}`, + newPriority: null, + config: DATAVIEW_CONFIG, + }), + ).toBe("- [ ] work #tag") + }) + }) + describe("replaceTaskLineDescription", () => { it("replaces description before metadata signifiers", () => { const line = "- [ ] Old description ⏫ ➕ 2026-08-01 ^my-task" @@ -2291,7 +2336,7 @@ describe("task line mutations", () => { expect(result).toBe("- [ ] My task 🏁 delete ➕ 2026-08-01 📅 2026-09-01 ^my-task") }) - it("clearing strips duplicated emoji-format fields", () => { + it("clearing strips duplicated emoji onCompletion fields", () => { const line = "- [ ] My task 🏁 delete 🏁 keep ➕ 2026-08-01 ^my-task" const result = tasks.updateTaskLineOnCompletion({ taskLine: line, @@ -2850,7 +2895,7 @@ describe("task line mutations", () => { ]) }) - it("resets the indent stack at heading boundaries", () => { + it("attributes top-level tasks to the next heading after a nested list", () => { const content = [ "---", @@ -2892,6 +2937,34 @@ describe("task line mutations", () => { ]) }) + it("starts an indented task after a heading without a parent from the prior section", () => { + const content = [ + "## Active", + "- [ ] Parent ^parent", + " - [ ] Child ^child", + "## Up Next", + " - [ ] Independent ^independent", + ].join("\n") + + expect(tasks.extractTasks(content)).toEqual([ + task({ line: 2, description: "Parent", blockId: "parent", heading: "Active" }), + task({ + line: 3, + description: "Child", + blockId: "child", + heading: "Active", + depth: 1, + parentLine: 2, + }), + task({ + line: 5, + description: "Independent", + blockId: "independent", + heading: "Up Next", + }), + ]) + }) + it("handles non-task lines between parent and child", () => { const content = [ diff --git a/src/vault-mcp/obsidian-markdown/canvas.ts b/src/vault-mcp/obsidian-markdown/canvas.ts index 3d6e5a2dd..e882d922e 100644 --- a/src/vault-mcp/obsidian-markdown/canvas.ts +++ b/src/vault-mcp/obsidian-markdown/canvas.ts @@ -10,8 +10,9 @@ import { posix } from "node:path" * opaque node ids — a reader consuming the raw JSON spends most of its * attention on `x`/`y`/`width`/`height` noise and mental id-joins. This * transform keeps exactly what carries meaning and re-expresses it as - * document structure: spatial grouping becomes heading nesting, canvas - * position becomes reading order (top-to-bottom, left-to-right), and + * document structure: spatial grouping becomes heading nesting within the + * ownership-work budget, canvas position becomes reading order (top-to-bottom, + * left-to-right), and * edges become an `A → B (label)` list with ids resolved to display * names. Geometry is dropped entirely — it is presentation, and the * rendition is a reading surface, not a round-trippable format (callers @@ -25,8 +26,7 @@ import { posix } from "node:path" * * Lenient by design: real Obsidian canvases carry extra properties and * occasionally partial entries, so unknown props are ignored and entries - * missing their required fields are skipped rather than thrown. Only - * unparseable JSON throws. + * missing their required fields are skipped rather than thrown. * * The module also extracts and rewrites the links a canvas holds, for the * search index and the note mover. Obsidian calls nodes "cards": a text card @@ -174,25 +174,48 @@ const renderNode = (node: CanvasNode): string => { return `[${node.type}]` } -/** Renders a group section: heading, member nodes in reading order, then - * child groups recursively one heading level deeper (capped at H6). */ -const renderGroup = ( - group: CanvasNode, - depth: number, - membersByGroupId: ReadonlyMap, - childGroupsByParentId: ReadonlyMap, -): string => { - // H1 is the canvas overview line, so top-level groups (depth 0) start at H2. +// Bounds both ownership axes while retaining nesting for ordinary multi-group canvases. +const CANVAS_GROUP_WORK_BUDGET = 16_000_000 + +export const canNestCanvasGroups = (counts: { contentNodes: number; groups: number }): boolean => { + return counts.groups * (counts.contentNodes + counts.groups) <= CANVAS_GROUP_WORK_BUDGET +} + +const groupHeading = (group: CanvasNode, depth: number): string => { + // H1 is the overview, so groups start at H2 and deeper groups cap at H6. const heading = "#".repeat(Math.min(2 + depth, 6)) - const members = membersByGroupId.get(group.id) ?? [] - const childGroups = childGroupsByParentId.get(group.id) ?? [] - return [ - `${heading} Group: ${group.label ?? "(unlabeled)"}`, - ...members.map(renderNode), - ...childGroups.map((child) => - renderGroup(child, depth + 1, membersByGroupId, childGroupsByParentId), - ), - ].join("\n\n") + return `${heading} Group: ${group.label ?? "(unlabeled)"}` +} + +/** Buckets keyed by node identity append each section once, even when IDs repeat. */ +const renderNestedNodes = ({ + contentNodes, + groups, +}: { + contentNodes: readonly CanvasNode[] + groups: readonly CanvasNode[] +}): string[] => { + const membersByGroup = groupBy(contentNodes, (node) => smallestContainingGroup(node, groups)) + const childGroupsByParent = groupBy(groups, (group) => smallestContainingGroup(group, groups)) + const sections = (membersByGroup.get(undefined) ?? []).map(renderNode) + const frames = (childGroupsByParent.get(undefined) ?? []) + .toReversed() + .map((group) => ({ group, depth: 0 })) + + // Reverse pushes retain the buckets' reading order during depth-first traversal. + while (frames.length) { + const frame = frames.pop() + + if (!frame) break + const { group, depth } = frame + + sections.push(groupHeading(group, depth)) + for (const member of membersByGroup.get(group) ?? []) sections.push(renderNode(member)) + for (const child of (childGroupsByParent.get(group) ?? []).toReversed()) { + frames.push({ group: child, depth: depth + 1 }) + } + } + return sections } /** Groups a list by a key function, preserving each bucket's insertion order. */ @@ -230,7 +253,9 @@ const parseCanvasJson = (canvasJson: string): unknown => { * Linearizes .canvas JSON into readable markdown: an overview line, ungrouped * node content first, each group's content under a `Group:` heading (nested * groups one level deeper), then a `Connections` edge list in `A → B (label)` - * form with ids resolved to display names. Throws only on unparseable JSON. + * form with ids resolved to display names. Above the ownership-work budget, + * node content and group headings stay flat in reading order. Throws on + * unparseable JSON. Every valid node is rendered once, even when IDs repeat. */ export const linearizeCanvas = (canvasJson: string): string => { const parsed = parseCanvasJson(canvasJson) @@ -245,14 +270,13 @@ export const linearizeCanvas = (canvasJson: string): string => { const groups = nodes.filter((node) => node.type === "group") const contentNodes = nodes.filter((node) => node.type !== "group") - const membersByGroupId = groupBy( - contentNodes, - (node) => smallestContainingGroup(node, groups)?.id, - ) - const childGroupsByParentId = groupBy( - groups, - (group) => smallestContainingGroup(group, groups)?.id, - ) + const keepNesting = canNestCanvasGroups({ + contentNodes: contentNodes.length, + groups: groups.length, + }) + const nodeSections = keepNesting + ? renderNestedNodes({ contentNodes, groups }) + : nodes.map((node) => (node.type === "group" ? groupHeading(node, 0) : renderNode(node))) const nodeById = new Map(nodes.map((node) => [node.id, node])) const edgeEndpointName = (id: string): string => { @@ -260,10 +284,6 @@ export const linearizeCanvas = (canvasJson: string): string => { return node ? displayName(node) : `(missing node "${id}")` } - // A node outside every group has no containing group id, so groupBy filed - // it under the key undefined. - const ungroupedNodes = membersByGroupId.get(undefined) ?? [] - const topLevelGroups = childGroupsByParentId.get(undefined) ?? [] const edgeLines = edges.map((edge) => { const label = edge.label ? ` (${edge.label})` : "" return `${edgeEndpointName(edge.fromNode)} → ${edgeEndpointName(edge.toNode)}${label}` @@ -271,10 +291,7 @@ export const linearizeCanvas = (canvasJson: string): string => { const sections = [ `# Canvas: ${nodes.length} ${nodes.length === 1 ? "node" : "nodes"}, ${edges.length} ${edges.length === 1 ? "edge" : "edges"}`, - ...ungroupedNodes.map(renderNode), - ...topLevelGroups.map((group) => - renderGroup(group, 0, membersByGroupId, childGroupsByParentId), - ), + ...nodeSections, ...(edgeLines.length > 0 ? [["## Connections", ...edgeLines].join("\n\n")] : []), ] return sections.join("\n\n") @@ -283,7 +300,7 @@ export const linearizeCanvas = (canvasJson: string): string => { /** Extracts deduplicated vault-relative file paths from canvas `file`-type * nodes. Text-node wikilinks are deliberately excluded — only explicit file * references create graph edges (matching Obsidian's behavior). Throws on - * unparseable JSON (same contract as `linearizeCanvas`). */ + * unparseable JSON. */ export const extractCanvasFileLinks = (canvasJson: string): string[] => { const parsed = parseCanvasJson(canvasJson) @@ -303,7 +320,7 @@ export const extractCanvasFileLinks = (canvasJson: string): string[] => { /** The text of every text-type node, in JSON order. Obsidian updates the * links in these cards when their target is renamed, so the note mover * reads them to find a Canvas that links to a moving note. Throws on - * unparseable JSON (same contract as `linearizeCanvas`). */ + * unparseable JSON. */ export const extractCanvasCardTexts = (canvasJson: string): string[] => { const parsed = parseCanvasJson(canvasJson) diff --git a/src/vault-mcp/obsidian-markdown/frontmatter.ts b/src/vault-mcp/obsidian-markdown/frontmatter.ts index e72efd843..3cdd76d3b 100644 --- a/src/vault-mcp/obsidian-markdown/frontmatter.ts +++ b/src/vault-mcp/obsidian-markdown/frontmatter.ts @@ -1,6 +1,16 @@ import matter from "gray-matter" -import { LineCounter, parseDocument, stringify as stringifyYaml, visit } from "yaml" -import type { Document } from "yaml" +import { + isMap, + isNode, + isScalar, + isSeq, + LineCounter, + parseDocument, + stringify as stringifyYaml, + visit, + YAMLParseError, +} from "yaml" +import type { Document, Pair } from "yaml" import { splitIntoLines } from "./lines.js" /** A parsed note, with its properties in `data` and its body in `content`. */ @@ -231,6 +241,72 @@ const convertDocumentToValue = ( } } +type DuplicateKeyStep = + | { kind: "node"; node: unknown } + | { kind: "key"; pair: Pair; nodes: Set; values: Set } + +/** Preserves the YAML composer's duplicate-key error position. */ +const duplicateKeyOffset = (pair: Pair): number => { + const lastPrefixToken = pair.srcToken?.start.at(-1) + + // For `?\n: value`, the prefix ends at the colon; the empty key's range starts earlier. + if (lastPrefixToken) return lastPrefixToken.offset + lastPrefixToken.source.length + if (pair.srcToken?.key) return pair.srcToken.key.offset + if (isNode(pair.key) && pair.key.range) return pair.key.range[0] + return pair.srcToken?.sep?.[0]?.offset ?? 0 +} + +/** Mirrors composition order without the library's repeated scan of all earlier keys. */ +const findFirstDuplicateKey = (document: Document): YAMLParseError | undefined => { + const steps: DuplicateKeyStep[] = [{ kind: "node", node: document.contents }] + + // A stack interleaves each key, duplicate check and value in native composition order. + while (steps.length) { + const step = steps.pop() + + if (!step) break + if (step.kind === "key") { + const { pair, nodes, values } = step + const key = pair.key + + // YAML compares scalar values with ===; unlike Set equality, that keeps NaN keys distinct. + const hasScalarValue = isScalar(key) && !Number.isNaN(key.value) + const repeated = nodes.has(key) || (hasScalarValue && values.has(key.value)) + + if (repeated) { + const offset = duplicateKeyOffset(pair) + return new YAMLParseError([offset, offset + 1], "DUPLICATE_KEY", "Map keys must be unique") + } + nodes.add(key) + if (hasScalarValue) values.add(key.value) + continue + } + const { node } = step + + if (isMap(node)) { + const nodes = new Set() + const values = new Set() + for (const pair of node.items.toReversed()) { + const check: DuplicateKeyStep = { kind: "key", pair, nodes, values } + const value: DuplicateKeyStep = { kind: "node", node: pair.value } + + // LIFO scheduling visits key → value → check in flow maps, key → check → value in blocks. + if (node.flow) { + steps.push(check, value) + } else { + steps.push(value, check) + } + steps.push({ kind: "node", node: pair.key }) + } + continue + } + if (isSeq(node)) { + for (const item of node.items.toReversed()) steps.push({ kind: "node", node: item }) + } + } + return undefined +} + /** * Reads a raw properties block with the `yaml` package (YAML 1.2 core * schema). That schema reads no untagged value as a timestamp, so datetime @@ -246,8 +322,14 @@ const readPropertiesBlock = (blockText: string): PropertiesBlockReading => { const lineCounter = new LineCounter() // prettyErrors would append its own position and a multi-line excerpt of // the block to each message; the message below states the position itself - const document = parseDocument(blockText, { prettyErrors: false, lineCounter }) - const [firstError] = document.errors + const document = parseDocument(blockText, { + prettyErrors: false, + lineCounter, + uniqueKeys: false, + keepSourceTokens: true, + }) + // Syntax failures may take precedence; duplicate-only blocks retain the native first error. + const firstError = document.errors[0] ?? findFirstDuplicateKey(document) if (firstError) { const [errorStartOffset] = firstError.pos diff --git a/src/vault-mcp/obsidian-markdown/headings.ts b/src/vault-mcp/obsidian-markdown/headings.ts index 128e1fc3c..14cce2541 100644 --- a/src/vault-mcp/obsidian-markdown/headings.ts +++ b/src/vault-mcp/obsidian-markdown/headings.ts @@ -25,10 +25,25 @@ export type HeadingInfo = Readonly<{ * Empty headings (`##` alone on a line) are valid — group 2 is undefined. */ const HEADING_REGEX = /^ {0,3}(#{1,6})(?:[ \t](.*))?$/ -/** Matches an ATX heading's optional closing sequence per CommonMark §4.2: one +/** Removes an ATX heading's optional closing sequence per CommonMark §4.2: one * or more `#` at the end of the text, after a space or tab or as the whole * text. `## Title ##` reads as `Title`, and `### ###` as an empty heading. */ -const CLOSING_HASHES_REGEX = /(?:^|[ \t]+)#+[ \t]*$/ +const stripClosingHashes = (text: string): string => { + // Each suffix run is consumed once, including a failed space-only suffix. + let boundary = text.length + + while (boundary > 0 && (text[boundary - 1] === " " || text[boundary - 1] === "\t")) boundary -= 1 + const hashEnd = boundary + + while (boundary > 0 && text[boundary - 1] === "#") boundary -= 1 + if (boundary === hashEnd) return text + const hashStart = boundary + + while (boundary > 0 && (text[boundary - 1] === " " || text[boundary - 1] === "\t")) boundary -= 1 + const hasClosingHashSeparator = hashStart === 0 || boundary < hashStart + + return hasClosingHashSeparator ? text.slice(0, boundary) : text +} /** Matches setext heading underlines per CommonMark §4.3: 0-3 leading spaces, * one or more `=` (H1) or `-` (H2) characters, optional trailing whitespace. @@ -181,7 +196,7 @@ export const parseHeadings = (lines: readonly string[]): HeadingInfo[] => { if (matchedHashes) { const matchedText = atxMatch?.[2] ?? "" collectedHeadings.push({ - text: matchedText.replace(CLOSING_HASHES_REGEX, "").trim(), + text: stripClosingHashes(matchedText).trim(), level: matchedHashes.length, startLine: i, bodyStartLine: i + 1, @@ -220,20 +235,22 @@ export const parseHeadings = (lines: readonly string[]): HeadingInfo[] => { // run to EOF, but must stop before a trailing `%% %%` comment block (e.g. a // Kanban board's `%% kanban:settings %%`) so replace/append don't clobber it. const trailingCommentBlockStart = findTrailingCommentBlockStart(lines) - return collectedHeadings.map((heading, index) => { - const nextSameOrHigher = collectedHeadings - .slice(index + 1) - .find((next) => next.level <= heading.level) - return { - text: heading.text, - level: heading.level, - startLine: heading.startLine, - bodyStartLine: heading.bodyStartLine, - // Math.max keeps bodyEndLine >= bodyStartLine for malformed input. - bodyEndLine: - nextSameOrHigher?.startLine ?? Math.max(heading.bodyStartLine, trailingCommentBlockStart), + const completedHeadings = collectedHeadings.map((heading) => ({ + ...heading, + bodyEndLine: Math.max(heading.bodyStartLine, trailingCommentBlockStart), + })) + const openHeadings: Array<(typeof completedHeadings)[number]> = [] + + // Each heading is pushed and closed once; only same-or-deeper levels close here. + for (const heading of completedHeadings) { + while (openHeadings.length && (openHeadings.at(-1)?.level ?? 0) >= heading.level) { + const closedHeading = openHeadings.pop() + + if (closedHeading) closedHeading.bodyEndLine = heading.startLine } - }) + openHeadings.push(heading) + } + return completedHeadings } /** diff --git a/src/vault-mcp/obsidian-markdown/links.ts b/src/vault-mcp/obsidian-markdown/links.ts index 79d6d16da..f604bba92 100644 --- a/src/vault-mcp/obsidian-markdown/links.ts +++ b/src/vault-mcp/obsidian-markdown/links.ts @@ -4,44 +4,30 @@ * (vault-operations/note-mover.ts) depend on this module as a peer, so the two * can never disagree about what a link is or where it points. * - * The raw `/g` grammar regexes are module-private: a shared global regex carries - * `lastIndex` between calls, which corrupts iteration the moment a caller reaches - * for `.exec()`-in-a-loop or `.test()`. Callers instead use the position-safe - * methods on `links` (matchLinksInLine, inlineCodeSpans), which drive the - * regexes only via matchAll/replaceAll internally. */ + * Recognition stays private so callers share the grammar and receive + * position-safe matches through the methods on `links`. */ import { posix } from "node:path" import { classifyLines } from "./lines.js" // ── Grammar (private) ─────────────────────────────────────────── -/** Matches wikilinks: [[target]], [[target|text]], [[target#heading]], - * [[target#heading|text]], and embeds ![[target]]. Captures the target - * path/name (group 1) before any # or |. Global — use only with matchAll. */ -const WIKILINK_RE = /!?\[\[([^\]#|]+)(?:#[^\]|]*)?(?:\|[^\]]+)?\]\]/g - -/** Matches markdown internal links and embeds to any vault target — - * [text](Note.md), ![alt](image.png), [doc](file.pdf), [text](Extensionless%20Note), - * each optionally with a #heading. Obsidian resolves all of these, so - * recognition must too. External targets are excluded by a URI-scheme - * lookahead ([a-zA-Z][a-zA-Z0-9+.-]*: is RFC 3986 scheme grammar; both letter - * cases because schemes are case-insensitive, and a colon is illegal in - * Obsidian filenames so a scheme-like prefix is never a vault path), plus - * same-page #anchors. Captures the full target including any extension - * (group 1). Global — use only with matchAll. */ -const MD_LINK_RE = /\[[^\]]*\]\((?![a-zA-Z][a-zA-Z0-9+.-]*:|#)([^)#\s]+)(?:#[^)\s]*)?\)/g +/** JavaScript whitespace also includes non-ASCII spaces and line terminators. */ +const WHITESPACE = /\s/ + +/** A URI scheme begins with an ASCII letter. */ +const SCHEME_START = /[a-zA-Z]/ + +/** RFC 3986 permits these characters after a URI scheme's first letter. */ +const SCHEME_CHARACTER = /[a-zA-Z0-9+.-]/ + +/** A dot without the s flag stops at each of these line terminators. */ +const LINE_TERMINATOR = /[\n\r\u2028\u2029]/ /** Matches inline code spans so links inside backticks (e.g. `[[Note]]`) can be * ignored. Global — use only with matchAll/replaceAll. */ const INLINE_CODE_RE = /`+[^`\n]*`+/g -/** Matches Templater expressions so links inside template directives - * (e.g. `<% tp.date.now("YYYY-MM-DD", 1, tp.file.title, "YYYY-MM-DD") %>`) - * are not extracted as link targets. Covers all Templater run-mode variants: - * `<%` `<%+` `<%*` `<%-` `<%_` `<%~`. Single-line only — consistent - * with INLINE_CODE_RE. Global — use only with matchAll/replaceAll. */ -const TEMPLATER_RE = /<%[-+*_~]?.*?%>/g - /** Splits a matched wikilink into [, embed `!`, target, `#heading`, `|alias`]. * Anchored and non-global — safe for .exec(). */ const WIKILINK_PARTS = /^(!?)\[\[([^\]#|]+)(#[^\]|]*)?(\|[^\]]+)?\]\]$/ @@ -82,6 +68,8 @@ type LinkMatch = { /** The half-open character range [start, end) of an inline code span in a line. */ type CodeSpan = { start: number; end: number } +type TargetSpan = CodeSpan & { targetStart: number; targetEnd: number } + /** The structural parts of a wikilink ![[target#heading|alias]]. heading/alias * keep their leading `#` / `|` (or `\|` for table-escaped pipes) so * reconstruction is plain concatenation; target is not trimmed (the caller @@ -108,22 +96,130 @@ type MarkdownLinkParts = { // ── Line scanning ─────────────────────────────────────────────── +/** Delimiter cursors only advance, so failed openers share suffix scans. */ +const scanWikilinks = (text: string, visit: (span: TargetSpan) => void): void => { + // An overlapping opener before a cached delimiter has no intervening delimiter to find. + let targetEnd = 0 + let headingEnd = 0 + let closeBracket = 0 + + for (let start = 0; start < text.length; start++) { + const open = text[start] === "!" ? start + 1 : start + + if (text[open] !== "[" || text[open + 1] !== "[") continue + const targetStart = open + 2 + targetEnd = Math.max(targetEnd, targetStart) + + while (targetEnd < text.length && !"]#|".includes(text.charAt(targetEnd))) targetEnd++ + if (targetEnd === targetStart) continue + // The grammar permits an empty heading, but requires a nonempty alias. + let end = targetEnd + + if (text[end] === "#") { + headingEnd = Math.max(headingEnd, end + 1) + while (headingEnd < text.length && !"]|".includes(text.charAt(headingEnd))) headingEnd++ + end = headingEnd + } + if (text[end] === "|") { + const aliasStart = end + 1 + closeBracket = Math.max(closeBracket, aliasStart) + while (closeBracket < text.length && text[closeBracket] !== "]") closeBracket++ + if (closeBracket === aliasStart) continue + end = closeBracket + } + if (text[end] !== "]" || text[end + 1] !== "]") continue + visit({ start, end: end + 2, targetStart, targetEnd }) + start = end + 1 + } +} + +/** URI schemes follow RFC 3986, including uppercase letters and punctuation. */ +const scanMarkdownLinks = (text: string, visit: (span: TargetSpan) => void): void => { + // Later openers reuse the same ] or reach a later one, so ]( target starts never move back. + // A cached scheme end stays valid until a target starts beyond it. + let textEnd = 0 + let targetEnd = 0 + let fragmentEnd = 0 + let schemeEnd = 0 + + for (let start = 0; start < text.length; start++) { + if (text[start] !== "[") continue + textEnd = Math.max(textEnd, start + 1) + while (textEnd < text.length && text[textEnd] !== "]") textEnd++ + if (text[textEnd + 1] !== "(") continue + const targetStart = textEnd + 2 + + if (text[targetStart] === "#") continue + if (SCHEME_START.test(text.charAt(targetStart))) { + schemeEnd = Math.max(schemeEnd, targetStart + 1) + while (schemeEnd < text.length && SCHEME_CHARACTER.test(text.charAt(schemeEnd))) schemeEnd++ + if (text[schemeEnd] === ":") continue + } + targetEnd = Math.max(targetEnd, targetStart) + while ( + targetEnd < text.length && + text[targetEnd] !== ")" && + text[targetEnd] !== "#" && + !WHITESPACE.test(text.charAt(targetEnd)) + ) + targetEnd++ + if (targetEnd === targetStart) continue + // A fragment may be empty; whitespace still ends the attempted match. + let end = targetEnd + + if (text[end] === "#") { + fragmentEnd = Math.max(fragmentEnd, end + 1) + while ( + fragmentEnd < text.length && + text[fragmentEnd] !== ")" && + !WHITESPACE.test(text.charAt(fragmentEnd)) + ) + fragmentEnd++ + end = fragmentEnd + } + if (text[end] !== ")") continue + visit({ start, end: end + 1, targetStart, targetEnd }) + start = end + } +} + +/** Line terminators stop Templater expressions even when the caller supplies a whole note. */ +const maskTemplaterExpressions = (text: string): string => { + const pieces: string[] = [] + // The closer cursor is shared by every failed opener before it. + let close = 0 + let copiedThrough = 0 + + for (let start = 0; start < text.length; start++) { + if (text[start] !== "<" || text[start + 1] !== "%") continue + close = Math.max(close, start + 2) + while (close < text.length && !LINE_TERMINATOR.test(text.charAt(close))) { + if (text[close] === "%" && text[close + 1] === ">") break + close++ + } + if (text[close] !== "%" || text[close + 1] !== ">") continue + const end = close + 2 + pieces.push(text.slice(copiedThrough, start), " ".repeat(end - start)) + copiedThrough = end + start = end - 1 + } + if (!copiedThrough) return text + pieces.push(text.slice(copiedThrough)) + return pieces.join("") +} + /** Finds wikilinks, then markdown links (not in text order), with the offsets a rewriter * needs to splice replacements. Links in inline code are kept for each caller to exclude. */ -const matchLinksInLine = (line: string): LinkMatch[] => [ - ...[...line.matchAll(WIKILINK_RE)].map((match) => ({ - text: match[0], - start: match.index, - end: match.index + match[0].length, - kind: "wikilink" as const, - })), - ...[...line.matchAll(MD_LINK_RE)].map((match) => ({ - text: match[0], - start: match.index, - end: match.index + match[0].length, - kind: "markdown" as const, - })), -] +const matchLinksInLine = (line: string): LinkMatch[] => { + const matches: LinkMatch[] = [] + scanWikilinks(line, ({ start, end }) => { + matches.push({ text: line.slice(start, end), start, end, kind: "wikilink" }) + }) + scanMarkdownLinks(line, ({ start, end }) => { + matches.push({ text: line.slice(start, end), start, end, kind: "markdown" }) + }) + return matches +} /** Returns the character ranges of inline code spans in a line, so a rewriter can * leave links inside backticks untouched. */ @@ -197,26 +293,21 @@ const extractFromBody = (content: string): string[] => { if (inCode) continue // Spaces keep text on either side of an ignored span from forming a new link. - const linkExtractableLine = text - .replace(INLINE_CODE_RE, (match) => " ".repeat(match.length)) - .replace(TEMPLATER_RE, (match) => " ".repeat(match.length)) + const withoutCode = text.replace(INLINE_CODE_RE, (match) => " ".repeat(match.length)) + const linkExtractableLine = maskTemplaterExpressions(withoutCode) - for (const match of linkExtractableLine.matchAll(WIKILINK_RE)) { - const rawTarget = match[1] - - if (rawTarget === undefined) continue + scanWikilinks(linkExtractableLine, ({ targetStart, targetEnd }) => { + const rawTarget = linkExtractableLine.slice(targetStart, targetEnd) const target = stripEscapedPipe(rawTarget.trim()) if (target.length > 0) targets.add(target) - } - for (const match of linkExtractableLine.matchAll(MD_LINK_RE)) { - const rawTarget = match[1] - - if (rawTarget === undefined) continue + }) + scanMarkdownLinks(linkExtractableLine, ({ targetStart, targetEnd }) => { + const rawTarget = linkExtractableLine.slice(targetStart, targetEnd) const target = safeDecodeURIComponent(rawTarget.trim()) if (target.length > 0) targets.add(target) - } + }) } return [...targets] } @@ -238,14 +329,12 @@ const extractFromFrontmatter = (data: Record): string[] => { const collectWikilinksFrom = (frontmatterValue: unknown): void => { // Leaf: a string may hold one or more wikilinks — pull out each target. if (typeof frontmatterValue === "string") { - for (const match of frontmatterValue.matchAll(WIKILINK_RE)) { - const rawTarget = match[1] - - if (rawTarget === undefined) continue + scanWikilinks(frontmatterValue, ({ targetStart, targetEnd }) => { + const rawTarget = frontmatterValue.slice(targetStart, targetEnd) const target = stripEscapedPipe(rawTarget.trim()) if (target.length > 0) targets.add(target) - } + }) return } // Array (e.g. a multi-value `related:`): recurse into every element. diff --git a/src/vault-mcp/obsidian-markdown/plaintext.ts b/src/vault-mcp/obsidian-markdown/plaintext.ts index 94846aa6f..0c7014f3e 100644 --- a/src/vault-mcp/obsidian-markdown/plaintext.ts +++ b/src/vault-mcp/obsidian-markdown/plaintext.ts @@ -3,16 +3,123 @@ * Pure transform: string in, string out. The result preserves semantic content * (words, sentences) while removing markup that degrades embedding quality. */ +/** JavaScript whitespace includes line terminators and non-ASCII spaces. */ +const WHITESPACE = /\s/ + +/** Multiline anchors also recognize these separators beside LF and CR. */ +const LINE_TERMINATOR = /[\n\r\u2028\u2029]/ + +const stripWikilinkAliases = (text: string): string => { + const pieces: string[] = [] + // The first | or ] still decides every overlapping opener before it; no earlier delimiter exists. + let targetEnd = 0 + let close = 0 + let copiedThrough = 0 + + for (let start = 0; start < text.length; start++) { + if (text[start] !== "[" || text[start + 1] !== "[") continue + const targetStart = start + 2 + targetEnd = Math.max(targetEnd, targetStart) + while (targetEnd < text.length && text[targetEnd] !== "|" && text[targetEnd] !== "]") + targetEnd++ + if (targetEnd === targetStart || text[targetEnd] !== "|") continue + const aliasStart = targetEnd + 1 + close = Math.max(close, aliasStart) + while (close < text.length && text[close] !== "]") close++ + if (close === aliasStart || text[close + 1] !== "]") continue + pieces.push(text.slice(copiedThrough, start), text.slice(aliasStart, close)) + copiedThrough = close + 2 + start = close + 1 + } + if (!copiedThrough) return text + pieces.push(text.slice(copiedThrough)) + return pieces.join("") +} + +const stripWikilinks = (text: string): string => { + const pieces: string[] = [] + // A failed opener never restarts the search for the next closing bracket. + let close = 0 + let copiedThrough = 0 + + for (let start = 0; start < text.length; start++) { + if (text[start] !== "[" || text[start + 1] !== "[") continue + const targetStart = start + 2 + close = Math.max(close, targetStart) + while (close < text.length && text[close] !== "]") close++ + if (close === targetStart || text[close + 1] !== "]") continue + pieces.push(text.slice(copiedThrough, start), text.slice(targetStart, close)) + copiedThrough = close + 2 + start = close + 1 + } + if (!copiedThrough) return text + pieces.push(text.slice(copiedThrough)) + return pieces.join("") +} + +const stripMarkdownLinks = (text: string): string => { + const pieces: string[] = [] + // Labels may overlap; the closing delimiters are still visited only once. + let labelEnd = 0 + let close = 0 + let copiedThrough = 0 + + for (let start = 0; start < text.length; start++) { + if (text[start] !== "[") continue + labelEnd = Math.max(labelEnd, start + 1) + while (labelEnd < text.length && text[labelEnd] !== "]") labelEnd++ + if (labelEnd === start + 1 || text[labelEnd + 1] !== "(") continue + const targetStart = labelEnd + 2 + close = Math.max(close, targetStart) + while (close < text.length && text[close] !== ")") close++ + if (close === targetStart || text[close] !== ")") continue + pieces.push(text.slice(copiedThrough, start), text.slice(start + 1, labelEnd)) + copiedThrough = close + 1 + start = close + } + if (!copiedThrough) return text + pieces.push(text.slice(copiedThrough)) + return pieces.join("") +} + +const stripCalloutMarkers = (text: string): string => { + const pieces: string[] = [] + // Whitespace can cross physical lines; both cursors keep that legacy grammar. + let markerStart = 0 + let close = 0 + let copiedThrough = 0 + + for (let start = 0; start < text.length; start++) { + const atLineStart = start === 0 || LINE_TERMINATOR.test(text.charAt(start - 1)) + + if (text[start] !== ">" || !atLineStart) continue + markerStart = Math.max(markerStart, start + 1) + while (markerStart < text.length && WHITESPACE.test(text.charAt(markerStart))) markerStart++ + if (text[markerStart] !== "[" || text[markerStart + 1] !== "!") continue + close = Math.max(close, markerStart + 2) + while (close < text.length && text[close] !== "]") close++ + if (text[close] !== "]") continue + // The trailing whitespace belongs to the removed marker too. + let end = close + 1 + + while (end < text.length && WHITESPACE.test(text.charAt(end))) end++ + pieces.push(text.slice(copiedThrough, start)) + copiedThrough = end + start = end - 1 + } + if (!copiedThrough) return text + pieces.push(text.slice(copiedThrough)) + return pieces.join("") +} + /** Strips markdown and Obsidian syntax, keeping only the semantic text content. * Wikilinks become their display text, comments are removed, formatting * markers (bold, italic, headings, callouts, block quotes) are stripped. */ -export const stripMarkdownSyntax = (text: string): string => - text - // Wikilinks: [[target|display]] → display, [[target]] → target - .replace(/\[\[([^|\]]+)\|([^\]]+)\]\]/g, "$2") - .replace(/\[\[([^\]]+)\]\]/g, "$1") - // Markdown links: [text](url) → text - .replace(/\[([^\]]+)\]\([^)]+\)/g, "$1") +export const stripMarkdownSyntax = (text: string): string => { + const withoutAliases = stripWikilinkAliases(text) + const withoutWikilinks = stripWikilinks(withoutAliases) + const withoutLinks = stripMarkdownLinks(withoutWikilinks) + const withoutFormatting = withoutLinks // Obsidian comments: %% ... %% (single-line and multi-line) .replace(/%%[^]*?%%/g, "") // Heading markers: ## → removed (keep the text; 0-3 leading spaces per §4.2, @@ -23,7 +130,7 @@ export const stripMarkdownSyntax = (text: string): string => // Bold/italic markers .replace(/\*{1,3}([^*]+)\*{1,3}/g, "$1") .replace(/_{1,3}([^_]+)_{1,3}/g, "$1") - // Callout markers: > [!type] Title → Title - .replace(/^>\s*\[![^\]]*\]\s*/gm, "") - // Block quotes: > text → text - .replace(/^>\s?/gm, "") + const withoutCallouts = stripCalloutMarkers(withoutFormatting) + // Block quotes: > text → text + return withoutCallouts.replace(/^>\s?/gm, "") +} diff --git a/src/vault-mcp/obsidian-markdown/remark-ports/__tests__/list.test.ts b/src/vault-mcp/obsidian-markdown/remark-ports/__tests__/list.test.ts new file mode 100644 index 000000000..4babcb6cf --- /dev/null +++ b/src/vault-mcp/obsidian-markdown/remark-ports/__tests__/list.test.ts @@ -0,0 +1,217 @@ +import { spawnSync } from "node:child_process" +import { readFileSync } from "node:fs" +import { createRequire } from "node:module" +import { Script } from "node:vm" +import ts from "typescript" +import { describe, expect, it } from "vitest" +import { list } from "../list.js" +import type { RemarkParserMembers } from "../../remark-parse-types.js" + +/** These values exercise every whitespace code point accepted by the replaced expression. */ +const WHITESPACE = [ + "\t", + "\n", + "\v", + "\f", + "\r", + " ", + "\u00a0", + "\u1680", + "\u2000", + "\u2001", + "\u2002", + "\u2003", + "\u2004", + "\u2005", + "\u2006", + "\u2007", + "\u2008", + "\u2009", + "\u200a", + "\u2028", + "\u2029", + "\u202f", + "\u205f", + "\u3000", + "\ufeff", +] +/** The fixed seed makes every equivalence failure repeatable. */ +const RANDOM_SEED = 0x712bca49 +/** This budget bounds a regressed predicate even when it blocks its own event loop. */ +const CHILD_TIMEOUT_MS = 2_000 +/** Repetition makes quadratic retries exceed the budget without timing ratios. */ +const TIMING_REPETITIONS = 20 +/** This expression is the MIT remark-parse 8.0.3 loose-list predicate. */ +const UPSTREAM_LOOSE_LIST_ITEM_RE = /\n\n(?!\s*$)/ + +const predicateSource = (): string => { + const sourcePath = new URL("../list.ts", import.meta.url) + const source = ts.createSourceFile( + "list.ts", + readFileSync(sourcePath, "utf8"), + ts.ScriptTarget.Latest, + ) + const declarations = source.statements + .filter(ts.isVariableStatement) + .flatMap((statement) => statement.declarationList.declarations) + const predicate = declarations.find( + (declaration) => declaration.name.getText(source) === "hasContentAfterBlankLine", + ) + const whitespace = declarations.find( + (declaration) => declaration.name.getText(source) === "WHITESPACE_RE", + ) + + if (!predicate?.initializer || !whitespace?.initializer) + throw new Error("list predicate is missing") + + const expression = `const WHITESPACE_RE = ${whitespace.initializer.getText(source)}; (${predicate.initializer.getText(source)})` + + return ts.transpile(expression, { target: ts.ScriptTarget.ESNext }) +} + +const isPredicate = (value: unknown): value is (content: string) => boolean => + typeof value === "function" + +const loadPredicate = (): ((content: string) => boolean) => { + const predicate: unknown = new Script(predicateSource()).runInNewContext() + + if (!isPredicate(predicate)) throw new Error("list predicate did not evaluate to a function") + + return predicate +} + +type Parser = RemarkParserMembers & { + setOptions: (options: { commonmark: boolean; gfm: boolean }) => void + parse: () => unknown +} + +type ParserConstructor = new (document: string, file: string) => Parser + +const isParserConstructor = (value: unknown): value is ParserConstructor => + typeof value === "function" + +const parseList = (document: string, listTokenizer?: typeof list): unknown => { + const require = createRequire(import.meta.url) + const Parser: unknown = require("remark-parse/lib/parser.js") + + if (!isParserConstructor(Parser)) throw new Error("remark parser is missing") + + const parser = new Parser(document, document) + + parser.setOptions({ commonmark: true, gfm: true }) + + if (listTokenizer) parser.blockTokenizers = { ...parser.blockTokenizers, list: listTokenizer } + + return parser.parse() +} + +const createRandom = (): (() => number) => { + // The seeded generator advances once for each character choice. + let state = RANDOM_SEED + + return () => { + state = (Math.imul(state, 1_664_525) + 1_013_904_223) >>> 0 + return state / 4_294_967_296 + } +} + +describe("list loose-item predicate", () => { + it.each([ + { label: "empty content", content: "", expected: false }, + { label: "no blank line", content: "a\nb", expected: false }, + { label: "trailing blank line", content: "a\n\n", expected: false }, + { label: "content after the first pair", content: "a\n\nb", expected: true }, + { label: "content after overlapping pairs", content: "a\n\n\n\nb", expected: true }, + { label: "content before trailing pairs", content: "a\n\nb\n\n", expected: true }, + { label: "CRLF blank line", content: "a\r\n\r\nb", expected: false }, + { label: "CR blank line", content: "a\r\rb", expected: false }, + { label: "CR after LF pair", content: "a\n\n\rb", expected: true }, + { label: "Unicode line endings", content: "a\u2028\u2029b", expected: false }, + { label: "zero-width space", content: "a\n\n\u200b", expected: true }, + { label: "next-line character", content: "a\n\n\u0085", expected: true }, + { label: "lone surrogate", content: "a\n\n\ud800", expected: true }, + ])("preserves $label", ({ content, expected }) => { + expect(loadPredicate()(content)).toBe(expected) + }) + + it.each( + WHITESPACE.map((character) => ({ character, label: character.charCodeAt(0).toString(16) })), + )("treats U+$label as trailing whitespace", ({ character }) => { + const predicate = loadPredicate() + + expect(predicate(`a\n\n${character.repeat(4)}`)).toBe(false) + expect(predicate(`a\n\n${character.repeat(4)}b`)).toBe(true) + }) + + it("matches the replaced expression on 10000 fixed-seed strings", () => { + const predicate = loadPredicate() + const random = createRandom() + const alphabet = [...WHITESPACE, "a", "#", "\u0085", "\u200b", "\ud800", "\udfff"] + const mismatches: { content: string; expected: boolean; actual: boolean }[] = [] + + for (let index = 0; index < 10_000; index += 1) { + const length = Math.floor(random() * 160) + const prefix = index % 2 === 0 ? "\n\n" : "" + const content = + prefix + + Array.from({ length }, () => alphabet[Math.floor(random() * alphabet.length)]).join("") + const expected = UPSTREAM_LOOSE_LIST_ITEM_RE.test(content) + const actual = predicate(content) + + if (actual !== expected) mismatches.push({ content, expected, actual }) + } + + expect(mismatches).toEqual([]) + }) + + it("finishes newline-only suffixes within a parent-enforced child deadline", () => { + const source = predicateSource() + + for (const length of [10_000, 20_000, 40_000]) { + const childSource = ` + const predicate = eval(${JSON.stringify(source)}); + const input = "\\n".repeat(${length}); + const start = performance.now(); + for (let index = 0; index < ${TIMING_REPETITIONS}; index += 1) { + if (predicate(input) !== false) throw new Error("newline-only content became loose"); + if (predicate(input + "x") !== true) throw new Error("content after newlines became tight"); + } + process.stdout.write(JSON.stringify({ length: ${length}, milliseconds: performance.now() - start, trailing: false, content: true })); + ` + const child = spawnSync(process.execPath, ["--input-type=module", "--eval", childSource], { + encoding: "utf8", + timeout: CHILD_TIMEOUT_MS, + killSignal: "SIGKILL", + }) + + expect({ + length, + error: child.error?.message, + signal: child.signal, + status: child.status, + stderr: child.stderr, + }).toEqual({ length, error: undefined, signal: null, status: 0, stderr: "" }) + const result: unknown = JSON.parse(child.stdout) + + expect(result).toEqual({ + length, + milliseconds: expect.any(Number), + trailing: false, + content: true, + }) + } + }) +}) + +describe("list tokenizer", () => { + it.each([ + { label: "tight paragraphs", document: "- a\n- b" }, + { label: "loose item", document: "- a\n\n b\n- c" }, + { label: "trailing indented blank lines", document: "- a\n \n \n- b" }, + { label: "CRLF lines", document: "- a\r\n\r\n b\r\n- c" }, + { label: "a single CRLF line break", document: "- a\r\n- b" }, + { label: "task and nested items", document: "- [x] a\n\n b\n - c" }, + ])("preserves the upstream AST for $label", ({ document }) => { + expect(parseList(document, list)).toEqual(parseList(document)) + }) +}) diff --git a/src/vault-mcp/obsidian-markdown/remark-ports/list.ts b/src/vault-mcp/obsidian-markdown/remark-ports/list.ts index efa9580d0..59b712a05 100644 --- a/src/vault-mcp/obsidian-markdown/remark-ports/list.ts +++ b/src/vault-mcp/obsidian-markdown/remark-ports/list.ts @@ -59,8 +59,8 @@ const RIGHT_PARENTHESIS = ")" const LOWERCASE_X = "x" const TAB_SIZE = 4 -/** Matches an item whose content holds a blank line followed by more content. */ -const LOOSE_LIST_ITEM_RE = /\n\n(?!\s*$)/ +/** Matches one whitespace character, including the Unicode whitespace in JavaScript's `\s`. */ +const WHITESPACE_RE = /\s/ /** Matches a task marker at the start of an item's content: any one * character between the brackets, then a space or tab. */ const TASK_ITEM_RE = /^\[(.)\][ \t]/ @@ -69,11 +69,26 @@ const TASK_ITEM_RE = /^\[(.)\][ \t]/ * the rest of the first line. */ const BULLET_RE = /^([ \t]*)([*+-]|\d+[.)])( {1,4}(?! )| |\t|$|(?=\n))([^\n]*)/ +/** A later blank line can only have a non-whitespace suffix when the first one does. */ +const hasContentAfterBlankLine = (content: string): boolean => { + const blankLineIndex = content.indexOf("\n\n") + + if (blankLineIndex === -1) return false + + // The counter advances through the suffix without retrying at later blank lines. + for (let index = blankLineIndex + 2; index < content.length; index += 1) { + if (!WHITESPACE_RE.test(content.charAt(index))) return true + } + + return false +} + export const isAsciiDigit = (character: string): boolean => character >= "0" && character <= "9" type ListItemLines = { value: string[] indent: number + /** Blank separators consumed after this item, outside its parsed content. */ trail: string[] } @@ -133,8 +148,10 @@ export const list = function ( index = 0 const items: ListItemLines[] = [] - let allLines: string[] = [] - let emptyLines: string[] = [] + let listLines: string[] = [] + + // Blank lines join the list only when a continuation or another item follows. + let pendingBlankLines: string[] = [] let item: ListItemLines | null = null let spread = false let empty = false @@ -163,23 +180,23 @@ export const list = function ( index = nextIndex + 1 if (indented && item) { - item.value = item.value.concat(emptyLines, line) - allLines = allLines.concat(emptyLines, line) - emptyLines = [] + item.value = item.value.concat(pendingBlankLines, line) + listLines = listLines.concat(pendingBlankLines, line) + pendingBlankLines = [] continue } if (prefixed) { - if (emptyLines.length !== 0 && item) { + if (pendingBlankLines.length !== 0 && item) { spread = true item.value.push("") - item.trail = [...emptyLines] + item.trail = [...pendingBlankLines] } item = { value: [line], indent: scan.size, trail: [] } items.push(item) - allLines = allLines.concat(emptyLines, line) - emptyLines = [] + listLines = listLines.concat(pendingBlankLines, line) + pendingBlankLines = [] continue } @@ -188,7 +205,7 @@ export const list = function ( // leaves; kept so the port reads against its source if (previousEmpty && !commonmark) break - emptyLines.push(line) + pendingBlankLines.push(line) continue } @@ -196,12 +213,13 @@ export const list = function ( if (interrupt(interruptors, tokenizers, this, [eat, line, true])) break if (!item) break - item.value = item.value.concat(emptyLines, line) - allLines = allLines.concat(emptyLines, line) - emptyLines = [] + item.value = item.value.concat(pendingBlankLines, line) + listLines = listLines.concat(pendingBlankLines, line) + pendingBlankLines = [] } - const node = eat(allLines.join(LINE_FEED)).reset({ + // reset records the parent span, then restores its text so each child can consume its own span. + const node = eat(listLines.join(LINE_FEED)).reset({ type: "list", ordered, start, @@ -369,13 +387,14 @@ const tokenizeListItem = ( const taskMarker = task ? task[0] : "" const taskCharacter = task?.[1] + // Nested tokenizeBlock keeps source coordinates; offsets restore the stripped marker widths. if (taskMarker) offsets[position.line] = (offsets[position.line] ?? 0) + taskMarker.length const content = withoutBullet.slice(taskMarker.length) return { type: "listItem", - spread: LOOSE_LIST_ITEM_RE.test(content), + spread: hasContentAfterBlankLine(content), checked: taskCharacter ? taskCharacter.toLowerCase() === LOWERCASE_X : null, children: parser.tokenizeBlock(content, position), } diff --git a/src/vault-mcp/obsidian-markdown/tasks.ts b/src/vault-mcp/obsidian-markdown/tasks.ts index 0fde5019f..a2e51d29a 100644 --- a/src/vault-mcp/obsidian-markdown/tasks.ts +++ b/src/vault-mcp/obsidian-markdown/tasks.ts @@ -99,7 +99,10 @@ export type ParsedTask = Readonly<{ * `1.` / `1)`), one or more spaces, and `[c]` with exactly one status * character. Captures: [1] status character, [2] everything after the * checkbox. Anchored and non-global — safe for .exec(). */ -const TASK_LINE_RE = /^[\s>]*(?:[-*+]|[0-9]+[.)]) +\[(.)\] *(.*)$/u +const TASK_LINE_RE = /^[\s>]*(?:[-*+]|[0-9]+[.)]) +\[(.)\](.*)$/u + +/** Matches ASCII spaces before a task body's content. */ +const LEADING_BODY_SPACES_RE = /^ +/ /** Matches any list item — the same prefix and marker grammar as * TASK_LINE_RE without the checkbox. A non-task item at a task's indent @@ -152,8 +155,8 @@ const HASHTAG_FROM_END_RE = /(^|\s)#[^ !@#$%^&*(),.?":{}|<>]+$/ // // Dataview regexes mirror toInlineFieldRegex(): the `key:: value` pair wrapped // in matching `[...]` or `(...)`, with an optional trailing comma, anchored to -// line end. Each opener's lookahead requires its own closer before the line -// ends, so a mismatched pair such as `[due:: 2026-01-01)` does not match. +// line end. The suffix scan requires paired brackets, so a mismatched pair +// such as `[due:: 2026-01-01)` does not match. /** The allowed characters in a single 🆔 / `id::` task ID. */ const TASK_ID = /[a-zA-Z0-9_-]+/ @@ -174,16 +177,56 @@ const TASK_ID_SEQUENCE = new RegExp(`${TASK_ID.source}(?:(?: *, *${TASK_ID.sourc const emojiField = ({ symbols, valuePattern }: { symbols: string; valuePattern: string }): RegExp => new RegExp(valuePattern === "" ? `${symbols}\\uFE0F?$` : `${symbols}\\uFE0F? *${valuePattern}$`) -/** Builds a Dataview inline-field regex: `[key:: value]` or `(key:: value)` - * with matched brackets and an optional trailing comma, anchored to line end - * (see block comment above). */ -const dataviewField = (innerPattern: string): RegExp => { - const squareBracketOpener = String.raw`(?=[^\]]+\])\[` - const parenthesisOpener = String.raw`(?=[^)]+\))\(` +type FieldMatch = Pick +type FieldMatcher = { exec: (line: string) => FieldMatch | null } - return new RegExp( - `(?:${squareBracketOpener}|${parenthesisOpener}) *${innerPattern} *[)\\]](?: *,)?$`, - ) +/** Matches only the last bracketed field, so failed openers never rescan its suffix. */ +const dataviewField = ({ + key, + valuePattern: valueSource, +}: { + key: string + valuePattern: string +}): FieldMatcher => { + const keyPrefix = `${key}::` + const innerPattern = `${keyPrefix} *${valueSource}` + /** Matches the complete field value after its surrounding ASCII spaces are removed. */ + const valuePattern = new RegExp(`^${valueSource}$`) + /** Matches a single field whose bracket pairing the suffix scan already checked. */ + const fieldPattern = new RegExp(`^[[(] *${innerPattern} *[)\\]](?: *,)?$`) + + return { + exec: (line) => { + // The cursors move backward once to the closing delimiter and last opener. + let close = line.length - 1 + + if (line[close] === ",") { + close -= 1 + while (line[close] === " ") close -= 1 + } + const closer = line[close] + + if (closer !== "]" && closer !== ")") return null + let open = close - 1 + + while (open >= 0 && line[open] !== "[" && line[open] !== "(") open -= 1 + if (open < 0 || closer !== (line[open] === "[" ? "]" : ")")) return null + let valueStart = open + 1 + + while (line[valueStart] === " ") valueStart += 1 + if (!line.startsWith(keyPrefix, valueStart)) return null + valueStart += keyPrefix.length + while (line[valueStart] === " ") valueStart += 1 + let valueEnd = close + + while (valueEnd > valueStart && line[valueEnd - 1] === " ") valueEnd -= 1 + if (!valuePattern.test(line.slice(valueStart, valueEnd))) return null + const match = fieldPattern.exec(line.slice(open)) + + if (match) match.index += open + return match + }, + } } /** `YYYY-MM-DD` — the only date format the plugin recognizes on task lines. */ @@ -194,37 +237,37 @@ const DATE_VALUE = "(\\d{4}-\\d{2}-\\d{2})" const DATE_FIELDS: ReadonlyArray<{ key: "done" | "cancelled" | "due" | "scheduled" | "start" | "created" emoji: RegExp - dataview: RegExp + dataview: FieldMatcher }> = [ { key: "done", emoji: emojiField({ symbols: "✅", valuePattern: DATE_VALUE }), - dataview: dataviewField(`completion:: *${DATE_VALUE}`), + dataview: dataviewField({ key: "completion", valuePattern: DATE_VALUE }), }, { key: "cancelled", emoji: emojiField({ symbols: "❌", valuePattern: DATE_VALUE }), - dataview: dataviewField(`cancelled:: *${DATE_VALUE}`), + dataview: dataviewField({ key: "cancelled", valuePattern: DATE_VALUE }), }, { key: "due", emoji: emojiField({ symbols: "(?:📅|📆|🗓)", valuePattern: DATE_VALUE }), - dataview: dataviewField(`due:: *${DATE_VALUE}`), + dataview: dataviewField({ key: "due", valuePattern: DATE_VALUE }), }, { key: "scheduled", emoji: emojiField({ symbols: "(?:⏳|⌛)", valuePattern: DATE_VALUE }), - dataview: dataviewField(`scheduled:: *${DATE_VALUE}`), + dataview: dataviewField({ key: "scheduled", valuePattern: DATE_VALUE }), }, { key: "start", emoji: emojiField({ symbols: "🛫", valuePattern: DATE_VALUE }), - dataview: dataviewField(`start:: *${DATE_VALUE}`), + dataview: dataviewField({ key: "start", valuePattern: DATE_VALUE }), }, { key: "created", emoji: emojiField({ symbols: "➕", valuePattern: DATE_VALUE }), - dataview: dataviewField(`created:: *${DATE_VALUE}`), + dataview: dataviewField({ key: "created", valuePattern: DATE_VALUE }), }, ] @@ -236,31 +279,39 @@ const EMOJI_PRIORITY_RE = emojiField({ /** Dataview priority field, anchored to line end. Captures the level word * (lowercase only, matching the plugin's regex; `highest` before `high` so * the longer word wins). */ -const DATAVIEW_PRIORITY_RE = dataviewField("priority:: *(highest|high|medium|low|lowest)") -/** Recurrence rule text after 🔁 — letters, digits, commas, spaces, `!`. */ -const EMOJI_RECURRENCE_RE = emojiField({ - symbols: "🔁", - valuePattern: "([a-zA-Z0-9, !]+)", +const DATAVIEW_PRIORITY_RE = dataviewField({ + key: "priority", + valuePattern: "(highest|high|medium|low|lowest)", }) -const DATAVIEW_RECURRENCE_RE = dataviewField("repeat:: *([a-zA-Z0-9, !]+)") +/** Recurrence rule text after 🔁 — letters, digits, commas, spaces, `!`. */ +const EMOJI_RECURRENCE_RE = /🔁\uFE0F?([a-zA-Z0-9, !]+)$/ +/** Matches recurrence fields with their leading/trailing spaces inside the value. */ +const DATAVIEW_RECURRENCE_RE = + /(?:\[ *repeat::([a-zA-Z0-9, !]+)\]|\( *repeat::([a-zA-Z0-9, !]+)\))(?: *,)?$/ /** On-completion action word after 🏁 (the plugin accepts delete/keep). */ const EMOJI_ON_COMPLETION_RE = emojiField({ symbols: "🏁", valuePattern: "([a-zA-Z]+)", }) -const DATAVIEW_ON_COMPLETION_RE = dataviewField("onCompletion:: *([a-zA-Z]+)") +const DATAVIEW_ON_COMPLETION_RE = dataviewField({ + key: "onCompletion", + valuePattern: "([a-zA-Z]+)", +}) /** The task's own ID after 🆔. */ const EMOJI_ID_RE = emojiField({ symbols: "🆔", valuePattern: `(${TASK_ID.source})`, }) -const DATAVIEW_ID_RE = dataviewField(`id:: *(${TASK_ID.source})`) +const DATAVIEW_ID_RE = dataviewField({ key: "id", valuePattern: `(${TASK_ID.source})` }) /** Comma-separated IDs this task depends on, after ⛔. */ const EMOJI_DEPENDS_ON_RE = emojiField({ symbols: "⛔", valuePattern: `(${TASK_ID_SEQUENCE.source})`, }) -const DATAVIEW_DEPENDS_ON_RE = dataviewField(`dependsOn:: *(${TASK_ID_SEQUENCE.source})`) +const DATAVIEW_DEPENDS_ON_RE = dataviewField({ + key: "dependsOn", + valuePattern: `(${TASK_ID_SEQUENCE.source})`, +}) /** Emoji signifier → priority level. */ const PRIORITY_BY_EMOJI: Readonly> = { @@ -329,10 +380,25 @@ const PLUGIN_MAX_RUNS = 20 /** The plugin's do/while executes once before checking its limit, allowing 21 passes. */ const MAX_STRIPPING_PASSES = PLUGIN_MAX_RUNS + 1 +const METADATA_MATCHERS = [ + EMOJI_PRIORITY_RE, + DATAVIEW_PRIORITY_RE, + ...DATE_FIELDS.flatMap((field) => [field.emoji, field.dataview]), + EMOJI_RECURRENCE_RE, + DATAVIEW_RECURRENCE_RE, + EMOJI_ON_COMPLETION_RE, + DATAVIEW_ON_COMPLETION_RE, + HASHTAG_FROM_END_RE, + EMOJI_ID_RE, + DATAVIEW_ID_RE, + EMOJI_DEPENDS_ON_RE, + DATAVIEW_DEPENDS_ON_RE, +] + /** Extracts a regex capture group, throwing if absent. Capture groups are * guaranteed by the engine when the regex matches, but noUncheckedIndexedAccess * adds `| undefined` to all indexed access. */ -const capturedGroup = (match: RegExpExecArray, index: number): string => { +const capturedGroup = (match: FieldMatch, index: number): string => { const value = match[index] if (value === undefined) { @@ -384,21 +450,22 @@ const parseTaskMetadata = (taskBody: string): TaskMetadata => { let dependsOn: readonly string[] = [] let trailingTags = "" - const extractField = (regex: RegExp, onMatch: (match: RegExpExecArray) => void): void => { + const extractField = (regex: FieldMatcher, onMatch: (match: FieldMatch) => void): void => { const match = regex.exec(line) if (!match) return onMatch(match) - line = line.replace(regex, "").trim() + line = line.slice(0, match.index).trim() matchedThisPass = true } - const extractDate = (regex: RegExp, key: (typeof DATE_FIELDS)[number]["key"]): void => { + const extractDate = (regex: FieldMatcher, key: (typeof DATE_FIELDS)[number]["key"]): void => { extractField(regex, (match) => { dates[key] = capturedGroup(match, 1) }) } + // Removing a suffix can expose a field whose turn already passed; the next pass reads it. for (let pass = 0; pass < MAX_STRIPPING_PASSES; pass++) { matchedThisPass = false @@ -418,7 +485,8 @@ const parseTaskMetadata = (taskBody: string): TaskMetadata => { recurrence = capturedGroup(match, 1).trim() }) extractField(DATAVIEW_RECURRENCE_RE, (match) => { - recurrence = capturedGroup(match, 1).trim() + const recurrenceValue = match[1] ?? capturedGroup(match, 2) + recurrence = recurrenceValue.trim() }) extractField(EMOJI_ON_COMPLETION_RE, (match) => { @@ -547,6 +615,9 @@ const scanBodyLinesForTasks = ({ // state across the loop (same pattern as classifyLines). let openFence: OpenFence = null let commentOpen = false + // Heading starts are ordered; each heading is visited once during the line scan. + let headingIndex = 0 + let nearestHeading: HeadingInfo | undefined for (let lineIndex = 0; lineIndex < bodyLines.length; lineIndex++) { const lineText = bodyLines[lineIndex] @@ -566,7 +637,11 @@ const scanBodyLinesForTasks = ({ if (commentResult.lineIsComment) continue // Reset indent stack at heading boundaries — sub-tasks can't span headings - const lineStartsHeading = headings.some((heading) => heading.startLine === lineIndex) + while (headings[headingIndex] && (headings[headingIndex]?.startLine ?? Infinity) < lineIndex) { + nearestHeading = headings[headingIndex] + headingIndex += 1 + } + const lineStartsHeading = headings[headingIndex]?.startLine === lineIndex if (lineStartsHeading) { indentStack = [] @@ -598,11 +673,9 @@ const scanBodyLinesForTasks = ({ // The block link sits at the end of the line — strip it before metadata // parsing, exactly as the plugin does. - const bodyWithBlockLink = capturedGroup(taskLineMatch, 2) + const bodyWithBlockLink = capturedGroup(taskLineMatch, 2).replace(LEADING_BODY_SPACES_RE, "") const { body: taskBody, blockId } = splitTrailingBlockLink(bodyWithBlockLink) - const nearestHeading = headings.findLast((heading) => heading.startLine < lineIndex) - const taskIndent = getStructuralIndent(lineText) const ancestors = ancestorsOf(indentStack, taskIndent) const parentEntry = ancestors.at(-1) @@ -801,11 +874,156 @@ const removeLastMetadataMatch = (metadata: string, regex: RegExp): string => { return `${beforeMatch}${afterMatch}`.replace(/ {2,}/g, " ").trim() } -/** Removes EVERY occurrence of a field regex from a metadata tail. - * Clear paths, onCompletion sets, and completion stamps discard every duplicate. */ +type MetadataRemovalGrammar = Readonly<{ + regex: RegExp + symbols: string + key: string + value: "date" | "priority" | "id" | "dependencies" | "recurrence" | "action" +}> + +/** Matches a lowercase priority-word character. */ +const PRIORITY_CHARACTER_RE = /[a-z]/u +/** Matches an on-completion action character. */ +const ACTION_CHARACTER_RE = /[a-zA-Z]/u +/** Matches a recurrence-rule character. */ +const RECURRENCE_CHARACTER_RE = /[a-zA-Z0-9, !]/u + +const METADATA_REMOVAL_GRAMMARS: ReadonlyArray = [ + { regex: DONE_DATE_INLINE_RE, symbols: "✅", key: "completion", value: "date" }, + { regex: CANCELLED_DATE_INLINE_RE, symbols: "❌", key: "cancelled", value: "date" }, + { regex: DUE_DATE_INLINE_RE, symbols: "📅📆🗓", key: "due", value: "date" }, + { regex: SCHEDULED_DATE_INLINE_RE, symbols: "⏳⌛", key: "scheduled", value: "date" }, + { regex: START_DATE_INLINE_RE, symbols: "🛫", key: "start", value: "date" }, + { regex: CREATED_DATE_INLINE_RE, symbols: "➕", key: "created", value: "date" }, + { regex: PRIORITY_INLINE_RE, symbols: "🔺⏫🔼🔽⏬", key: "priority", value: "priority" }, + { regex: TASK_ID_INLINE_RE, symbols: "🆔", key: "id", value: "id" }, + { regex: DEPENDS_ON_INLINE_RE, symbols: "⛔", key: "dependsOn", value: "dependencies" }, + { regex: RECURRENCE_INLINE_RE, symbols: "🔁", key: "repeat", value: "recurrence" }, + { regex: ON_COMPLETION_INLINE_RE, symbols: "🏁", key: "onCompletion", value: "action" }, +] + +/** Removes duplicates from right to left, including fields exposed by an earlier removal. */ const removeAllMetadataMatches = (metadata: string, regex: RegExp): string => { - const stripped = removeLastMetadataMatch(metadata, regex) - return stripped === metadata ? metadata : removeAllMetadataMatches(stripped, regex) + const grammar = METADATA_REMOVAL_GRAMMARS.find((entry) => entry.regex === regex) + + if (!grammar) throw new Error("unknown metadata removal grammar") + const matcher = new RegExp(`^(?:${regex.source})`, regex.flags) + const suffix: string[] = [] + // The stack and its lower bound retain surviving text without copying it after each removal. + let suffixStart = 0 + let removed = false + const read = (offset: number): string => { + const index = suffix.length - 1 - offset + + return index >= suffixStart ? (suffix[index] ?? "") : "" + } + const skipSpace = (offset: number): number => offset + Number(read(offset) === " ") + const scan = (offset: number, characterPattern: RegExp): number => { + // Each successful variable-length scan consumes its characters; failed fields stop at an opener. + let end = offset + + while (end < suffix.length - suffixStart && characterPattern.test(read(end))) end += 1 + return end + } + const valueEnd = (offset: number): number => { + switch (grammar.value) { + case "date": + return offset + 10 + case "priority": + return scan(offset, PRIORITY_CHARACTER_RE) + case "id": + return scan(offset, TASK_ID_WHOLE_RE) + case "action": + return scan(offset, ACTION_CHARACTER_RE) + case "recurrence": + return scan(offset, RECURRENCE_CHARACTER_RE) + case "dependencies": { + let end = scan(offset, TASK_ID_WHOLE_RE) + let comma = skipSpace(end) + + while (read(comma) === ",") { + const nextStart = skipSpace(comma + 1) + const nextEnd = scan(nextStart, TASK_ID_WHOLE_RE) + + if (nextEnd === nextStart) break + end = nextEnd + comma = skipSpace(end) + } + return end + } + } + } + const matchLength = (character: string): number => { + const bracketed = character === "[" || character === "(" + + if (!bracketed && !grammar.symbols.includes(character)) return 0 + let offset = 1 + + if (bracketed) { + offset = skipSpace(offset) + for (const keyCharacter of `${grammar.key}::`) { + if (read(offset) !== keyCharacter) return 0 + offset += 1 + } + } else if (read(offset) === "\uFE0F") { + offset += 1 + } + if (bracketed || grammar.value !== "priority") { + offset = valueEnd(skipSpace(offset)) + } + if (bracketed) { + offset = skipSpace(offset) + if (read(offset) !== "]" && read(offset) !== ")") return 0 + offset += 1 + const comma = skipSpace(offset) + + if (read(comma) === ",") offset = comma + 1 + } + if (offset > suffix.length - suffixStart) return 0 + const candidate = suffix + .slice(suffix.length - offset) + .reverse() + .join("") + const match = matcher.exec(candidate) + + return match?.[0].length ? Array.from(match[0]).length : 0 + } + + // Walk code points backward without copying the full input into character arrays. + for (let index = metadata.length - 1; index >= 0; index -= 1) { + const codeUnit = metadata.charCodeAt(index) + const precedingCodeUnit = metadata.charCodeAt(index - 1) + const hasSurrogatePair = + codeUnit >= 0xdc00 && + codeUnit <= 0xdfff && + precedingCodeUnit >= 0xd800 && + precedingCodeUnit <= 0xdbff + const character = metadata.slice(index - Number(hasSurrogatePair), index + 1) + + index -= Number(hasSurrogatePair) + if (character === " " && read(0) === " ") continue + if (removed && suffix.length === suffixStart && character.trim() === "") continue + const nextCharacter = read(0) + const nextCodeUnit = nextCharacter.charCodeAt(0) + const formsSurrogatePair = + character.length === 1 && + codeUnit >= 0xd800 && + codeUnit <= 0xdbff && + nextCodeUnit >= 0xdc00 && + nextCodeUnit <= 0xdfff + const fieldCharacter = formsSurrogatePair ? character + nextCharacter : character + + // A deletion can join previously unpaired surrogates into a metadata emoji. + if (formsSurrogatePair) suffix.pop() + suffix.push(fieldCharacter) + const length = matchLength(fieldCharacter) + + if (!length) continue + suffix.length -= length + removed = true + while (suffixStart < suffix.length && suffix[suffixStart]?.trim() === "") suffixStart += 1 + } + return removed ? suffix.slice(suffixStart).reverse().join("").trim() : metadata } // Re-export config types so consumers of tasks.ts don't need a @@ -1025,6 +1243,7 @@ const updateTaskLineOnCompletion = ({ config: TaskFormatConfig }): string => { return transformMetadata(taskLine, (metadata) => { + // Parsing keeps the leftmost action; a surviving copy could still make completion delete the task. const metadataWithoutOnCompletion = removeAllMetadataMatches(metadata, ON_COMPLETION_INLINE_RE) if (onCompletion === null) { @@ -1064,24 +1283,74 @@ const updateTaskLineDependsOn = ({ }) } -/** Global twin of FIRST_METADATA_SIGNIFIER_RE for matchAll — kept separate - * so the non-global regex never carries a lastIndex. */ -const METADATA_SIGNIFIER_CANDIDATES_RE = new RegExp(FIRST_METADATA_SIGNIFIER_RE.source, "gu") - -/** Index in a task body (checkbox prefix and block link removed) where the - * metadata tail begins, or -1 when the whole body is description. - * - A signifier opens the tail only when everything after it parses as - * metadata — the parser strips from the right, so an emoji mid-sentence - * ("Prefer 🔼 arrows in docs 📅 2026-09-15") stays in the description. - * - Tags interleaved with fields belong to the tail, not the description. */ +/** Finds the earliest signifier consumed within the plugin's stripping budget. + * Only hashtags can contain another signifier, so each stripped hashtag gets + * one suffix walk starting immediately after its turn in the current pass. */ const findMetadataStart = (taskBody: string): number => { - for (const candidate of taskBody.matchAll(METADATA_SIGNIFIER_CANDIDATES_RE)) { - const tail = parseTaskMetadata(taskBody.slice(candidate.index)) - const tailDescription = tail.description.replace(HASHTAG_RE, "").trim() - - if (tailDescription === "") return candidate.index + const scanSuffix = ({ + text, + offset, + firstPass, + firstMatcher, + hadEarlierMatch, + }: { + text: string + offset: number + firstPass: number + firstMatcher: number + hadEarlierMatch: boolean + }): number => { + // Suffix removal and the earliest consumed signifier are sequential parser state. + let line = text.trimEnd() + let earliestStart = Infinity + + for (let pass = firstPass; pass < MAX_STRIPPING_PASSES; pass++) { + let matchedThisPass = pass === firstPass && hadEarlierMatch + const matcherStart = pass === firstPass ? firstMatcher : 0 + + for ( + let matcherIndex = matcherStart; + matcherIndex < METADATA_MATCHERS.length; + matcherIndex++ + ) { + const matcher = METADATA_MATCHERS[matcherIndex] + const match = matcher?.exec(line) + + if (!match) continue + const matchedText = capturedGroup(match, 0) + + if (matcher === HASHTAG_FROM_END_RE) { + // Dropping this tag's '#' exposes its inner fields. Later tags have already + // spent their passes; replaying them for every inner signifier is unnecessary. + const tagStart = match.index + matchedText.indexOf("#") + 1 + const innerStart = scanSuffix({ + text: line.slice(tagStart), + offset: offset + tagStart, + firstPass: pass, + firstMatcher: matcherIndex + 1, + hadEarlierMatch: matchedThisPass, + }) + earliestStart = Math.min(earliestStart, innerStart) + } else if (FIRST_METADATA_SIGNIFIER_RE.exec(matchedText)?.index === 0) { + // Spaced Dataview openers parse as fields but are not legacy boundary candidates. + earliestStart = Math.min(earliestStart, offset + match.index) + } + matchedThisPass = true + line = line.slice(0, match.index).trimEnd() + } + if (!matchedThisPass) break + } + return earliestStart } - return -1 + const earliestStart = scanSuffix({ + text: taskBody, + offset: 0, + firstPass: 0, + firstMatcher: 0, + hadEarlierMatch: false, + }) + + return Number.isFinite(earliestStart) ? earliestStart : -1 } type TaskLineParts = { @@ -1184,8 +1453,9 @@ const appendMetadataField = ({ fieldText: string }): string => [metadata, fieldText].filter(Boolean).join(" ") -/** Inserts before the first present field in the caller's canonical field order. - * Existing fields retain their order, even when it differs from the canonical order. */ +/** Inserts before the first present field in the caller's canonical field order, + * or appends when none is present. Existing fields keep their order, even when + * it differs from the canonical order. */ const insertFieldAtPosition = ({ metadata, fieldText, @@ -1211,7 +1481,7 @@ const parseTaskLineMetadata = (taskLine: string): TaskMetadata | null => { const taskLineMatch = TASK_LINE_RE.exec(taskLine) if (!taskLineMatch) return null - const bodyWithBlockLink = capturedGroup(taskLineMatch, 2) + const bodyWithBlockLink = capturedGroup(taskLineMatch, 2).replace(LEADING_BODY_SPACES_RE, "") const { body: taskBody } = splitTrailingBlockLink(bodyWithBlockLink) return parseTaskMetadata(taskBody) } @@ -1498,26 +1768,30 @@ const replaceTaskLineDescription = ({ return joinTaskLine({ ...parts, description: newDescription }) } -/** Matches a specific hashtag as a whole token — preceded by start-of-string - * or whitespace, followed by whitespace or end-of-string. */ -const hashtagTokenPattern = (escapedTag: string): RegExp => { - return new RegExp(`(^|\\s)${escapedTag}(?=\\s|$)`) -} +/** Matches the characters allowed after '#' in the plugin's hashtag grammar. */ +const HASHTAG_CHARACTER_RE = /[^ !@#$%^&*(),.?":{}|<>]/u +/** Matches whitespace consumed between successive trailing tags. */ +const TAG_WHITESPACE_RE = /\s/u -/** Extracts trailing hashtags from a string by repeated right-to-left - * stripping — the same loop shape as parseTaskMetadata's tag pass. */ +/** Extracts the trailing hashtag run without rescanning earlier tags. */ const extractTrailingTags = (text: string): readonly string[] => { const tags: string[] = [] - // Each iteration shortens the string from the right; later passes - // depend on the shortened result (sequential parser state). - let remaining = text - let tagMatch = HASHTAG_FROM_END_RE.exec(remaining) - while (tagMatch) { - tags.unshift(tagMatch[0].trim()) - remaining = remaining.slice(0, tagMatch.index).trim() - tagMatch = HASHTAG_FROM_END_RE.exec(remaining) + // Both boundaries move left once across the suffix, preserving the stripping order. + let end = text.length + + while (end > 0) { + let start = end - 1 + + while (start >= 0 && HASHTAG_CHARACTER_RE.test(text.charAt(start))) start -= 1 + const hasTagValue = start < end - 1 + const hasTagBoundary = start === 0 || TAG_WHITESPACE_RE.test(text.charAt(start - 1)) + + if (text[start] !== "#" || !hasTagValue || !hasTagBoundary) break + tags.push(text.slice(start, end).trim()) + end = start + while (end > 0 && TAG_WHITESPACE_RE.test(text.charAt(end - 1))) end -= 1 } - return tags + return tags.toReversed() } /** Strips tags from the metadata tail that byte-match trailing tags in the @@ -1539,18 +1813,42 @@ const deduplicateDescriptionTags = ( return { taskLine, deduplicatedTags: [] } } - // Strip from the metadata only tags that byte-match a trailing - // description tag — non-overlapping tags stay in both positions. - // Each iteration shortens the metadata (sequential stripping state). + // Each stripping pass consumes at most one hashtag, and other fields exclude + // its marker. The metadata therefore has at most MAX_STRIPPING_PASSES markers; + // checking those positions avoids a whole-tail search for every description tag. + const tagPositionsIn = (metadata: string): readonly number[] => { + const positions: number[] = [] + + for (let index = 0; index < metadata.length; index++) { + if ( + metadata[index] === "#" && + (index === 0 || TAG_WHITESPACE_RE.test(metadata.charAt(index - 1))) + ) { + positions.push(index) + } + } + return positions + } + + // Successful removals change subsequent positions and consume a hashtag. let dedupedMetadata = parts.metadata + let tagPositions = tagPositionsIn(dedupedMetadata) const deduplicatedTags: string[] = [] for (const tag of descriptionTrailingTags) { - const tagPattern = hashtagTokenPattern(RegExp.escape(tag)) + const tagStart = tagPositions.find((position) => { + const tagEnd = position + tag.length + const hasTokenEnd = + tagEnd === dedupedMetadata.length || TAG_WHITESPACE_RE.test(dedupedMetadata.charAt(tagEnd)) + return hasTokenEnd && dedupedMetadata.startsWith(tag, position) + }) - if (tagPattern.test(dedupedMetadata)) { - dedupedMetadata = dedupedMetadata.replace(tagPattern, "").trim() - deduplicatedTags.push(tag) - } + if (tagStart === undefined) continue + const removalStart = tagStart === 0 ? 0 : tagStart - 1 + dedupedMetadata = ( + dedupedMetadata.slice(0, removalStart) + dedupedMetadata.slice(tagStart + tag.length) + ).trim() + tagPositions = tagPositionsIn(dedupedMetadata) + deduplicatedTags.push(tag) } if (deduplicatedTags.length === 0) { diff --git a/src/vault-mcp/search/__tests__/chunker.test.ts b/src/vault-mcp/search/__tests__/chunker.test.ts index 0c942cc19..d10f09802 100644 --- a/src/vault-mcp/search/__tests__/chunker.test.ts +++ b/src/vault-mcp/search/__tests__/chunker.test.ts @@ -81,6 +81,14 @@ describe("chunkContent", () => { }) describe("heading-based splitting", () => { + it("chunks heading counts above the engine's call-argument limit", () => { + const bodyContent = "# h\n".repeat(150_000) + + expect(chunkContent({ noteTitle: "Many", bodyContent })).toEqual([ + { index: 0, text: `Many\n\n${Array.from({ length: 449 }, () => "h").join("\n")}` }, + ]) + }) + it("splits a long note into per-section chunks, each carrying its Section line", () => { const section1 = generateLabeledTokens(200, "first") const section2 = generateLabeledTokens(200, "second") diff --git a/src/vault-mcp/search/chunker.ts b/src/vault-mcp/search/chunker.ts index 180f9232e..5525bd910 100644 --- a/src/vault-mcp/search/chunker.ts +++ b/src/vault-mcp/search/chunker.ts @@ -195,7 +195,10 @@ const collectSectionSpans = ( headings: readonly HeadingInfo[], preambleExists: boolean, ): SectionSpan[] => { - const topLevel = Math.min(...headings.map((heading) => heading.level)) + const topLevel = headings.reduce( + (shallowestLevel, heading) => Math.min(shallowestLevel, heading.level), + Infinity, + ) const topLevelHeadingCount = headings.filter((heading) => heading.level === topLevel).length // A lone top-level heading that opens the note (the `# Title` wrapper diff --git a/src/vault-mcp/vault-operations/__tests__/task-mutations-recurrence-boundary.test.ts b/src/vault-mcp/vault-operations/__tests__/task-mutations-recurrence-boundary.test.ts new file mode 100644 index 000000000..67c2c7e93 --- /dev/null +++ b/src/vault-mcp/vault-operations/__tests__/task-mutations-recurrence-boundary.test.ts @@ -0,0 +1,71 @@ +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises" +import { tmpdir } from "node:os" +import { join } from "node:path" +import { describe, expect, it, onTestFinished } from "vitest" +import { taskMutations } from "../task-mutations.js" +import { logger } from "../../../logger.js" +import { DEFAULT_STATUS_REGISTRY } from "../../obsidian-markdown/tasks.js" + +describe("recurrence mutation metadata boundary", () => { + it.each([ + { format: "emoji", malformed: "🔁", newField: "🔁 every week" }, + { format: "dataview", malformed: "[repeat::", newField: "[repeat:: every week]" }, + ] as const)( + "keeps malformed $format recurrence outside the nonempty metadata tail", + async ({ format, malformed, newField }) => { + const vaultPath = await mkdtemp(join(tmpdir(), "probe-recurrence-boundary-")) + onTestFinished(() => rm(vaultPath, { recursive: true, force: true })) + const notePath = join(vaultPath, "Task.md") + for (const size of [500, 1_000, 2_000]) { + const description = `Work ${malformed}${" ".repeat(size)}?]` + const original = `- [ ] ${description} 📅 2026-01-02\n` + await writeFile(notePath, original) + const setResult = await taskMutations.updateTask( + { + vaultPath, + path: "Task.md", + line: 1, + recurrence: "every week", + format, + statusRegistry: DEFAULT_STATUS_REGISTRY, + }, + logger, + ) + + expect(setResult.changes).toEqual(["recurrence: (none) → every week"]) + expect(await readFile(notePath, "utf8")).toBe( + `- [ ] ${description} ${newField} 📅 2026-01-02\n`, + ) + const followUpResult = await taskMutations.updateTask( + { + vaultPath, + path: "Task.md", + line: 1, + recurrence: null, + format, + statusRegistry: DEFAULT_STATUS_REGISTRY, + }, + logger, + ) + + expect(followUpResult.changes).toEqual(["recurrence: every week → (none)"]) + expect(await readFile(notePath, "utf8")).toBe(original) + await writeFile(notePath, original) + const clearResult = await taskMutations.updateTask( + { + vaultPath, + path: "Task.md", + line: 1, + recurrence: null, + format, + statusRegistry: DEFAULT_STATUS_REGISTRY, + }, + logger, + ) + + expect(clearResult.changes).toEqual(["recurrence: (none) → (none)"]) + expect(await readFile(notePath, "utf8")).toBe(original) + } + }, + ) +}) diff --git a/src/vault-mcp/vault-operations/__tests__/task-mutations.test.ts b/src/vault-mcp/vault-operations/__tests__/task-mutations.test.ts index 69004abcc..451a52d07 100644 --- a/src/vault-mcp/vault-operations/__tests__/task-mutations.test.ts +++ b/src/vault-mcp/vault-operations/__tests__/task-mutations.test.ts @@ -1,9 +1,10 @@ -import { describe, it, expect, onTestFinished } from "vitest" +import { describe, it, expect, onTestFinished, vi } from "vitest" import { mkdtemp, rm, writeFile, readFile, mkdir } from "node:fs/promises" import { join } from "node:path" import { tmpdir } from "node:os" import { DateTime } from "luxon" import { taskMutations } from "../task-mutations.js" +import * as lineParsers from "../../obsidian-markdown/lines.js" import { DEFAULT_STATUS_REGISTRY, type StatusClassification, @@ -2985,6 +2986,129 @@ kanban-plugin: board // ── integer position ────────────────────────────────────── + it("inserts at position 2 after counting a card that opens a comment", async () => { + const vault = await createVault() + const board = + "## Active\n- [ ] Commented %%\n%%\n- [ ] First ^first\n- [ ] Second ^second\n" + await writeTestNote(vault, "board.md", board) + + const created = await taskMutations.createTask( + { + statusRegistry: DEFAULT_STATUS_REGISTRY, + vaultPath: vault, + path: "board.md", + description: "Inserted", + blockId: "inserted", + heading: "Active", + position: 2, + }, + logger, + ) + + expect(created).toEqual({ + path: "board.md", + line: 4, + description: "Inserted", + block_id: "inserted", + heading: "Active", + changes: [`created: (none) → ${today()}`], + }) + expect(await readTestNote(vault, "board.md")).toBe( + `## Active\n- [ ] Commented %%\n%%\n- [ ] Inserted ➕ ${today()} ^inserted\n- [ ] First ^first\n- [ ] Second ^second\n`, + ) + }) + + it("reports position 3 when reordering after a card that opens a comment", async () => { + const vault = await createVault() + const board = + "## Active\n- [ ] Commented %%\n%%\n- [ ] First ^first\n- [ ] Second ^second\n" + await writeTestNote(vault, "board.md", board) + + const moved = await taskMutations.updateTask( + { + statusRegistry: DEFAULT_STATUS_REGISTRY, + vaultPath: vault, + path: "board.md", + blockId: "second", + position: 2, + }, + logger, + ) + + expect(moved.changes).toEqual(["position: 3 → 2"]) + expect(await readTestNote(vault, "board.md")).toBe( + "## Active\n- [ ] Commented %%\n%%\n- [ ] Second ^second\n- [ ] First ^first\n", + ) + }) + + it("classifies each line a bounded number of times during large lane insertions and reorders", async () => { + const vault = await createVault() + const cards = Array.from( + { length: 1000 }, + (_, index) => `- [ ] Card ${index} ^card-${index}`, + ) + const commentedCards = Array.from({ length: 1000 }, () => "- [ ] Commented ^hidden").join( + "\n", + ) + const hiddenCards = `\`\`\`\n- [ ] Fenced ^hidden\n\`\`\`\n%%\n${commentedCards}\n%%\n- [>] Non-task\n` + const prefix = `---\nkanban-plugin: board\n---\n## Active\n${hiddenCards}` + const original = `${prefix}${cards.join("\n")}\n` + await writeTestNote(vault, "board.md", original) + const originalAdvanceFence = lineParsers.advanceFence + // Count actual parser work across the whole mutation, including its heading scans. + let fenceVisits = 0 + const fenceSpy = vi + .spyOn(lineParsers, "advanceFence") + .mockImplementation((line, openFence) => { + fenceVisits += 1 + return originalAdvanceFence(line, openFence) + }) + onTestFinished(() => fenceSpy.mockRestore()) + + const created = await taskMutations.createTask( + { + statusRegistry: NON_TASK_REGISTRY, + vaultPath: vault, + path: "board.md", + description: "Inserted", + blockId: "hidden", + heading: "Active", + position: 2, + }, + logger, + ) + const inserted = `- [ ] Inserted ➕ ${today()} ^hidden` + const afterInsertion = `${prefix}${cards.toSpliced(1, 0, inserted).join("\n")}\n` + + expect(created).toEqual({ + path: "board.md", + line: prefix.split("\n").length + 1, + description: "Inserted", + block_id: "hidden", + heading: "Active", + changes: [`created: (none) → ${today()}`], + }) + expect(await readTestNote(vault, "board.md")).toBe(afterInsertion) + expect(fenceVisits).toBeLessThan(20 * cards.length) + fenceVisits = 0 + const moved = await taskMutations.updateTask( + { + statusRegistry: NON_TASK_REGISTRY, + vaultPath: vault, + path: "board.md", + blockId: "card-999", + position: 2, + }, + logger, + ) + const remainingCards = cards.slice(0, -1) + const afterMove = remainingCards.toSpliced(1, 0, cards.at(-1) ?? "", inserted) + + expect(moved.changes).toEqual(["position: 1001 → 2"]) + expect(await readTestNote(vault, "board.md")).toBe(`${prefix}${afterMove.join("\n")}\n`) + expect(fenceVisits).toBeLessThan(30 * cards.length) + }) + it("position=2 inserts the card as the 2nd top-level card", async () => { const vault = await createVault() const note = `---\ntitle: Board\nkanban-plugin: board\n---\n\n## Active\n\n- [ ] First ^first\n- [ ] Second ^second\n- [ ] Third ^third\n` diff --git a/src/vault-mcp/vault-operations/task-mutations.ts b/src/vault-mcp/vault-operations/task-mutations.ts index c86bc1431..9ad3e2e19 100644 --- a/src/vault-mcp/vault-operations/task-mutations.ts +++ b/src/vault-mcp/vault-operations/task-mutations.ts @@ -471,16 +471,14 @@ const isNonTaskCheckbox = ({ * position-counting purposes. */ const isExcludedFromLane = ({ line, - lineIndex, - lines, + excluded, statusRegistry, }: { line: string - lineIndex: number - lines: readonly string[] + excluded: boolean statusRegistry: ReadonlyMap | undefined }): boolean => { - if (isInsideFenceOrComment(lines, lineIndex)) return true + if (excluded) return true if (!statusRegistry) return false @@ -529,6 +527,7 @@ const headingInsertIndexAtPosition = ({ // Cards under child headings belong to those headings, not this lane. const firstChildStart = findFirstChildHeadingStart({ lines, heading }) const walkEnd = firstChildStart ?? sectionEnd + const fenceOrCommentLineIndices = findFenceOrCommentLineIndices(lines) const cardStartIndices: number[] = [] // Variable-length jumps through the section: each card's sub-item block @@ -546,7 +545,13 @@ const headingInsertIndexAtPosition = ({ walkIndex++ continue } - if (isExcludedFromLane({ line, lineIndex: walkIndex, lines, statusRegistry })) { + if ( + isExcludedFromLane({ + line, + excluded: fenceOrCommentLineIndices.has(walkIndex), + statusRegistry, + }) + ) { walkIndex = findTaskBlockEnd(lines, walkIndex) continue } @@ -904,35 +909,36 @@ const rejectNonTaskCheckbox = ({ throw new Error(`checkbox "[${statusChar}]" is a NON_TASK status in the Tasks plugin registry`) } -/** Returns true when the body-line index falls inside a fenced code block - * or a `%% %%` comment block — the same exclusions extractTasks applies. */ -const isInsideFenceOrComment = (bodyLines: readonly string[], lineIndex: number): boolean => { - // Sequential parser state — fence and comment scanners are inherently - // stateful (same pattern as scanBodyLinesForTasks in tasks.ts). +/** Classifies a note once so lane walks and ID checks never rescan its prefix per candidate. */ +const findFenceOrCommentLineIndices = (bodyLines: readonly string[]): ReadonlySet => { + const fenceOrCommentLineIndices = new Set() + // Fence and comment state must flow through every line, including skipped card sub-items. let openFence: OpenFence = null let commentOpen = false - for (let index = 0; index <= lineIndex; index++) { - const lineText = bodyLines[index] - - if (lineText === undefined) return false - + for (const [lineIndex, lineText] of bodyLines.entries()) { if (!commentOpen) { const fenceResult = advanceFence(lineText, openFence) openFence = fenceResult.openFence - if (index === lineIndex) return fenceResult.lineIsCode - if (fenceResult.lineIsCode) continue + if (fenceResult.lineIsCode) { + fenceOrCommentLineIndices.add(lineIndex) + continue + } } - + // A line outside a comment is classified by its fence state before its comment opener. + const wasCommentOpen = commentOpen const commentResult = advanceComment(lineText, commentOpen) commentOpen = commentResult.commentOpen - if (index === lineIndex) return commentResult.lineIsComment + if (wasCommentOpen && commentResult.lineIsComment) fenceOrCommentLineIndices.add(lineIndex) } - return false + return fenceOrCommentLineIndices } +const isInsideFenceOrComment = (bodyLines: readonly string[], lineIndex: number): boolean => + findFenceOrCommentLineIndices(bodyLines).has(lineIndex) + /** No-op when the task already sits under the target heading and no * explicit position is requested. With a position, same-lane reorders * go through the extract-reinsert cycle: @@ -1102,6 +1108,7 @@ const positionOfTaskInLane = ({ const firstChildStart = findFirstChildHeadingStart({ lines, heading }) const walkEnd = firstChildStart ?? sectionEnd + const fenceOrCommentLineIndices = findFenceOrCommentLineIndices(lines) // Each card's sub-items are skipped via findTaskBlockEnd, so the // step size varies per iteration — both counters must be mutable. @@ -1115,7 +1122,13 @@ const positionOfTaskInLane = ({ walkIndex++ continue } - if (isExcludedFromLane({ line, lineIndex: walkIndex, lines, statusRegistry })) { + if ( + isExcludedFromLane({ + line, + excluded: fenceOrCommentLineIndices.has(walkIndex), + statusRegistry, + }) + ) { walkIndex = findTaskBlockEnd(lines, walkIndex) continue } @@ -1187,19 +1200,6 @@ type RecurrenceSpawn = | { kind: "advisory"; advisory: string } | { kind: "none" } -/** taskBefore.status already reflects the registry (extractTasks receives - * it), so the registry lookup is belt-and-suspenders — guards against a - * caller that extracted tasks without the registry. */ -const wasTaskAlreadyDone = ({ - taskBefore, - statusRegistry, -}: { - taskBefore: ParsedTask - statusRegistry: ReadonlyMap -}): boolean => { - return taskBefore.status === "done" || statusRegistry.get(taskBefore.statusChar) === "done" -} - /** Resolves the recurrence spawn for an update, mirroring the Tasks plugin: * spawn only on a transition to done from a not-done status (custom * DONE-typed checkbox chars from the status registry count as done), with @@ -1227,7 +1227,7 @@ const resolveRecurrenceSpawn = ({ }): RecurrenceSpawn => { if (status !== "done") return { kind: "none" } - if (wasTaskAlreadyDone({ taskBefore, statusRegistry: config.statusRegistry })) { + if (taskBefore.status === "done") { return { kind: "none" } } @@ -1375,10 +1375,11 @@ const validateBlockId = ({ // - Every line counts, not only tasks: Obsidian resolves a block id on any line. // - trimEnd, so a hard break's trailing spaces cannot hide an existing id; a // missed duplicate on a task line would make later lookups by it ambiguous. + const fenceOrCommentLineIndices = findFenceOrCommentLineIndices(bodyLines) const existingIndex = bodyLines.findIndex((bodyLine, index) => { if (index === excludeLineIndex) return false if (!bodyLine.trimEnd().endsWith(` ^${blockId}`)) return false - return !isInsideFenceOrComment(bodyLines, index) + return !fenceOrCommentLineIndices.has(index) }) if (existingIndex !== -1) { @@ -2276,7 +2277,7 @@ const updateTask = async (params: UpdateTaskParams, logger: Logger): Promise