Skip to content

Fix Markdown scope and multilingual evidence routing with paired validation - #73

Merged
tenphi merged 12 commits into
mainfrom
codex/language-discourse-reliability
Sep 15, 2026
Merged

tenphi merged 12 commits into
mainfrom
codex/language-discourse-reliability

Conversation

@tenphi

@tenphi tenphi commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Refs #61, #62; remaining semantic reliability work continues in #66.

Ordinary Markdown could lose its qualifying heading, and English/Russian or mixed-language queries could select the wrong evidence view. This PR fixes heading and list-container scope, repairs stale derived projections without rewriting authored bytes, and recognizes bounded report, speculation, discussion and rejected-proposal queries.

For example, a ## Details heading inside a list under ## Hypothesis must not turn the enclosing scenario into facts. List boundaries now use a pinned CommonMark parser dependency. Code blocks, literal comments, unchecked tasks, backward qualifications and frame limits retain their meaning. Routing no longer lets an unrelated assistant-speculation sentence, conditional rejection, machine error message or English document-title token override the requested view.

Production changes remain in kb/prose.ts and memory/intent.ts, plus the parser dependency. Retention, answer generation and answer verification are unchanged from the actual base. Luna remains the runtime model; no stronger fallback or additional model calls are introduced.

Review and current validation

Four independent review/fix rounds followed by a clean final code review cover the reported regressions and nearby cases. The final reviewer ran 463 assertions with no remaining findings. The comparison harness also now rejects failed or incomplete runs, including an empty result index with an otherwise successful completion receipt.

Current production commit 558954d completes all 48/48 offline pairs against actual base ce8fbe4. Complete arm observations are deeply equal to the previous offline run, with no fields excluded: reads, projections, routing, recall evidence, answer inputs/outcomes, factual controls and source preservation. Projection upgrades are separately checked against the recorded runtime version.

Current deterministic measurement Base Reviewed code
Correct inferred views 66/100 100/100
Correct explicit views 100/100 100/100
Answer operations supplied with eligible evidence 153/200 172/200
Incorrect ordinary factual projections 5 0

All 88 retained explicit-view evidence controls remain identical. All four ordinary fixtures upgrade a base-created index correctly, and all 48 pairs preserve source bytes. Additional review regressions have dedicated tests without changing the frozen fixture denominator.

Typecheck/build, lint, knip, formatting, repository safety, 4,121 tests across 156 files, 34 smoke checks, and the installed-release-package smoke pass. The rebuilt CLI/client passes 61 assertions across two isolated service starts. Build-only redeploy succeeds; no pre-existing live service was restarted. A Changeset covers the additional published behavior.

Historical Luna evidence and limits

The earlier paired live experiment belongs to frozen runtime f23c0da, not the reviewed 558954d runtime. No live generation was rerun during these review rounds.

Historical fixed live subset Base Frozen PR runtime
Source-useful Luna answers 43/88 68/88
Fully grounded useful Luna answers 38/88 68/88
Published source-meaning errors 12 8
Published qualification errors 8 1

Independent review graded all 220 main/control outputs from 11 fixtures and two repetitions. There were 25 source-useful gains and zero useful-answer losses, but three candidate source errors replaced baseline nulls. Eight source-meaning errors and one qualification omission remained. Matching requests shared responses across arms; different requests sampled independently. Reconstructed retained fixtures lack the original complete provenance metadata. This is bounded downstream evidence, not a new retention trial or proof of general accuracy/noninferiority.

The 233/320 full Luna result and failed quality gate remain unchanged. Its corpus, fresh retention and denominator differ from the paired experiment. #61/#62 remain bounded implementation closures; #66 retains unresolved semantic requirements. These results do not establish a cheap-model intelligence ceiling.

The current review report, complete results, source hashes and independent audit are linked below. Earlier evaluation artifacts remain byte-identical.

Current review report · Independent final audit · Exact offline comparison · Historical paired Luna report · Unchanged full Luna evaluation

@tenphi tenphi changed the title Fix Markdown scope and mixed-language evidence selection Preserve Markdown scope and improve multilingual evidence selection Sep 15, 2026
@tenphi tenphi changed the title Preserve Markdown scope and improve multilingual evidence selection Fix Markdown scope and multilingual evidence routing with paired validation Sep 15, 2026
@tenphi
tenphi merged commit 8b2e7b4 into main Sep 15, 2026
5 checks passed
@github-actions github-actions Bot mentioned this pull request Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant