Repository navigation
P0: independent holdout for semantic corroboration quality #3
Description
Goal
Establish a blind, independently labeled corroboration benchmark before treating synthetic development scores as quality estimates.
Acceptance criteria
- Draw a diverse, timestamped, source-linked holdout of at least 150 candidate pairs from distinct GitHub repositories and Hacker News discussions.
- Include hard paraphrases, same-technology/different-failure negatives, repeated posts from one account, and true cross-community comparisons.
- Separate
same_problem,different_problem, anduncertain; track annotator agreement and unresolved cases. - Keep the holdout separate from prompt tuning/training data and disclose selection bias.
- Measure precision of merges, false-merge severity, recall, and coverage, not just F1 on a tiny synthetic benchmark.
- Avoid copying private data or claiming distinct accounts identify distinct people.
Human review required
The evaluation labels should be independently checked by a human, not generated and judged solely by the same model used in the pipeline.
Activity
drop-agent-bot commentedon Oct 12, 2026
Progress — 12 Oct 2026
Next: mine hard paraphrases and more cross-community true recurrence candidates, freeze a more balanced holdout, and obtain independent human reviews and adjudication before closing this issue.
drop-agent-bot commentedon Oct 12, 2026
P0 progress: holdout readiness check (PR #9)
npm run eval:quality-gate) with source/pair integrity, duplicate pair and concentration checks; 190 local tests pass, including PostgreSQL.drop-agent-bot commentedon Oct 12, 2026
Follow-up: quality gate and hard-pair discovery
needs_more_candidates: 2 high-similarity pairs, 53 medium-similarity pairs, 33 cross-platform; no independent human labels or measurable precision/recall.drop-agent-bot commentedon Oct 12, 2026
Further P0 implementation — 12 Oct 2026
Unresolved
Only 2 high-similarity pairs currently survive the holdout sampler. No independent human labels, accuracy figures or buyer claims can be established yet. Continue sourcing hard paraphrases and arrange genuinely independent human annotation/adjudication. Keep this issue open.
guillaume-flambard commentedon Oct 12, 2026
P0 update: PRs #12 and #13 merged with green Node 22/24 CI. 203 tests pass locally. Recurrence searches now use short technical queries and classify GitHub 403/429 quota failures explicitly instead of treating them as no matches. Holdout quality preflight reports 150 candidate pairs, 2 high-similarity candidates, and 33 cross-platform pairs relying on only 3 distinct Hacker News observations. Still needs independent source diversity and two independent human annotation sets; accuracy is not validated. Keep issue open.
guillaume-flambard commentedon Oct 12, 2026
P0 progress: PR #14 merged with Node.js 22/24 CI green. HN comment-source provenance is now stored with original thread_root_id; corroboration, evidence graph, dossiers and holdout exclude repeated replies from the same discussion as independent threads. Local PostgreSQL-inclusive test suite: 214 passed. Public smoke discovery for coding agents found 23 candidate observations across 19 root discussions (7 HN comments), all sources succeeded. These are UNVERIFIED observations, not independently corroborated problems. Next: source-grounded extraction of candidate comments and rebuild holdout; independent human labels remain outstanding.
guillaume-flambard commentedon Oct 12, 2026
P0 update: PR #15 merged with green Node 22/24 CI, bringing original HN story roots into private holdout audit. Local PostgreSQL-inclusive suite: 217 tests passed. Real 150-pair preflight still identifies only 2 high-similarity candidates, and the 33 GitHub↔HN pairs derive from only 3 HN original threads; benchmark remains
needs_more_candidates, not validated. Two bounded public searches fordatabase migration/coding agentsnow yielded 27 lexically selected HN comments with original story IDs and stored source URLs; these are unverified leads. A bounded four-comment Nemotron evidence review is in progress. Independent human annotation/adjudication remains outstanding; keep issue open.guillaume-flambard commentedon Oct 12, 2026
P0 follow-up: PR #15 is merged with CI green. The private holdout audit now counts original discussion threads, not just comment observation IDs. Two bounded searches (
database migration,coding agents) added public source leads with discussion roots; 4 HN comments were sent to Nemotron extraction/verification (3 model-verifiedfound, 1none, 0 technical errors). Decisions and embeddings were persisted; now 42 current embeddings. Recomputed 150-pair preflight stillneeds_more_candidates: 2 high-similarity candidates, 6 HN observations but only 5 distinct HN original threads in cross-platform pairings (previously 3). No independent human labels exist; do not claim validated model accuracy or market demand. Continue collecting varied original threads and hard true paraphrases.guillaume-flambard commentedon Oct 12, 2026
P0 update: PR #16 merged after green Node 22/24 CI. Targeted discovery now carries source discussion roots into LLM candidate selection and prioritizes distinct original threads, while preserving the source mix. PostgreSQL-inclusive suite: 219 tests pass. A dry local 8-observation selection represented 8 distinct discussions. Benchmark is still
needs_more_candidates: 150 candidate pairs, 2 high-similarity cases, 5 distinct HN original threads among cross-platform pairs. Small three-thread evidence verification is running; no claim of human-validated labels or market demand.guillaume-flambard commentedon Oct 12, 2026
P0 follow-up after merged PR #16 (CI Node 22/24 green): a bounded 3-comment review from distinct HN original threads produced 2 model-verified
found, 1rejected, 0 technical errors. Decisions archived (run 1d745f1c) and 2 local embeddings added. Current DB: 208 observations and 44 embeddings. Holdout remainsneeds_more_candidates: 150 unlabeled pairs, 2 high-similarity candidates, and 7 distinct HN original threads represented across cross-platform pairs (up from 5); 8 distinct HN observation IDs. Blockers: insufficient difficult positives and insufficient original thread diversity. Human labels/adjudication still absent. No accuracy or market-demand claim.guillaume-flambard commentedon Oct 12, 2026
P0 update: PR #17 merged (both Node 22/24 CI green): global corroboration review now prioritizes new pairs from different original discussions; bounded live review of 3 high-similarity candidates resulted in 3
different_problemverdicts and 0 service errors, highlighting why embeddings are candidate retrieval only. PR #18 also merged with green CI: reconstruct globally confirmed groups from historical, policy-matched, source-current stored edges, not merely pairs selected this run. A real zero-review PostgreSQL smoke test restored the historical Turbopack same-problem pair across 2 discussions, with no LLM requests. 222 PostgreSQL-inclusive tests passed. Benchmark still lacks adequate hard paraphrases and independent human annotations; issue remains open.guillaume-flambard commentedon Oct 12, 2026
P0 update: PR #17 merged after green Node 22/24 CI, prioritizing novel pair reviews from distinct originating discussions (3 real high-similarity pairs were independently model-judged
different_problem, 0 errors). PR #18 merged with green CI, preserving previously confirmed corroboration groups across runs while invalidating edges whose source snapshot changed. A real 0-new-review PostgreSQL run retained the verified two-issue Turbopack group. PR #19 now auto-merge-pending CI adds a development-only local embedding diagnostic; eight synthetic cases show overlap between same-problem similarities (min 0.429) and different-problem similarities (max 0.583), proving a raw threshold cannot perfectly separate this toy sample. 226 local tests pass. This is NOT independent model accuracy; hard source-backed paraphrases and two human annotation sets remain outstanding. Keep issue open.guillaume-flambard commentedon Oct 12, 2026
P0 progress: PR #20 has merged with green Node 22/24 CI. A local, development-only comparison of all-minilm (384D) and nomic-embed-text (768D) ranked 11/15 positive-vs-negative SYNTHETIC comparisons correctly for each (dev-only AUC ≈0.733). No production vector/model change or claim of independent accuracy. PR #21 is now awaiting auto-merge on CI: a local-only alternate-embedding miner that prioritizes previously unjudged pairs from distinct discussion roots, excludes same-thread and current-reviewed pairs, limits source repetition, and does not infer human identity across platforms. Local suite: 236 tests passed with PostgreSQL. Real mining from 43 source-current problems identified 12 unlabeled leads from 868 candidate comparisons; many appear to be tricky different-problem neighbors and none have been independently annotated. P0 remains open until source-diverse hard paraphrases and human adjudication are available.
guillaume-flambard commentedon Oct 12, 2026
2026-10-13 P0 technical completion: PR #22 opened with auto-merge (Node 22/24 CI required). Added source-snapshot-pinned alternative pair review with canonical Nemotron same-problem-v2 model verdicts, revision recheck, prior-verdict caching and explicit error states. Two bounded live six-pair batches completed without model errors: 12 new decisions (1 model
same_problem, 11different_problem). Current corpus: 2 model-confirmed same-problem edges, 18 different-problem edges. Local PostgreSQL-inclusive tests: 244 pass. A private 150-pair offline annotation pilot and SHA256 manifest are prepared on Mac at ~/.local/share/memolabs/pie/p0-annotation-pilot-20261013. Quality gate FAILS (needs_more_candidates): 0 high-similarity candidates and 7 distinct HN original discussions in cross-platform pairs versus target 8. Keep issue #3 OPEN because independent human annotators and balanced positive-case coverage are still missing. No independent precision/recall or market validation claims. The technical delivery and blocker handoff are recorded in docs/P0_HANDOFF_2026-10-13.md.guillaume-flambard commentedon Oct 12, 2026
Implementation update: PR #23 merged with Node 22/24 CI green. Human annotation disagreement/adjudication workflow is now implemented, with source/reviewer SHA-256 binding, mandatory rationales and no automatic human labels. Additional P1/P2 baselines shipped in PRs #24–#26 and #29, with 263 PostgreSQL-inclusive tests passing. The current private 150-pair pilot remains NOT READY (
needs_more_candidates); two independent human reviewer submissions, adjudication and hard positive real-source coverage are still required. Keep P0 open until externally verified.