guillaume-flambard · Oct 12, 2026 · Oct 12, 2026
@@ -397,3 +397,17 @@ PIE_ALTERNATE_EMBED_MODEL=nomic-embed-text \
 ```
 
 This is a **development candidate-mining** tool, not a corroboration verdict. It loads only latest source-current, model-screened problems, computes alternative embeddings **in memory only** using localhost Ollama, and ranks previously unreviewed pairs from different original conversations. It excludes current-snapshot pairs already semantically decided, caps each source's exposure and reports source URLs for manual inspection. It neither writes vectors to production pgvector nor sends any candidate to a semantic or paid model. The optional JSON report is private, refuses overwrites and never exposes account hashes. Similarity scores are not comparable between embedding models without calibration. Mined pairs must **not** be inserted into a frozen independent holdout without a fresh sampling protocol and leakage review.
+
+### P0: audited semantic review of alternative-embedding leads
+
+Local retrieval from `research:alternate-pairs` produces **unreviewed** source pairs. To verify a bounded subset, first freeze the source report privately and run:
+
+```bash
+PIE_ALTERNATE_REVIEW_LIMIT=6 \
+PIE_ALTERNATE_REVIEW_REPORT_PATH=/tmp/pie-reviewed-leads.json \
+  npm run research:review-alternate -- /tmp/pie-alternate-pairs.json
+```
+
+The reviewer uses the configured, approved **free OpenRouter Nemotron** model and the current canonical semantic policy, regardless of which *local* embedding model retrieved the leads. Every source is looked up against its latest source-content hash and evidence-grounded extraction, and rechecked after the remote reviewer responds. Stale or same-discussion pairs are rejected. Existing verdicts are reused, while new same/different/uncertain decisions are persisted in `corroboration_edges` with source run IDs and policy version. Failures remain explicit errors, not semantic negatives. The private report is created without overwriting a previous report. This does **not** annotate the independent human holdout or assert independent people, precision, buyer interest or market demand.
+
+**Snapshot binding:** newly mined leads carry the exact source extraction `runId` for each endpoint. The semantic reviewer rejects legacy reports without these run IDs, as well as proposals whose source was re-extracted since mining. Re-run `research:alternate-pairs` when upgrading the workflow or when sources change; never treat a stale similarity score as evidence.
@@ -44,3 +44,9 @@ The source corpus consists of public posts and can contain incidental personal i
 ### Discussion independence after the HN comment connector
 
 Public HN comments retain `thread_root_id` pointing to the parent story. A pair of comments within the same story is not a cross-discussion independent source pair even if two pseudonymous accounts posted it. The holdout query excludes these same-root pairs. Human annotators should still distinguish *same underlying technical problem* from *different problem*: the annotation label itself does not assert distinct human identities. HN search queries that retrieve comments are filtered by relevant words before LLM screening; consequently the sampled corpus is **not** a representative random sample of online discussion.
+
+### Technical P0 release handoff
+
+The automated evidence collection, provenance, alternative retrieval, source-current model corroboration, quality preflight and offline reviewer UI can be considered implementation-complete once CI is green. The separate P0 **independent-validation acceptance criterion** cannot pass without two real human reviewers and an adjudicated, frozen, appropriately challenging blind holdout.
+
+An annotator uses `npm run eval:annotation-ui -- /absolute/private/holdout-blind.json /absolute/private/reviewer-a.html` and repeats for a separate reviewer. Both must export their own private results. Validate with `npm run eval:review-holdout -- <blind> <reviewer-a> <reviewer-b> [predictions]`, keeping source snapshots and review provenance. Do not use the model's own predictions to fill human annotation templates, and do not close issue #3 until the independent evaluation is actually completed.
@@ -0,0 +1,33 @@
+# P0 — Technical completion and independent-validation handoff
+
+Date: 2026-10-13 · Memo Labs Dev · Problem Intelligence Engine
+
+## Technical workflow implemented
+
+- Public-source discovery with source hashes and original-thread lineage; source-root-aware LLM selection.
+- Evidence-grounded extraction, independent model screening and model-run persistence.
+- Local 384-dimensional production pgvector (`all-minilm`) and separate in-memory alternative retrieval (`nomic-embed-text`).
+- Audited semantic reviews of alternate leads with pinned source extraction IDs, policy-scoped cached decisions, a second source freshness check and explicit model/service error states.
+- Complete-link problem grouping with valid historical corroboration edges retained across invocations.
+- Deterministic, source-leakage-aware, blinded human annotation queue; private offline two-reviewer UI; rubric and disagreement/coverage evaluator.
+- Main protected by required Node 22/24 + PostgreSQL CI; contributor bot may auto-merge after green checks.
+
+## Evidence and remaining acceptance criteria
+
+As of this handoff, two bounded batches of six alternate-model leads produced **12 new semantic reviewer decisions**: **1 `same_problem`, 11 `different_problem`, 0 errors**. The total database contains **2 model-confirmed same-problem edges** and **18 different-problem edges**. These are **model decisions**, not independently labeled ground truth.
+
+The current local benchmark pilot has **150 unlabeled pairs** and **fails the pre-annotation quality gate**:
+
+- 0 pairs above the configured high-similarity cutoff of 0.60 in the current `all-minilm` development pool.
+- 7 distinct Hacker News original discussions in cross-platform comparisons, below the gate's minimum of 8.
+- Source selection and earlier automatic judgments can introduce sampling bias; these are not population-level metrics.
+
+Therefore the **technical implementation is ready for independent evaluation**, but **P0 independent validation is NOT finished** and GitHub issue #3 must remain open. Do not claim recall, semantic precision or market fit from the development corpus or LLM-generated verdicts.
+
+## Private review pilot on maintainer Mac
+
+Private directory: `~/.local/share/memolabs/pie/p0-annotation-pilot-20261013/` (not committed to GitHub). It contains frozen source queue, sampling audit, SHA-256 manifest, blank independent reviewer templates, browser-based reviewer A/B HTML pages and a machine-readable quality-gate result.
+
+The files are a **pilot**, not a passing holdout. Keep the audit separate from annotators; they should only see `pairs-blind.json` or the generated reviewer HTML. Collect two independent human judgments of source-grounded candidate pairs and independently authenticate review independence outside the tool. Reviewers must not see each other's answers or existing model verdicts. Before a publishable benchmark, mine more challenging cross-thread, cross-community positive candidates and freeze a *new*, passing holdout with no prior tuning/review leakage. Adjudicate disagreements and publish test-set composition, coverage and uncertainty together with any metrics. The pipeline must not fill human review templates automatically.
+
+For a completed, valid holdout use `npm run eval:review-holdout -- <blind.json> <reviewer-a.json> <reviewer-b.json> [predictions.json]` and retain original submissions and adjudication evidence. Never close P0 #3 until those two independent human annotation sets and a genuinely qualifying holdout exist.
@@ -30,7 +30,8 @@
     "eval:annotation-ui": "node packages/agents/annotation-ui.mjs",
     "eval:embedding-retrieval": "node packages/agents/evaluate-embeddings.mjs",
     "eval:compare-embeddings": "node packages/agents/compare-embeddings.mjs",
-    "research:alternate-pairs": "node packages/agents/mine-alternate-pairs.mjs"
+    "research:alternate-pairs": "node packages/agents/mine-alternate-pairs.mjs",
+    "research:review-alternate": "node --env-file-if-exists=.env packages/agents/review-alternate-pairs.mjs"
   },
   "dependencies": {
     "pg": "^8.23.1"

@@ -56,8 +56,8 @@ export function rankNovelCrossDiscussionPairs(rows,embeddings,priorEdges=[],{lim
   exposure.set(row.rightId,(exposure.get(row.rightId)||0)+1);
   const distinctAccounts=row.left.source===row.right.source&&row.left.reporter_key&&row.right.reporter_key
    ?row.left.reporter_key!==row.right.reporter_key:null;
-  result.push({left:{id:row.leftId,url:row.left.url,source:row.left.source,statement:row.left.statement},
-   right:{id:row.rightId,url:row.right.url,source:row.right.source,statement:row.right.statement},
+  result.push({left:{id:row.leftId,runId:row.left.run_id,url:row.left.url,source:row.left.source,statement:row.left.statement},
+   right:{id:row.rightId,runId:row.right.run_id,url:row.right.url,source:row.right.source,statement:row.right.statement},
    similarity:Number(row.similarity.toFixed(4)),platformPair:row.platformPair,
    distinctSourceAccounts:distinctAccounts,independentlyVerifiedHumans:null,
    semanticEquivalence:'unreviewed',commercialDemand:'unknown'});

@@ -0,0 +1,106 @@
+import {readFile,writeFile} from 'node:fs/promises';
+import {createPool} from '../storage/postgres.mjs';
+import {verifyProblemPair,SEMANTIC_REVIEW_POLICY_VERSION} from './corroborate.mjs';
+import {EMBEDDING_MODEL} from './embed.mjs';
+
+/** All source revisions must still be current at review time. */
+export const ALTERNATE_REVIEW_CURRENT_SQL=`
+WITH latest AS (
+ SELECT d.run_id,d.observation_id,d.status,d.statement,d.excerpt,d.source_content_hash,
+ row_number() OVER(PARTITION BY d.observation_id ORDER BY r.started_at DESC,r.id DESC) rn
+ FROM model_decisions d JOIN model_runs r ON r.id=d.run_id
+)
+SELECT d.run_id,d.observation_id,d.status,d.statement,d.excerpt,
+ o.thread_root_id,o.content_hash,o.url,o.source
+FROM latest d JOIN observations o ON o.id=d.observation_id
+WHERE d.rn=1 AND d.status='found' AND d.source_content_hash=o.content_hash
+AND d.observation_id=ANY($1::text[])`;
+const priorSQL=`SELECT verdict,rationale FROM corroboration_edges
+WHERE left_run=$1 AND left_observation=$2 AND right_run=$3 AND right_observation=$4 AND model=$5 LIMIT 1`;
+const saveSQL=`INSERT INTO corroboration_edges
+(left_run,left_observation,right_run,right_observation,model,verdict,rationale)
+VALUES ($1,$2,$3,$4,$5,$6,$7)
+ON CONFLICT(left_run,left_observation,right_run,right_observation,model)
+DO UPDATE SET verdict=EXCLUDED.verdict,rationale=EXCLUDED.rationale,compared_at=NOW()
+WHERE EXCLUDED.verdict<>'error'`;
+export function validateAlternateMiningQueue(queue,{limit=12}={}){
+ if(queue?.kind!=='unreviewed_alternate_embedding_candidates'||!Array.isArray(queue.pairs)||
+  !Number.isInteger(limit)||limit<1||limit>24)throw Error('Invalid alternate candidate report or review budget');
+ const seen=new Set();
+ const pairs=[];
+ for(const item of queue.pairs.slice(0,limit)){
+  const a=item?.left?.id,b=item?.right?.id;
+  if(typeof a!=='string'||typeof b!=='string'||!a||!b||a===b||
+   typeof item.left.runId!=='string'||!item.left.runId||typeof item.right.runId!=='string'||!item.right.runId)
+   throw Error('Invalid mined source pair or missing source snapshot; regenerate the mined report');
+  const sorted=[a,b].sort();const key=sorted.join('|');
+  if(seen.has(key))throw Error('Repeated mined source pair');
+  seen.add(key);pairs.push(sorted);
+ }
+ return pairs;
+}
+export async function reviewAlternatePairs(pool,queue,{limit=12,reviewerModel='nvidia/nemotron-3.5-lightning:free',
+ judge=verifyProblemPair}={}){
+ if(!reviewerModel.endsWith(':free'))throw Error('Review requires the approved free semantic model');
+ const pairs=validateAlternateMiningQueue(queue,{limit});
+ const ids=[...new Set(pairs.flat())];
+ const {rows}=ids.length?await pool.query(ALTERNATE_REVIEW_CURRENT_SQL,[ids]):{rows:[]};
+ const current=new Map(rows.map(r=>[r.observation_id,r]));
+ const modelKey=EMBEDDING_MODEL+'+'+reviewerModel+'+'+SEMANTIC_REVIEW_POLICY_VERSION;
+ const results=[];
+ for(const [leftId,rightId] of pairs){
+  const a=current.get(leftId),b=current.get(rightId);
+  const submitted=queue.pairs.find(item=>[item.left.id,item.right.id].sort().join('|')===leftId+'|'+rightId);
+  const expected=new Map([[submitted.left.id,submitted.left.runId],[submitted.right.id,submitted.right.runId]]);
+  if(!a||!b||a.run_id!==expected.get(leftId)||b.run_id!==expected.get(rightId)){
+   results.push({left:leftId,right:rightId,status:'stale_or_unverified'});continue;
+  }
+  if((a.thread_root_id||a.observation_id)===(b.thread_root_id||b.observation_id)){
+   results.push({left:leftId,right:rightId,status:'same_original_discussion'});continue;
+  }
+  const params=[a.run_id,a.observation_id,b.run_id,b.observation_id,modelKey];
+  const old=await pool.query(priorSQL,params);
+  if(old.rows[0]&&old.rows[0].verdict!=='error'){
+   results.push({left:leftId,right:rightId,status:'cached',verdict:old.rows[0].verdict});continue;
+  }
+  let verdict;
+  try{
+   verdict=await judge({left_id:leftId,right_id:rightId,
+    left_statement:a.statement,left_excerpt:a.excerpt,
+    right_statement:b.statement,right_excerpt:b.excerpt},
+    {env:{...process.env,PIE_LLM_PROVIDER:'openrouter',PIE_LLM_MODEL:reviewerModel}});
+   if(!['same_problem','different_problem','uncertain'].includes(verdict?.verdict)||
+    typeof verdict.rationale!=='string'||!verdict.rationale.trim())throw Error('Invalid semantic reviewer verdict');
+  }catch(error){results.push({left:leftId,right:rightId,status:'error',error:String(error?.message||error).slice(0,180)});continue;}
+  // Confirm the source snapshot has not changed during the remote LLM call.
+  const recheck=await pool.query(ALTERNATE_REVIEW_CURRENT_SQL,[[leftId,rightId]]);
+  const now=new Map(recheck.rows.map(x=>[x.observation_id,x.run_id]));
+  if(now.get(leftId)!==a.run_id||now.get(rightId)!==b.run_id){
+   results.push({left:leftId,right:rightId,status:'stale_during_review'});continue;
+  }
+  await pool.query(saveSQL,[...params,verdict.verdict,verdict.rationale.slice(0,240)]);
+  results.push({left:leftId,right:rightId,status:'reviewed',verdict:verdict.verdict});
+ }
+ return {kind:'audited_alternate_semantic_review',retrievalModel:queue.model,
+  model:reviewerModel,policy:SEMANTIC_REVIEW_POLICY_VERSION,
+  independentHumanLabels:false,marketDemand:'unknown',
+  reviewed:results.filter(r=>r.status==='reviewed').length,
+  cached:results.filter(r=>r.status==='cached').length,
+  errors:results.filter(r=>r.status==='error').length,
+  sameProblem:results.filter(r=>r.verdict==='same_problem').length,
+  differentProblem:results.filter(r=>r.verdict==='different_problem').length,
+  uncertain:results.filter(r=>r.verdict==='uncertain').length,
+  results};
+}
+if(process.argv[1]?.endsWith('/packages/agents/review-alternate-pairs.mjs')){
+ const source=process.argv[2];
+ if(!source)throw Error('Usage: npm run research:review-alternate -- /private/mined-pairs.json');
+ const queue=JSON.parse(await readFile(source,'utf8'));
+ const pool=createPool();try{
+  const result=await reviewAlternatePairs(pool,queue,{limit:Number(process.env.PIE_ALTERNATE_REVIEW_LIMIT||6)});
+  const destination=process.env.PIE_ALTERNATE_REVIEW_REPORT_PATH;
+  if(destination)await writeFile(destination,JSON.stringify(result,null,2)+'\n',{mode:0o600,flag:'wx'});
+  console.log(JSON.stringify(result,null,2));
+  if(result.errors)process.exitCode=1;
+ }finally{await pool.end();}
+}
@@ -12,6 +12,7 @@ test('rejects same-story comments, previously reviewed current snapshots, and ca
  assert.ok(result.sameThreadSkipped>=1);
  assert.equal(result.alreadyReviewedSkipped,1);
  assert.equal(result.selected.length,2);
+ assert.equal(result.selected[0].left.runId,'run-'+result.selected[0].left.id);
  assert.ok(!result.selected.some(x=>x.left.id==='a'&&x.right.id==='b'));
  assert.ok(result.selected.every(x=>x.semanticEquivalence==='unreviewed'));
  assert.ok(result.selected.every(x=>x.independentlyVerifiedHumans===null));