diff --git a/README.md b/README.md index 7b90a21..1f149b8 100644 --- a/README.md +++ b/README.md @@ -397,3 +397,17 @@ PIE_ALTERNATE_EMBED_MODEL=nomic-embed-text \ ``` This is a **development candidate-mining** tool, not a corroboration verdict. It loads only latest source-current, model-screened problems, computes alternative embeddings **in memory only** using localhost Ollama, and ranks previously unreviewed pairs from different original conversations. It excludes current-snapshot pairs already semantically decided, caps each source's exposure and reports source URLs for manual inspection. It neither writes vectors to production pgvector nor sends any candidate to a semantic or paid model. The optional JSON report is private, refuses overwrites and never exposes account hashes. Similarity scores are not comparable between embedding models without calibration. Mined pairs must **not** be inserted into a frozen independent holdout without a fresh sampling protocol and leakage review. + +### P0: audited semantic review of alternative-embedding leads + +Local retrieval from `research:alternate-pairs` produces **unreviewed** source pairs. To verify a bounded subset, first freeze the source report privately and run: + +```bash +PIE_ALTERNATE_REVIEW_LIMIT=6 \ +PIE_ALTERNATE_REVIEW_REPORT_PATH=/tmp/pie-reviewed-leads.json \ + npm run research:review-alternate -- /tmp/pie-alternate-pairs.json +``` + +The reviewer uses the configured, approved **free OpenRouter Nemotron** model and the current canonical semantic policy, regardless of which *local* embedding model retrieved the leads. Every source is looked up against its latest source-content hash and evidence-grounded extraction, and rechecked after the remote reviewer responds. Stale or same-discussion pairs are rejected. Existing verdicts are reused, while new same/different/uncertain decisions are persisted in `corroboration_edges` with source run IDs and policy version. Failures remain explicit errors, not semantic negatives. The private report is created without overwriting a previous report. This does **not** annotate the independent human holdout or assert independent people, precision, buyer interest or market demand. + +**Snapshot binding:** newly mined leads carry the exact source extraction `runId` for each endpoint. The semantic reviewer rejects legacy reports without these run IDs, as well as proposals whose source was re-extracted since mining. Re-run `research:alternate-pairs` when upgrading the workflow or when sources change; never treat a stale similarity score as evidence. diff --git a/docs/HOLDOUT_ANNOTATION_PROTOCOL.md b/docs/HOLDOUT_ANNOTATION_PROTOCOL.md index ce2f067..e17aa4e 100644 --- a/docs/HOLDOUT_ANNOTATION_PROTOCOL.md +++ b/docs/HOLDOUT_ANNOTATION_PROTOCOL.md @@ -44,3 +44,9 @@ The source corpus consists of public posts and can contain incidental personal i ### Discussion independence after the HN comment connector Public HN comments retain `thread_root_id` pointing to the parent story. A pair of comments within the same story is not a cross-discussion independent source pair even if two pseudonymous accounts posted it. The holdout query excludes these same-root pairs. Human annotators should still distinguish *same underlying technical problem* from *different problem*: the annotation label itself does not assert distinct human identities. HN search queries that retrieve comments are filtered by relevant words before LLM screening; consequently the sampled corpus is **not** a representative random sample of online discussion. + +### Technical P0 release handoff + +The automated evidence collection, provenance, alternative retrieval, source-current model corroboration, quality preflight and offline reviewer UI can be considered implementation-complete once CI is green. The separate P0 **independent-validation acceptance criterion** cannot pass without two real human reviewers and an adjudicated, frozen, appropriately challenging blind holdout. + +An annotator uses `npm run eval:annotation-ui -- /absolute/private/holdout-blind.json /absolute/private/reviewer-a.html` and repeats for a separate reviewer. Both must export their own private results. Validate with `npm run eval:review-holdout -- [predictions]`, keeping source snapshots and review provenance. Do not use the model's own predictions to fill human annotation templates, and do not close issue #3 until the independent evaluation is actually completed. diff --git a/docs/P0_HANDOFF_2026-10-13.md b/docs/P0_HANDOFF_2026-10-13.md new file mode 100644 index 0000000..6187f7a --- /dev/null +++ b/docs/P0_HANDOFF_2026-10-13.md @@ -0,0 +1,33 @@ +# P0 — Technical completion and independent-validation handoff + +Date: 2026-10-13 · Memo Labs Dev · Problem Intelligence Engine + +## Technical workflow implemented + +- Public-source discovery with source hashes and original-thread lineage; source-root-aware LLM selection. +- Evidence-grounded extraction, independent model screening and model-run persistence. +- Local 384-dimensional production pgvector (`all-minilm`) and separate in-memory alternative retrieval (`nomic-embed-text`). +- Audited semantic reviews of alternate leads with pinned source extraction IDs, policy-scoped cached decisions, a second source freshness check and explicit model/service error states. +- Complete-link problem grouping with valid historical corroboration edges retained across invocations. +- Deterministic, source-leakage-aware, blinded human annotation queue; private offline two-reviewer UI; rubric and disagreement/coverage evaluator. +- Main protected by required Node 22/24 + PostgreSQL CI; contributor bot may auto-merge after green checks. + +## Evidence and remaining acceptance criteria + +As of this handoff, two bounded batches of six alternate-model leads produced **12 new semantic reviewer decisions**: **1 `same_problem`, 11 `different_problem`, 0 errors**. The total database contains **2 model-confirmed same-problem edges** and **18 different-problem edges**. These are **model decisions**, not independently labeled ground truth. + +The current local benchmark pilot has **150 unlabeled pairs** and **fails the pre-annotation quality gate**: + +- 0 pairs above the configured high-similarity cutoff of 0.60 in the current `all-minilm` development pool. +- 7 distinct Hacker News original discussions in cross-platform comparisons, below the gate's minimum of 8. +- Source selection and earlier automatic judgments can introduce sampling bias; these are not population-level metrics. + +Therefore the **technical implementation is ready for independent evaluation**, but **P0 independent validation is NOT finished** and GitHub issue #3 must remain open. Do not claim recall, semantic precision or market fit from the development corpus or LLM-generated verdicts. + +## Private review pilot on maintainer Mac + +Private directory: `~/.local/share/memolabs/pie/p0-annotation-pilot-20261013/` (not committed to GitHub). It contains frozen source queue, sampling audit, SHA-256 manifest, blank independent reviewer templates, browser-based reviewer A/B HTML pages and a machine-readable quality-gate result. + +The files are a **pilot**, not a passing holdout. Keep the audit separate from annotators; they should only see `pairs-blind.json` or the generated reviewer HTML. Collect two independent human judgments of source-grounded candidate pairs and independently authenticate review independence outside the tool. Reviewers must not see each other's answers or existing model verdicts. Before a publishable benchmark, mine more challenging cross-thread, cross-community positive candidates and freeze a *new*, passing holdout with no prior tuning/review leakage. Adjudicate disagreements and publish test-set composition, coverage and uncertainty together with any metrics. The pipeline must not fill human review templates automatically. + +For a completed, valid holdout use `npm run eval:review-holdout -- [predictions.json]` and retain original submissions and adjudication evidence. Never close P0 #3 until those two independent human annotation sets and a genuinely qualifying holdout exist. diff --git a/package.json b/package.json index 350138a..b3c4c9c 100644 --- a/package.json +++ b/package.json @@ -30,7 +30,8 @@ "eval:annotation-ui": "node packages/agents/annotation-ui.mjs", "eval:embedding-retrieval": "node packages/agents/evaluate-embeddings.mjs", "eval:compare-embeddings": "node packages/agents/compare-embeddings.mjs", - "research:alternate-pairs": "node packages/agents/mine-alternate-pairs.mjs" + "research:alternate-pairs": "node packages/agents/mine-alternate-pairs.mjs", + "research:review-alternate": "node --env-file-if-exists=.env packages/agents/review-alternate-pairs.mjs" }, "dependencies": { "pg": "^8.23.1" diff --git a/packages/agents/mine-alternate-pairs.mjs b/packages/agents/mine-alternate-pairs.mjs index f0f9547..f2445c3 100644 --- a/packages/agents/mine-alternate-pairs.mjs +++ b/packages/agents/mine-alternate-pairs.mjs @@ -56,8 +56,8 @@ export function rankNovelCrossDiscussionPairs(rows,embeddings,priorEdges=[],{lim exposure.set(row.rightId,(exposure.get(row.rightId)||0)+1); const distinctAccounts=row.left.source===row.right.source&&row.left.reporter_key&&row.right.reporter_key ?row.left.reporter_key!==row.right.reporter_key:null; - result.push({left:{id:row.leftId,url:row.left.url,source:row.left.source,statement:row.left.statement}, - right:{id:row.rightId,url:row.right.url,source:row.right.source,statement:row.right.statement}, + result.push({left:{id:row.leftId,runId:row.left.run_id,url:row.left.url,source:row.left.source,statement:row.left.statement}, + right:{id:row.rightId,runId:row.right.run_id,url:row.right.url,source:row.right.source,statement:row.right.statement}, similarity:Number(row.similarity.toFixed(4)),platformPair:row.platformPair, distinctSourceAccounts:distinctAccounts,independentlyVerifiedHumans:null, semanticEquivalence:'unreviewed',commercialDemand:'unknown'}); diff --git a/packages/agents/review-alternate-pairs.mjs b/packages/agents/review-alternate-pairs.mjs new file mode 100644 index 0000000..6bd7230 --- /dev/null +++ b/packages/agents/review-alternate-pairs.mjs @@ -0,0 +1,106 @@ +import {readFile,writeFile} from 'node:fs/promises'; +import {createPool} from '../storage/postgres.mjs'; +import {verifyProblemPair,SEMANTIC_REVIEW_POLICY_VERSION} from './corroborate.mjs'; +import {EMBEDDING_MODEL} from './embed.mjs'; + +/** All source revisions must still be current at review time. */ +export const ALTERNATE_REVIEW_CURRENT_SQL=` +WITH latest AS ( + SELECT d.run_id,d.observation_id,d.status,d.statement,d.excerpt,d.source_content_hash, + row_number() OVER(PARTITION BY d.observation_id ORDER BY r.started_at DESC,r.id DESC) rn + FROM model_decisions d JOIN model_runs r ON r.id=d.run_id +) +SELECT d.run_id,d.observation_id,d.status,d.statement,d.excerpt, + o.thread_root_id,o.content_hash,o.url,o.source +FROM latest d JOIN observations o ON o.id=d.observation_id +WHERE d.rn=1 AND d.status='found' AND d.source_content_hash=o.content_hash +AND d.observation_id=ANY($1::text[])`; +const priorSQL=`SELECT verdict,rationale FROM corroboration_edges +WHERE left_run=$1 AND left_observation=$2 AND right_run=$3 AND right_observation=$4 AND model=$5 LIMIT 1`; +const saveSQL=`INSERT INTO corroboration_edges +(left_run,left_observation,right_run,right_observation,model,verdict,rationale) +VALUES ($1,$2,$3,$4,$5,$6,$7) +ON CONFLICT(left_run,left_observation,right_run,right_observation,model) +DO UPDATE SET verdict=EXCLUDED.verdict,rationale=EXCLUDED.rationale,compared_at=NOW() +WHERE EXCLUDED.verdict<>'error'`; +export function validateAlternateMiningQueue(queue,{limit=12}={}){ + if(queue?.kind!=='unreviewed_alternate_embedding_candidates'||!Array.isArray(queue.pairs)|| + !Number.isInteger(limit)||limit<1||limit>24)throw Error('Invalid alternate candidate report or review budget'); + const seen=new Set(); + const pairs=[]; + for(const item of queue.pairs.slice(0,limit)){ + const a=item?.left?.id,b=item?.right?.id; + if(typeof a!=='string'||typeof b!=='string'||!a||!b||a===b|| + typeof item.left.runId!=='string'||!item.left.runId||typeof item.right.runId!=='string'||!item.right.runId) + throw Error('Invalid mined source pair or missing source snapshot; regenerate the mined report'); + const sorted=[a,b].sort();const key=sorted.join('|'); + if(seen.has(key))throw Error('Repeated mined source pair'); + seen.add(key);pairs.push(sorted); + } + return pairs; +} +export async function reviewAlternatePairs(pool,queue,{limit=12,reviewerModel='nvidia/nemotron-3.5-lightning:free', + judge=verifyProblemPair}={}){ + if(!reviewerModel.endsWith(':free'))throw Error('Review requires the approved free semantic model'); + const pairs=validateAlternateMiningQueue(queue,{limit}); + const ids=[...new Set(pairs.flat())]; + const {rows}=ids.length?await pool.query(ALTERNATE_REVIEW_CURRENT_SQL,[ids]):{rows:[]}; + const current=new Map(rows.map(r=>[r.observation_id,r])); + const modelKey=EMBEDDING_MODEL+'+'+reviewerModel+'+'+SEMANTIC_REVIEW_POLICY_VERSION; + const results=[]; + for(const [leftId,rightId] of pairs){ + const a=current.get(leftId),b=current.get(rightId); + const submitted=queue.pairs.find(item=>[item.left.id,item.right.id].sort().join('|')===leftId+'|'+rightId); + const expected=new Map([[submitted.left.id,submitted.left.runId],[submitted.right.id,submitted.right.runId]]); + if(!a||!b||a.run_id!==expected.get(leftId)||b.run_id!==expected.get(rightId)){ + results.push({left:leftId,right:rightId,status:'stale_or_unverified'});continue; + } + if((a.thread_root_id||a.observation_id)===(b.thread_root_id||b.observation_id)){ + results.push({left:leftId,right:rightId,status:'same_original_discussion'});continue; + } + const params=[a.run_id,a.observation_id,b.run_id,b.observation_id,modelKey]; + const old=await pool.query(priorSQL,params); + if(old.rows[0]&&old.rows[0].verdict!=='error'){ + results.push({left:leftId,right:rightId,status:'cached',verdict:old.rows[0].verdict});continue; + } + let verdict; + try{ + verdict=await judge({left_id:leftId,right_id:rightId, + left_statement:a.statement,left_excerpt:a.excerpt, + right_statement:b.statement,right_excerpt:b.excerpt}, + {env:{...process.env,PIE_LLM_PROVIDER:'openrouter',PIE_LLM_MODEL:reviewerModel}}); + if(!['same_problem','different_problem','uncertain'].includes(verdict?.verdict)|| + typeof verdict.rationale!=='string'||!verdict.rationale.trim())throw Error('Invalid semantic reviewer verdict'); + }catch(error){results.push({left:leftId,right:rightId,status:'error',error:String(error?.message||error).slice(0,180)});continue;} + // Confirm the source snapshot has not changed during the remote LLM call. + const recheck=await pool.query(ALTERNATE_REVIEW_CURRENT_SQL,[[leftId,rightId]]); + const now=new Map(recheck.rows.map(x=>[x.observation_id,x.run_id])); + if(now.get(leftId)!==a.run_id||now.get(rightId)!==b.run_id){ + results.push({left:leftId,right:rightId,status:'stale_during_review'});continue; + } + await pool.query(saveSQL,[...params,verdict.verdict,verdict.rationale.slice(0,240)]); + results.push({left:leftId,right:rightId,status:'reviewed',verdict:verdict.verdict}); + } + return {kind:'audited_alternate_semantic_review',retrievalModel:queue.model, + model:reviewerModel,policy:SEMANTIC_REVIEW_POLICY_VERSION, + independentHumanLabels:false,marketDemand:'unknown', + reviewed:results.filter(r=>r.status==='reviewed').length, + cached:results.filter(r=>r.status==='cached').length, + errors:results.filter(r=>r.status==='error').length, + sameProblem:results.filter(r=>r.verdict==='same_problem').length, + differentProblem:results.filter(r=>r.verdict==='different_problem').length, + uncertain:results.filter(r=>r.verdict==='uncertain').length, + results}; +} +if(process.argv[1]?.endsWith('/packages/agents/review-alternate-pairs.mjs')){ + const source=process.argv[2]; + if(!source)throw Error('Usage: npm run research:review-alternate -- /private/mined-pairs.json'); + const queue=JSON.parse(await readFile(source,'utf8')); + const pool=createPool();try{ + const result=await reviewAlternatePairs(pool,queue,{limit:Number(process.env.PIE_ALTERNATE_REVIEW_LIMIT||6)}); + const destination=process.env.PIE_ALTERNATE_REVIEW_REPORT_PATH; + if(destination)await writeFile(destination,JSON.stringify(result,null,2)+'\n',{mode:0o600,flag:'wx'}); + console.log(JSON.stringify(result,null,2)); + if(result.errors)process.exitCode=1; + }finally{await pool.end();} +} diff --git a/tests/mine-alternate-pairs.test.mjs b/tests/mine-alternate-pairs.test.mjs index bfa7e28..3032637 100644 --- a/tests/mine-alternate-pairs.test.mjs +++ b/tests/mine-alternate-pairs.test.mjs @@ -12,6 +12,7 @@ test('rejects same-story comments, previously reviewed current snapshots, and ca assert.ok(result.sameThreadSkipped>=1); assert.equal(result.alreadyReviewedSkipped,1); assert.equal(result.selected.length,2); + assert.equal(result.selected[0].left.runId,'run-'+result.selected[0].left.id); assert.ok(!result.selected.some(x=>x.left.id==='a'&&x.right.id==='b')); assert.ok(result.selected.every(x=>x.semanticEquivalence==='unreviewed')); assert.ok(result.selected.every(x=>x.independentlyVerifiedHumans===null)); diff --git a/tests/review-alternate-pairs.test.mjs b/tests/review-alternate-pairs.test.mjs new file mode 100644 index 0000000..3e2f13e --- /dev/null +++ b/tests/review-alternate-pairs.test.mjs @@ -0,0 +1,65 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {reviewAlternatePairs,validateAlternateMiningQueue,ALTERNATE_REVIEW_CURRENT_SQL} from '../packages/agents/review-alternate-pairs.mjs'; +const queue={kind:'unreviewed_alternate_embedding_candidates',model:'nomic-embed-text',pairs:[{left:{id:'github:a',runId:'run-a'},right:{id:'github:b',runId:'run-b'}}]}; +const source=(id,root=id)=>({run_id:'run-'+id,observation_id:'github:'+id,statement:'Specific failure '+id,excerpt:'The source reports failure '+id,thread_root_id:root,source:'github',url:'https://github.com/a/b/issues/'+id}); +function db(rows,{previous=null,recheck=null}={}){ + let reads=0;const writes=[]; + return {writes,query:async(sql,args)=>{ + if(sql===ALTERNATE_REVIEW_CURRENT_SQL){reads++;return {rows:reads===2&&recheck?recheck:rows};} + if(sql.startsWith('SELECT verdict,rationale FROM corroboration_edges'))return {rows:previous?[previous]:[]}; + if(sql.startsWith('INSERT INTO corroboration_edges')){writes.push(args);return {rowCount:1};} + throw Error('Unexpected query '+sql.slice(0,30)); + }}; +} +test('queue validation rejects duplicate or identical observation pairs',()=>{ + assert.deepEqual(validateAlternateMiningQueue(queue),[['github:a','github:b']]); + assert.throws(()=>validateAlternateMiningQueue({...queue,pairs:[queue.pairs[0],{left:{id:'github:b',runId:'run-b'},right:{id:'github:a',runId:'run-a'}}]}),/Repeated/); + assert.throws(()=>validateAlternateMiningQueue({...queue,pairs:[{left:{id:'github:a',runId:'run-a'},right:{id:'github:a',runId:'run-a'}}]}),/Invalid/); + assert.throws(()=>validateAlternateMiningQueue(queue,{limit:100}),/Invalid/); +}); +test('semantic review persists verdict under canonical production reviewer policy and source revisions',async()=>{ + const pool=db([source('a'),source('b')]);let judged=0; + const result=await reviewAlternatePairs(pool,queue,{judge:async (pair,options)=>{judged++;assert.match(pair.left_excerpt,/failure/);assert.equal(options.env.PIE_LLM_MODEL,'nvidia/nemotron-3.5-lightning:free');return {verdict:'different_problem',rationale:'Different bugs'};}}); + assert.equal(result.reviewed,1);assert.equal(result.differentProblem,1);assert.equal(judged,1); + assert.equal(pool.writes.length,1); + assert.match(pool.writes[0][4],/^all-minilm\+nvidia\/nemotron-3\.5-lightning:free\+same-problem-v2$/); + assert.equal(result.independentHumanLabels,false); +}); +test('previous valid verdict is cached without asking the model again',async()=>{ + const pool=db([source('a'),source('b')],{previous:{verdict:'same_problem',rationale:'Same failure'}}); + const result=await reviewAlternatePairs(pool,queue,{judge:async()=>{throw Error('Should not review');}}); + assert.equal(result.cached,1);assert.equal(result.sameProblem,1);assert.equal(pool.writes.length,0); +}); +test('missing or reclassified sources cannot enter corroboration evidence',async()=>{ + const pool=db([source('a')]);const result=await reviewAlternatePairs(pool,queue,{judge:async()=>{throw Error('No');}}); + assert.equal(result.results[0].status,'stale_or_unverified'); + assert.equal(pool.writes.length,0); +}); +test('a source change during review prevents stale verdict persistence',async()=>{ + const pool=db([source('a'),source('b')],{recheck:[source('a'),{...source('b'),run_id:'new-run'}]}); + const result=await reviewAlternatePairs(pool,queue,{judge:async()=>({verdict:'same_problem',rationale:'Possibly same'})}); + assert.equal(result.results[0].status,'stale_during_review'); + assert.equal(pool.writes.length,0); +}); +test('multiple HN replies within same original thread are not reviewed',async()=>{ + const hn=structuredClone(queue);hn.pairs=[{left:{id:'github:a',runId:'run-a'},right:{id:'github:b',runId:'run-b'}}]; + const pool=db([source('a','hackernews:original'),source('b','hackernews:original')]); + const result=await reviewAlternatePairs(pool,hn,{judge:async()=>{throw Error('Never');}}); + assert.equal(result.results[0].status,'same_original_discussion'); +}); +test('model errors are tracked separately from semantically negative verdicts',async()=>{ + const pool=db([source('a'),source('b')]); + const result=await reviewAlternatePairs(pool,queue,{judge:async()=>{throw Error('HTTP 429');}}); + assert.equal(result.errors,1);assert.equal(result.differentProblem,0);assert.equal(pool.writes.length,0); +}); + +test('candidate report must preserve exact extraction run IDs from mining time',async()=>{ + const pool=db([source('a'),source('b')]); + const modified=structuredClone(queue);modified.pairs[0].left.runId='older-run'; + const result=await reviewAlternatePairs(pool,modified,{judge:async()=>{throw Error('Must not review stale mining pair');}}); + assert.equal(result.results[0].status,'stale_or_unverified'); + assert.equal(pool.writes.length,0); + const legacy=structuredClone(queue);delete legacy.pairs[0].right.runId; + assert.throws(()=>validateAlternateMiningQueue(legacy),/missing source snapshot/); +});