[Typed evaluation roadmap] Ship reusable, durable decision programs across Tangle

[drewstone]

Outcome

One caller-owned typed evaluator works in products, campaigns, trace analysts, workflows and agent checkpoints. Code retains authorization and action execution; generative models retain open-ended work. Prove downstream improvement, not only API success.

Ten milestones

• M1: Released packages and real scoped calls work across product, campaign, workflow and graph with attributable receipts.
• M2: Multi-day execution survives crashes and credential rotation without silently repeating paid work or external effects; unknown outcomes stay explicit.
• M3: Independent labels establish judge errors, calibration and useful automation coverage with application-owned escalation policies.
• M4: Evidence-linked trace triage feeds deep investigation and repairs verified by executing changed behavior.
• M5: Context, memory and skill selection improves total task economics without losing mandatory constraints.
• M6: Outcome-based compute allocation beats a fixed policy on independently checked results and total resources.
• M7: Existing workflows mix code, native evaluations and ordinary agents with structured results and no duplicate runners.
• M8: Existing optimization methods improve executable decision programs under independent final assessment and complete cost accounting.
• M9: Verified production feedback advances exact candidates through shadow/limited rollout with exact rollback.
• M10: Existing UI explains consequential decisions and turns failures into reproducible evaluation cases.

Canonical execution backlog

• 1. Consumer package cohort and deployment proof — tangle-network/agent-dev-container#7686
• 2. Workflow replay across credential rotation — tangle-network/agent-dev-container#7687
• 3. Real persistence/process fault tests — tangle-network/tangle-router#538
• 4. Conserved auxiliary-inference budget — [Typed evaluation P0] Conserve parent authority across auxiliary evaluator calls agent-runtime#1294
• 5. Native SDK/JSON conformance — [Typed evaluation P0] Keep SDK and plain-JSON evaluation contracts identical across repos #769
• 6. Complete persisted decision observations — [Typed evaluation P0] Persist complete decision observations in existing trace and artifact storage #770
• 7. Independent calibration and risk/coverage reports — [Typed evaluation P0] Measure classifier risk, calibration and useful automation coverage #771
• 8. Classifier-assisted trace analysis — [Typed evaluation P1] Ship evidence-linked classifier-assisted trace analysis #772
• 9. Actual awaited execution hooks — [Typed evaluation P1] Connect evaluators to real awaited execution boundaries agent-runtime#1295
• 10. Context/memory/skill selection — [Typed evaluation P1] Select context, memory and skills without losing required state agent-runtime#1296
• 11. Outcome-based compute allocation — [Typed evaluation P1] Allocate compute using verified outcomes rather than confidence alone agent-runtime#1297
• 12. Structured workflow authoring/results — tangle-network/agent-dev-container#7688
• 13. Budgeted graph candidate selection — [Typed evaluation P1] Compare and select candidates through the existing budgeted graph agent-runtime#1298
• 14. Whole-program optimization — [Typed evaluation P2] Optimize executable decision programs with independent final assessment agent-runtime#1299
• 15. Verified feedback and controlled promotion — tangle-network/agent-dev-container#7689
• 16. Decision inspector and reproducible cases — tangle-network/agent-dev-container#7690

Behavioral validation extension: #774 covers independent/adversarial evidence for reward gaming, evaluator tampering, policy/authorization/data-boundary violations, injection-following, contradicted completion, oversight bypass, matched-baseline anomalies and cross-run hypotheses. These remain optional caller-owned recipes, not a default safety gate.

Current implementation status

Lane	Current work	Actual evidence
Eval	#776 on fix/evaluation-definition-identity, head 17453b832080b45db8a36e3e84a709906da6c0c9	Full CI passed. Fixes static evaluator identity collisions from question/Choice reordering and post-construction options mutation. Five regressions failed on baseline; eight new tests pass after fixes.
Platform / Intelligence	tangle-network/agent-dev-container#7698 on feat/native-workflow-results, head 6267f1d7fb61daef7399dcfd8f383df5b83b0a23	New output.evaluation works through shared action mappings without text parsing. Fixes native metadata loss and dropped child tool summaries (#7699 there). Nineteen focused local tests pass; final-head full CI/browser checks remain queued.
Router	tangle-network/tangle-router#539 is merged; part of tangle-network/tangle-router#538	Seven real PostgreSQL tests were added to the existing CI job. Latest database/process/billing proof was not established in this pass; broader issue remains open.
Runtime	tangle-network/agent-runtime#1294 and tangle-network/agent-runtime#1295 remain first	Existing budget/hook/receipt owners retained. No new Runtime patch claimed in the latest pass.

Eval #776 full CI: https://github.com/tangle-network/agent-eval/actions/runs/35416744633
Platform #7698 queued CI: https://github.com/tangle-network/agent-dev-container/actions/runs/35417116721

Merged foundation and previous evidence

• feat(meta-eval): audit frozen probability policies with existing independent error bounds #773 is merged: auditProbabilityPolicy reuses exact independent-unit audit bounds. Full head CI passed: https://github.com/tangle-network/agent-eval/actions/runs/35412288330 . Real-task labels/quality remain [Typed evaluation P0] Measure classifier risk, calibration and useful automation coverage #771.
• feat(jev): evidence-backed behavior reviews on existing evaluator and analyst contracts #775 is merged at cb4ff9c7f3e5721910277a99b5553bc5edd74d7d: prepared evidence reviews, supported/refuted/unresolved mapping, native probabilities and optional behavioral recipes. Its previously queued full-head CI is now verified successful: https://github.com/tangle-network/agent-eval/actions/runs/35414286915 . Detection effectiveness/adaptive robustness remain [Typed evaluation P1] Validate behavioral risk monitors against independent and adversarial traces #774; product adoption remains [Typed evaluation P1] Ship evidence-linked classifier-assisted trace analysis #772.
• Initial merged contracts: feat: reusable typed evaluations with native Jev adapters #761; fix: preserve analyst outcomes and stop authority in Jev-compatible execution agent-runtime#1283; tangle-network/agent-dev-container#7620; tangle-network/tangle-router#531, feat(benchmarks): official-metric leaderboard — extraction-verified head-to-head + the instrument-quality ledger #533 and refactor(analyst): move DSPy RLM execution behind a caller-owned callback #537 in that repo. They establish code contracts, not live quality or wallet-settlement proof.

Current Platform tests execute real modified source in a partial checkout with injected runAgent and isolated unrelated boundaries. No local full-monorepo signoff or deployed trace-ingest proof is claimed. Actual committed/tested blob hashes and baseline failures are recorded in both PRs.

Parallel work and integration boundaries

A / Router: tangle-network/tangle-router#538. Reuse durable request, dispatch and billing stores; no parallel settlement implementation.
B / Eval: #769, #770 and #771 can run independently in non-overlapping modules. #772 consumes observations and measured quality.
C / Runtime: tangle-network/agent-runtime#1294 and #1295 in that repo first; then selectors/candidates; optimization after a measured baseline.
D / Platform: #7698 owns current shared action/output changes. Cross-key recovery coordinates with identity/budget owners; inspector follows the observation contract.

Use isolated worktrees/file ownership and one open integration PR per repository. Reload current main/develop and inspect concurrent work before writing. No new scheduler, registry, ledger, memory manager or question language. No auto-deploy or frozen-release bypass.

The earlier GitHub attempt to assign the official coding agent received HTTP 403. No autonomous coding worker was started; the listed PRs were authored directly. Workstream names are ownership boundaries, not claims of background execution.

Existing ownership retained

• Payer/budget/renewal: tangle-network/agent-dev-container#7675, #7676 and #7677 in that repo.
• Managed process recovery: tangle-network/agent-dev-container#7658. Harness receipts: #7618 and #7365 there.
• Receipt/unknown-usage semantics: fix(accounting): preserve exact sandbox receipts and distinguish estimates from unpriced subscription work agent-runtime#1175, #1240 and #1252 in that repo. Retained execution recovery: #1288/#1290 there.
• Reuse statistics: browser-safe subpath, interpolating median, explicit empty-input policy #758 statistics and analyst: shape-collapse retry with a conformance record #759 analyst conformance here. Overlap is not proof an existing bug is obsolete.

Reconciliation

• Previously closed tangle-network/tangle-router#489 after verifying merged fix(benchmark): gate builds on source identity only, release 0.139.1 #502 and current availability code/tests/docs; not a new deployment measurement.
• Previously corrected fix(accounting): preserve exact sandbox receipts and distinguish estimates from unpriced subscription work agent-runtime#1175's diagnosis; exact receipt and subscription-pricing work remains open.
• Keep Runtime #1216/#1212 open: inspected legacy prompt accumulation differs from the improved executor path.
• Newly reproduced Platform #7699 has its fix and five regressions in #7698; it is linked for closure on merge, not closed before merge.
• Updated [Typed evaluation P0] Persist complete decision observations in existing trace and artifact storage #770, [Typed evaluation P1] Ship evidence-linked classifier-assisted trace analysis #772 and Platform #7688 with current implementation/proof boundaries; broader milestones remain open rather than equating unit tests with operational completion.

Closure evidence

Record exact revision, commands/results and untested boundaries. Distinguish source/types, local tests, final-head CI, real database/process tests, deployed API receipts, wallet reconciliation and independent quality measurements. Merges, queued/skipped CI and model-generated judgments do not satisfy stronger proof. No infrastructure purchase, increased spend, live provider run, release or production promotion was performed in these follow-through patches.

[drewstone]

Source consolidation after the latest merges

Confirmed #776 and tangle-network/agent-dev-container#7698 merged. The next changes stay in one initiative PR per owning repository:

Owner	PR / revision	Verified evidence
Eval	#778, a3abe93233dd480e8e6564e657ba37c9af7ec592	Full clean repository CI passed: https://github.com/tangle-network/agent-eval/actions/runs/35420637574 . Fixes generic asAnalyst replacing a configured capped evaluator account with a fresh unlimited one. Removes Jev's duplicate workaround; forwards existing physical callId to the ledger/transport. Fourteen new regressions, eight failing on baseline, pass after the fix.
Router	tangle-network/tangle-router#540, b8b6a8b52da4d756e9f05706056945fed36d2624	Core CI and actual browser/viewport matrix passed: https://github.com/tangle-network/tangle-router/actions/runs/35421283646 and https://github.com/tangle-network/tangle-router/actions/runs/35421283692 . Fixes native-route typechecking and smoke cleanup failure handling; removes ambient live-catalog dependence from alias tests without weakening assertions. The seven real PostgreSQL request-replay tests executed and passed.
Platform	tangle-network/agent-dev-container#7701, fd31be97aa1b5ba7a8330f28341f1e286e46aee3	Thirteen local regressions plus targeted typecheck pass; nine fail on the exact baseline. Standard Object.fromEntries fixes caller JSON loss/prototype mutation in the shared mapper, preserving native question order and existing expression access restrictions. Full CI remains queued: https://github.com/tangle-network/agent-dev-container/actions/runs/35421806899 . Local checks use the documented partial checkout, not full monorepo signoff.

No new runtime, provider wrapper, ledger class, mapping language or dependency. Runtime#1294 is updated to consume the fixed generic account selection instead of introducing another workaround. Its actual conserved-supervisor integration remains open.

Platform#7699 is now closed by the merged field-retention fix; confirmed against the issue state. Broader #770/#772, Platform#7686/#7687/#7688, and Router#538 remain open for their remaining acceptance, rather than equating these source fixes with installed/deployed integration, cross-key recovery, OS process-failure coverage or independently measured behavioral accuracy. Router#538 now records the executed PostgreSQL evidence instead of the stale unverified status.

No release, production promotion, secret/configuration change, live funded evaluation or wallet-settlement claim. The separate profile-cell PR #777 is another agent's non-overlapping work and was preserved.

[drewstone]

Publication status correction — verified 2026-09-29

The prepared consolidation changes are already published and merged. Do not republish the old ZIP or overwrite newer source.

• Eval feat(jev): preserve recorded evidence separately from live admission #897: feat(jev): preserve recorded evidence separately from live admission. Merged at 2026-09-29T01:05:11Z, merge commit 4fd034caf86b6c867ef8736f4f6e48bde5c1bf74. Adds the recorded-request decoder through the existing protocol owner while retaining strict live-request admission. feat(jev): preserve recorded evidence separately from live admission #897
• Platform/Intelligence tangle-network/agent-dev-container#8443: fix(intelligence): consolidate summary inference and preserve exact report evidence. Merged at 2026-09-29T02:16:31Z, merge commit 0a158c16d6c828acfd0c8159b782ecdc9cb4686a. Publishes the ten-file prepared consolidation: accounted summary execution, one JSON-schema definition, original customer receipts and failed-report settlement, historical report intervals, bounded exemplar evidence, HTTP/database regressions, and explicit live-smoke semantics. https://github.com/tangle-network/agent-dev-container/pull/8443

Current ADC develop was inspected at ba8dba7e50d61509b18d195c25e4609cc420eb3c. Its model-owner file includes the later customer-receipt corrections from #8443; the older bundle version must not replace it. This is a fresh verification of remote publication/merge state, not a claim these PRs were created again or that full CI/deployment/live-quality tests ran in this check.

Earlier conversation reports that the Intelligence changes had no published PR, or required an online desktop to publish through GitHub, were stale and incorrect. The current connected GitHub write actions are available; this correction is being posted through them. The broader production-adoption and outcome-quality milestones remain separate from source publication.

[drewstone]

Published consolidation follow-through

Two non-draft implementation PRs are open, one per affected repository. These are remote source changes, not unpublished patch bundles.

• Eval fix(llm): consolidate fallback into one physical attempt budget #902, head 64aeaf8683fff99bd9c99905d20a3ba1dea22717: schema, temperature and transient fallback share the existing physical request allowance, original deadline origin and raw-attempt sequence. The duplicate schema retry owner is removed and the reservation no longer includes a phantom second batch. Full final-head CI passed, including build, package exports and optimizer integrations: https://github.com/tangle-network/agent-eval/actions/runs/36649148569 . Local focused suite: 96 passed; five new HTTP regressions fail on baseline and pass after correction. Current implementation digest updated; historical benchmark evidence untouched.

• tangle-network/agent-dev-container#8533, head b493ec259875b20541bcd1e08e8c4540eab321dc: shared success/error response cleanup, bounded diagnostic reads, correct cumulative usage and terminal in-stream errors. Existing #8520 consolidation and concurrent #8534 parser cleanup are preserved by a normal two-parent merge. 17 real-loopback HTTP cases pass after reconciliation; baseline 13 pass / 4 fail. Exact-head CI is pending: https://github.com/tangle-network/agent-dev-container/actions/runs/36649587192 . No full-workspace or coverage claim from the restored local dependency tree.

No new package, ledger, model default, provider SDK or transport framework. No weakened test/signoff/coverage gates. Temporary isolated delivery workflows were removed; feature diffs contain source, relevant tests/docs and Eval's source line-ending contract only.

The core safety/reliability benefit is one owner for attempts and one response lifetime, not replacing authorization or provider accounting. Keep product fetch guards until the fixed Eval release is actually published and installed. Broader typed finding retention, unified cost provenance across optimization/investigation, workflow credential-rotation recovery and independently measured product quality remain open rather than being falsely closed by these transport fixes. No production deployment, package publication or funded inference performed.

[drewstone]

Publication reconciled and remaining transport fix published

Verified current GitHub state: agent-dev-container#8627 is merged (2026-10-01), and agent-eval#902 is merged. The old retained transport bundle largely duplicates #8627 and must not be reapplied wholesale.

Published a new non-draft PR for the actual remaining gap: tangle-network/agent-dev-container#8660, head d4c6c09aeec9a850720b522acb80f712b68fe377, based on develop 44f636f77797f6005306ca76294396626cef4b55.

It changes one production condition: a provider-specific typed event carrying error cannot be silently ignored and subsequently release pending tools or completed usage. tool_result keeps its legitimate tool-level errors. The PR extends the existing real-HTTP case table by one row and updates the existing owner document. Remote diff verified: exactly three files, 11 additions / 6 deletions. No new transport, dependency, runner, test harness, or weakened gate.

Executed the same 34-case suite this pass against exact current-source bytes: baseline 33 passed / 1 failed, candidate 34 passed / 0 failed. All 33 prior cases remain. Local execution uses native fetch/loopback HTTP but restored older dependency artifacts and resolution aliases, not the current frozen monorepo graph. Required current-workspace verification and independent review remain unclaimed.

New exact-head CI is queued: https://github.com/tangle-network/agent-dev-container/actions/runs/36831267221 . The browser job is also queued. No deployment, package publication, funded provider request or merge was performed.

This update corrects the stale claim that these changes cannot be published: this pass successfully created Git blobs, the commit, the feature branch update, and PR #8660 through the connected GitHub write actions. Broader Intelligence receipt-provenance, typed-finding and measured-quality work is still separate from this focused transport fix.

[drewstone]

Publication/status correction: the remaining provider-specific stream-error fix is published in non-draft tangle-network/agent-dev-container#8660. I updated that existing PR rather than creating another overlapping PR.

This turn successfully created and pushed reconciliation commit f55d54908d145a411d7dba610b91e5d049bf86f4 through the connected GitHub write actions. It incorporates develop 2afbdd0b3550d7ce7f11760abbe863b88ff50e3b, preserves both histories, uses a non-forced fast-forward, and changes only the same backend, existing HTTP suite, and contract document relative to that base. GitHub reports the branch merges cleanly at this update.

Fresh local rerun: identical 34-case real-loopback/native-fetch suite gives baseline 33 pass / 1 fail, candidate 34 pass / 0 fail. Production/test Git blob hashes match the committed source. Local dependencies are restored older Runtime/Eval versions with resolution-only aliases, not the full declared workspace; remote/current-cohort checks remain required. No signoff, full CI pass, deployment or merge is claimed.

#8533, #8627 and agent-eval#902 are already merged and are not duplicated. The prior chat statements that this source was wholly unpublished or that the connected session could not write were incorrect. Broader Intelligence provenance, typed findings and quality benchmark work remains separate from this small transport correction.