Skip to content

feat(llm): collect bounded request-cost evidence - #668

Draft
zzwong wants to merge 2 commits into
dot/issue597-tool-evidence-v4-20261008from
feat/request-cost-evidence-621
Draft

zzwong wants to merge 2 commits into
dot/issue597-tool-evidence-v4-20261008from
feat/request-cost-evidence-621

Conversation

@zzwong

@zzwong zzwong commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Refs #621.

Collect bounded per-request usage evidence for the direct OpenAI Responses adapter and retain it across structured-output corrections, transport retries, cache reloads, and task resets.

  • Preserve nullable usage, observed model/service tier, dispatch uncertainty, and independent attempt ordinals without storing prompts, response bodies, URLs, headers, or credentials.
  • Keep owned single-send transport evidence distinct from opaque clients. Disable redirect/replay behavior for the owned transport.
  • Persist bounded generation checkpoints under the task lock, record pending work before dispatch, and preserve known observations when metadata, sidecars, or bindings disagree.
  • Retain detached primary/repair histories and keep optional cost data isolated from reusable output and reviewer-tool evidence.

This is collection-only. Task-history completeness remains false because a fresh accounting origin is not established. Interrupted, missing, conflicting, legacy, opaque, and overflowing evidence stays uncertain. No request pricing, invoice estimate, catalog activation, or estimator activation is added. Per-attempt crash durability and power-loss durability are not claimed.

Stack and source scope

This draft targets #662's branch, dot/issue597-tool-evidence-v4-20261008, pinned at 982341589d5c3db76ac260626faa325df4ff0dc1. It also depends on #666 at 29edd5c6877be9cda7341ee6f5b11b0acdcff5fb.

The source was reviewed against the exact #662 tree with only #666's internal/llmadapters/api.go and api_test.go overlaid. The collection slice changes 15 paths over that composition. Relative to #662, this PR changes 16 paths because it also carries #666's unchanged api_test.go; the adapter source includes both the cache-write prerequisite and collection changes. Neither dependency's passing checks are inherited as validation of this tree.

Original reviewed source009 tree: 1e596df55fa9e5f1baccde01d4f7f21cb814c446.
Current lint-correction head: 4c1d9887e3d88fac17b749630936ed5e0ebefd77; tree 5d5686bf08641900a73cfab7acd13f8131b12b6c.

The current head is a non-force child of the original publication head. It applies a separately reviewed five-file lint correction: equivalent bounded length calculations, fixture-read confinement, comments/names, and a documented test-only nil-context exception. All 13 fixture files are unchanged.
Source009 inventory SHA-256: 8a37ba4e7c61915f1a14c54799da00631449a10358ca43b51f6cae491e7e00fc.

All 423 source files and modes were checked. Only the 16 listed source, documentation, test, and synthetic-fixture paths differ from #662. go.mod, go.sum, and workflow files are unchanged. The catalog-only draft #667 is separate and is not included.

Validation and limits

Current lint-correction head

Head 4c1d9887e3d88fac17b749630936ed5e0ebefd77 passed source review, bounded formatting, and remote verification of all 423 files and modes.

Hosted CI run 37757949065 passed all 10 jobs: lint, standard tests, no-1Password opt-out tests, static smoke, Linux/macOS/Windows builds, the build aggregate, PR title, and identity. The checkout was synthetic merge 02aa567e056da311120c0dd680f11f0d053a0794; its independently verified tree is the exact successor 5d5686bf08641900a73cfab7acd13f8131b12b6c.

All 43 authored request-cost tests passed again in each of the three test profiles, totaling 73 selected test/subtest pass events per profile, with no selected skips or failures. The profiles used Go 1.26.8 on Linux and reported no cached package results. Standard/static profiles each recorded 3,677 pass events and four unrelated skips; opt-out recorded 3,661 pass events and 17 unrelated skips, in the same categories retained below. Lint reported zero issues.

These are new results for the successor, separate from the original run below. Local target tests, compilation, lint, race validation, and the 12 historical-defect controls remain NOT RUN. Hosted success does not establish provider-invoice completeness, fresh accounting origin, or estimator readiness.

Original source009 hosted evidence

  • PASS: independent source, corrective-delta, and formatting-only reviews of the pinned source lineage.
  • PASS: byte-preserving patch application to the exact base, followed by comparison of all 423 files and modes.
  • HOSTED CI: run 37756483210 completed for publication head 813c993708cd13da830c299948b5994eb110b05e. Its checkout was synthetic merge e1e8a5d04668b078131774710fa88fbe01d47acd, whose verified tree equals 1e596df55fa9e5f1baccde01d4f7f21cb814c446.
  • PASS: nine hosted jobs: standard tests, no-1Password opt-out tests, static smoke, Linux/macOS/Windows builds, the build aggregate, PR title, and identity.
  • PASS: all 43 authored request-cost tests, totaling 73 test/subtest pass events, in each of the three hosted test profiles. None of those selected tests failed or skipped; no cached package-result lines appeared. All three test profiles used Go 1.26.8 on Linux.
  • PRESERVED SKIPS: standard tests and static smoke each had 3,677 pass events and four unrelated skips, covering unavailable 1Password behavior, installed Pi, opt-in live-provider integration, and cross-volume behavior. The opt-out profile had 3,661 pass events and 17 unrelated skips, including the 1Password-specific cases excluded by that profile.
  • FAIL: lint, with 11 diagnostics: three integer-conversion warnings, two variable-path fixture-read warnings, five comment/name/unused-parameter findings, and one nil-context finding. This was the original run's sole failed job; that run remains recorded as failed.
  • NOT RUN locally: target tests, compilation, type checks, full packages/repository tests, lint, or build for this candidate. Local formatting and source-text transport checks remain separate evidence.
  • NOT RUN: race validation and all 12 planned historical-defect controls. The hosted workflow does not execute that separate control plan. No live-provider validation is claimed.

These hosted results cover only the exact source and profiles above. They do not supply validation for a later correction or inherit checks from either dependency. The earlier combined integration attempt remains UNKNOWN and supplies no test credit for this PR.

This remains a draft. Merge readiness is unestablished and #621 stays open.

zzwong added 2 commits October 8, 2026 05:24
Retain bounded direct-API request observations and durable unknown history without enabling cost estimates.

Refs #621.
Preserve bounded accounting and synthetic fixture coverage while correcting lint findings.

Refs #621.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant