Skip to content

fix(cost): enforce verifiable scan spending limits - #465

Draft
mldangelo-oai wants to merge 32 commits into
mainfrom
mdangelo/codex/enforce-verifiable-scan-budgets
Draft

fix(cost): enforce verifiable scan spending limits#465
mldangelo-oai wants to merge 32 commits into
mainfrom
mdangelo/codex/enforce-verifiable-scan-budgets

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Enforce explicitly requested scan spending limits when final root or delegated-worker usage cannot be verified. Preserve usage across observed counter resets without charging inherited fork history to a child worker.

Fixes #223. Related prior work: #224.

Changes

  • Accumulate normalized, owned root and worker usage across observed counter resets without counting the same session twice.
  • Treat both decreases in aggregate usage and equal-total per-field counter resets as new billable usage in both the live TypeScript reader and persisted Python reconstruction, without double-counting identical snapshots.
  • Keep cached-input, cache-write, and reasoning deltas within their corresponding input or output totals; preserve every primary token and prioritize higher-priced cache writes when counters reset independently.
  • Compare complete UUIDv7 values at fork boundaries so a retained parent turn from the same millisecond cannot become child-owned spending.
  • Attribute fork-only delegated sessions to their parent during live polling, matching persisted nested-parent, direct-parent, and fork-parent precedence so an explicit spending limit can stop a standard scan immediately.
  • Require attributable, readable, complete, and priceable usage only when a spending limit was explicitly requested. Keep optional accounting best-effort.
  • Require every authenticated owned rollout to be present and accounted for when an explicitly budgeted scan finishes, using the existing read-only Codex session graph and scan workbench records, including independently rooted Deep Scan workers. Reject missing, unobserved, empty, incomplete, malformed, or legacy-identity owned rollouts while ignoring confirmed unrelated sessions.
  • Resolve ownership only during explicitly budgeted finalization; normal polling, cleanup-only stops, and optional accounting launch no ownership subprocess. The ownership subprocess inherits the existing isolated, credential-free workbench environment.
  • Preserve authoritative completed root usage, verified worker usage, observed overages, and cancellation while draining completed Deep-worker streams.
  • Keep malformed completed rollout records fail-closed for owned budgeted sessions without rejecting unrelated sessions or optional accounting.
  • Advance the bundled plugin manifest and SDK cache marker together to 0.1.53 so previously installed 0.1.52 plugins receive the corrected persisted reader.
  • Keep unsupported internal helpers and dependency-injection overloads out of public TypeScript declarations.

Testing

Validated exact head b4406c7f16a268f615c5ddcd907b6e6aeb329eb1:

  • Cost, API, runtime, and Deep-worker shutdown suites: 432 passed, 9 platform-specific tests skipped, 0 failed across 3,135 assertions.
  • Two independent exact-head reviews found no actionable issues; independent cost verification passed 169 tests, and focused committed-head fork-lineage, ownership, and accounting regressions passed.
  • Reproduced same-millisecond fork misattribution before the fix: 1,100 input / 110 output tokens and $0.0088 instead of 100 / 10 tokens and $0.0008 under a $0.001 spending limit.
  • Reproduced a fork-only standard worker remaining invisible to live budget tracking at $0.00032 instead of the actual $0.00128 total. Real-file regressions now trigger immediate cancellation above a $0.001 limit while excluding inherited same-millisecond history and unrelated, empty, or malformed parent identifiers.
  • Reproduced persisted-reader/reset disagreement before the fix: 2,600 / 1,150 tokens instead of 1,600 / 150. Existing cross-reader and reset controls now pass with same-millisecond UUIDv7 fixtures.
  • Reproduced equal-total counter rotation hiding spending in both readers: $0.01333 stayed below a $0.017 limit instead of the correct $0.02002. Live and persisted accounting now agree across input, cached input, cache-write, output, reasoning, and total counters, and identical snapshots are not counted twice.
  • Reproduced an independent cached-counter reset accepting stale $0.00122 usage under a $0.0015 budget instead of the corrected $0.0024082 total. Verified cached-input, premium cache-write, competing cache categories, and reasoning resets preserve valid counters without hiding billable input or output.
  • Reproduced incomplete worker metadata accepting $0.00032 against a $0.001 limit while a completed worker later proved actual usage of $0.03232. Budget finalization now rejects incomplete owned worker metadata while accepting authenticated unrelated partial sessions.
  • Exercised real WAL-backed workbench and Codex SQLite fixtures for delegated workers, independently rooted Deep workers, and unrelated sessions with missing or unobserved owned rollouts, empty files, partial or malformed metadata, complete unattributed token records, and compatible session_id metadata, plus missing graph descendants, root sessions, state databases, and scan records.
  • Verified the public scan API rejects an authenticated missing worker under an explicit limit while the equivalent optional scan completes without launching ownership verification.
  • Confirmed ambiguous sessions never hide other incomplete owned usage, refresh polling never launches the ownership subprocess, Python interpreter selection supports Windows, and isolated subprocesses receive none of four supported API credential variables.
  • Independently compared both readers across 60 generated seven-snapshot rollout sequences, confirming exact six-counter parity, valid cached/reasoning bounds, accepted persisted completion records, and no pricing undercounts.
  • Used the actual bundled Codex executable to install a legacy 0.1.52 plugin, upgrade its real versioned cache to 0.1.53, and execute the newly installed persisted reader against equal-total duplicate and cached-counter-reset snapshots while preserving stored credentials.
  • Added five real-file malformed-record controls covering owned workers, root completion recovery, unrelated sessions, and optional accounting.
  • Generated-model consistency, TypeScript type checking, complete Prettier validation, SDK build, and Python syntax validation passed.
  • Packed and validated the installed package offline: 226 tarball entries, 106 bundled plugin files, public SDK import, CLI startup, bundled Codex version, and an actual nested worker without a global Codex executable.

Risk and rollout

Explicit spending limits now fail closed when owned usage or its authenticated session ownership cannot be verified; scans without an explicit limit retain best-effort reporting, and verified unrelated incomplete sessions do not block budgeted scans. Reset-containing histories can reconstruct higher legitimate usage, while same-millisecond inherited parent history is excluded from child budgets. Existing stored costs and public CLI options are unchanged.

The bundled plugin payload and both cache-version markers are synchronized at 0.1.53. This pull request remains a draft and currently conflicts with main; its base must be reconciled separately before any merge.

Public disclosure review

The branch, commit, source changes, description, and current public pull-request surface were reviewed. Existing automated review comments contain access-restricted report references, so the second attestation remains unchecked.

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@github-actions github-actions Bot added the bug Something isn't working label Aug 15, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: a147e46ca9

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a147e46ca9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/tests-ts/cost.test.ts Outdated
Comment thread sdk/typescript/src/cost.ts Outdated
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 95b3782aef

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95b3782aef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/cost.ts Outdated
Comment thread sdk/typescript/src/cost.ts Outdated
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: ce460fe36a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: ce460fe36a

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current head, b9f37a9b17ed5f3c9122b25c6316ba01cc2ffde0.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: b9f37a9b17

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9f37a9b17

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/cost.ts
Comment thread sdk/typescript/tests-ts/cost.test.ts
Comment thread sdk/typescript/src/cost.ts Outdated
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review the current head 0ba0c69.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex security review the current head 0ba0c69.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0ba0c6936f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/cost.ts Outdated
@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 0ba0c6936f

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review the current head 046c7ef.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex security review the current head 046c7ef.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 046c7ef8ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/cost.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: 046c7ef8ae

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Comment thread sdk/typescript/src/cost.ts
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review the current head 5f0baaf.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex security review the current head 5f0baaf.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5f0baafd64

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/cost.ts Outdated
@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 5f0baafd64

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review the current head cb6b36b.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex security review the current head cb6b36b.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cb6b36b452

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/_bundled_plugin/scripts/workbench_scan_usage.py
Comment thread sdk/typescript/src/cost.ts Outdated
@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: cb6b36b452

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review the current head 6bca8b1.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex security review the current head 6bca8b1.

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6bca8b19c2

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6bca8b19c2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/cost.ts
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review the current head b4406c7.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex security review the current head b4406c7.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: b4406c7f16

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: b4406c7f16

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An unreadable Codex session log fails every cost poll for the rest of the scan, and can fail a scan that succeeded

2 participants