Add the #485 design simulations behind an invariant test - #486
Open
jeremymanning wants to merge 6 commits into
Open
Add the #485 design simulations behind an invariant test#486jeremymanning wants to merge 6 commits into
jeremymanning wants to merge 6 commits into
Conversation
The supported providers are Dartmouth Chat and the HuggingFace Inference API (#484). The Anthropic / OpenAI / Google / Ollama / local-HuggingFace adapters were frozen yesterday; this removes them. Deleted: - integrations/{anthropic,google,openai,ollama,lazy_ollama,huggingface,lazy_huggingface}_model.py - models/{anthropic,openai}_model.py (the duplicate AnthropicModel from #429) - models/providers/anthropic_provider.py - models/registry.py + models/config.py -- the skills-era 'unified' registry, a competing implementation that only ever supported Anthropic - tools/update_models.py -- existed only to refresh retired providers' model lists - The packaged default model pool (config/models.yaml) listed ONLY retired providers; it now ships empty. Dartmouth models come from the live catalog. - 27 test files that tested exclusively the retired code, incl. the three unit-marked ones (test_anthropic_model_names, test_update_models_no_hardcoded_ids, one probe in test_cli_hermetic_and_quiet) Canonical-path edits: - _api.populate_model_registry: retired-source models.yaml entries are skipped with a naming warning, never raised on; the Ollama probe and the credential-copying backcompat map are gone - hybrid_control_system: the no-registry fallback used to construct a gpt-4 client -- a misconfigured run silently spending money on an API the user never chose. It now returns the existing 'No suitable model' error. - Public surface: HuggingFaceModel / OllamaModel / AnthropicProvider and the frozen api/ layer names (PipelineAPI et al., unimportable after the competing registry's removal) no longer resolve - pyproject: anthropic/openai/google extras removed Contract tests first: tests/test_provider_retirement.py went RED on all four new-behavior cases (packaged pool offers retired providers; retired sources register; gpt-4 fallback constructs; retired exports resolve) and is GREEN after the cut; the no-credentials characterization test stayed green throughout. Gates: blocking suite 1028 passed / 0 failed (1053 baseline - 30 adapter-only unit tests + 5 new); ruff clean; catalogue 52/117, no regressions; wheel modules 333 -> 320. Verified on the real surface: orchestrator run examples/supported/01_hello_filesystem.yaml exits 0 and writes output/greeting.txt.
The README's legacy-suite numbers predated d452c2a, and the first cut missed a handful of test files that can only ever exercise the retired adapters. A fresh full legacy run (506 failed / 1708 passed / 111 skipped / 93 errors, 14m50s) is now classified by exception signature and posted on #354 in triage order; the residual after these deletions is 500 failed / 93 errors / 1707 passed / 109 skipped. Deleted, each exclusively testing modules retired in d452c2a: - tests/local/test_ollama_local.py -- every test constructs the retired integrations.ollama_model.OllamaModel (6 of the import-class failures) - tests/local/test_simple_ollama.py -- its one test constructs the retired OllamaModel - tests/models/test_integration.py -- builds only the retired models.registry.ModelRegistry plus the retired openai/anthropic/local providers; already module-skipped as "uses removed providers" (#426) - tests/models/validate_integration.py -- uncollected script (not test_*.py); top-level imports of four retired modules make it unimportable dead code - tests/test_creative_image_pipeline.py -- every test requires the retired models.openai_model.OpenAIModel and a real OpenAI key, and it imports create_test_orchestrator/TestModel/TestProvider, which no longer exist in tests/test_infrastructure.py Kept deliberately: tests that mention retired names but still cover shipped code (tests/models/test_selection.py, test_performance.py, tests/integration/test_real_models.py et al.) -- they fail at collection or per-test and are tracked in the #354 classification. Blocking layer unchanged: 1028 passed / 0 failed. README legacy numbers updated to the post-retirement re-baseline; prose pass left to the docs wave.
The second supported provider, modeled on the Dartmouth adapter: the router at router.huggingface.co/v1 is OpenAI-compatible, so the adapter speaks it with plain aiohttp -- no huggingface_hub dependency, per the ADR 0001 dependency policy. Endpoint shapes were verified against the live router and current HF docs, not memory: Bearer HF_TOKEN auth, GET /v1/models carrying per-provider status/pricing-per-million-tokens/ is_free/context_length/throughput, and OpenAI-style error envelopes. Cost policy mirrors Dartmouth with one HF-specific twist: nothing on the router is permanently zero-cost, so a model is free only while a live provider carries an is_free promo or explicit zero pricing, and a free model is pinned to that provider on the wire (model_id:provider) because an unpinned request routes :fastest and could bill the account. Unpriced entries are unknown, and unknown is treated as paid; estimate_cost on an unknown-priced model raises rather than reporting a confidently wrong $0.00. Paid usage needs ORCHESTRATOR_ALLOW_PAID_MODELS=1, the same lever as Dartmouth. Both Dartmouth lessons are carried in. Flapping free routes skip rather than fail per-test, while the new live-huggingface CI job hard-fails on a missing HF_TOKEN secret and greps for at least one real pass. A reasoning model that burns its whole budget thinking is not a success: content-absent plus reasoning_content raises ReasoningTruncated (observed live: 31 of 32 completion tokens were reasoning). Two failure modes found by real probing are classified, not guessed: 402 (monthly credits depleted) is PaymentRequired and 429 is RateLimited -- both account-level, so generate_free lets them propagate instead of hammering the quota. Credential resolution mirrors dartmouth_credentials: HF_TOKEN, then ~/.orchestrator/.env, then the HF CLI's own ~/.cache/huggingface/token store. Verified live: 8/8 live tests passed against the real router with a valid token, including generation through the fallback chain; hermetic suite is 102 unit/contract tests with deterministic fakes.
Lane F: provider + credentials + model over plain aiohttp, live job live-huggingface, 105 new blocking tests. Verified: blocking 1133/0, CI-scoped ruff clean, import clean, catalogue 52/117, live tests 5 passed / 3 skipped (402 credit-depleted skips -- the designed flap behavior).
The design review on #485 leans on four quantitative claims. They were argued from simulations that lived only in a scratch directory, so nobody could re-run them, and nobody could tell which numbers were measured and which were guessed. All of them were guessed: no live model was reachable when the review was written. `scripts/simulations/` makes them reproducible. Each script states its `ASSUMPTIONS` at module scope, and the README says how to measure each one and which conclusions survive replacing it. Standard library only; nothing under `src/` imports them. The tests assert the *structure*, not the tables: m = b*f < 1 for termination, a throughput ceiling of 1/critical-section that fleet size cannot move, quadratic-vs-linear summary churn, and a 1-rho ceiling on same-family review. A golden-output test would fail on every cosmetic edit and pass on a silently wrong model, which is the wrong way round. Writing those tests already found one conflation in the review: the recursion converging (r < 1) and the recursion fitting under a depth cap are different claims. r=0.9 converges and needs 29 levels. The corollary is new and useful -- top-level fidelity is payload/N whatever r is, so r only buys fewer lossy hops, and fewer harder compressions beat more gentle ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRFg27WG13iijK313jCpqi
The three design reviews on #485 all reasoned about mechanisms in isolation. This adds the other half: one small system that actually runs every mechanism the issue proposes, plus the two measurements the reviews had to guess at. scripts/prototypes/minikernel/ -- ~1,600 lines, stdlib only, nothing under src/ imports it and it imports nothing from src/: store.py one substrate; scratchpad / insights / context / tool history are queries over it, not four stores ir.py typed plan IR (no goto), validator, Authority, Budget capabilities.py one lifecycle for tools, skills and plans + bug triage library.py solved-problem library, two-key retrieval, dead ends review.py separation of duty, frozen criteria, evidential gate planner.py StubPlanner (deterministic) and LLMPlanner (real) runtime.py durable executor, crash-resume, budgets, bus, admission run_scenarios.py: 55 end-to-end checks (real SQLite, real SIGKILL). tests/test_minikernel.py: 57 tests, no mocks. Six kernel changes were forced by scenarios failing, each a design finding: - the library must store NEGATIVE results, or the one regime where learning matters is the one regime where learning cannot start - an escalating sibling must not cancel the siblings that could still run - a wildcard signature defeats the two-key match, so untyped solutions are never published - budget exhaustion must not count as evidence against a cached plan - a planner must be a function of its inputs for a cache over it to mean anything - `atomic` must be a checked claim, not a label the planner assigns Measured, not assumed (scripts/prototypes/measurements/): measure_ambiguity.py f over 16 real problems x 4 library tiers. The library drives f down but drives b up, and m is their product, so m falls 2.50 -> 0.88 and then rises again to 1.88 at 30 capabilities. probe_optimism.py 35-44% of steps a planner calls "atomic" cannot be done by the capability it names. The strongest planner's apparent m of 0.12 is really 1.62. Also fixes the blocking suite, which failed at COLLECTION under the locked pytest 9.1.1: builtin marks applied through a module-level `pytestmark` list are now validated against the markers table, so `filterwarnings` has to be declared there. 1166 passed, 11 skipped after the fix. Adds the six round-2 simulations left uncommitted in scripts/simulations/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019k47Guj8QzcFRvuRVLutcM
This was referenced Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backs the design review on #485 (comment) with reproducible code.
Four models under
scripts/simulations/, standard library only, nothing undersrc/imports them:decomposition_branching.pym = b·f < 1. At ~10 steps/pipeline, <1 step in 10 may be ambiguous, at every level.scratchpad_contention.py3600/critical_sectionsteps/hour (450/hr at 8s), invariant to fleet size.context_recursion.pyr**depth ≈ 0.0016at the top — a navigation structure, not a retrieval structure.red_team_gate.pyp_detect = 0.5, only 45% of clean verdicts are truly clean. Same-family review is capped at1-ρ.Every number rests on a guess. No live model was reachable when the review was written (
HF_TOKEN401s on/v1/chat/completions; no Dartmouth key). Each script states itsASSUMPTIONSat module scope, and the README says how to measure each one and which conclusions survive replacing it.tests/test_design_simulations.py(33 tests,unit, ~1.4s) asserts the structural claims rather than golden table text — a golden test would fail on every cosmetic edit and pass on a silently wrong model.Writing those tests already found a conflation in the review: converging (
r < 1) and fitting under a depth cap are different claims —r = 0.9converges and needs 29 levels. The corollary is new: top-level fidelity ispayload/Nwhateverris, soronly buys fewer lossy hops, and fewer harder compressions beat more gentle ones.Checks:
ruff checkclean on the CI rule set and the default set; blocking suite 1166 passed, 11 skipped, 0 failed.🤖 Generated with Claude Code
https://claude.ai/code/session_01JRFg27WG13iijK313jCpqi