Repository navigation
fix(harness): TH-8729 check scenarios on one database lane per CPU - #198
Conversation
…start-up fits the ready budget
| def _database_lanes() -> int: | ||
| """One lane per CPU by default: each check starts a worker inside its time budget.""" | ||
| configured = os.environ.get("ALK_VALIDATION_DB_LANES") | ||
| cpus = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count() |
There was a problem hiding this comment.
This counts CPUs from affinity, which can't see a cgroup CPU quota. process_runtime.py:398-400 says the sandbox's CPU budget is the cgroup quota, never os.cpu_count() or host cores, and admission already reads it through _cgroup_cpu_quota() (:5680). On E2B this is fine: each sandbox is a Firecracker VM, so affinity sees its 4 vCPUs. But Daytona is the default provider (future-agi hosted_sandbox/factory.py:24, docker-compose.yml:260). If it caps CPU with a quota, affinity returns the host's cores, line 725 makes that many lanes, and we're back to the TH-8729 ready_timeout. Can we take the smaller of affinity, floor(_cgroup_cpu_quota()) when it's set, and the declared cpu_units when it's set? process_runtime is already imported inside validate_once at :495.
There was a problem hiding this comment.
Good catch. It now takes the smaller of affinity and floor(_cgroup_cpu_quota()) when a quota is set. I left declared cpu_units out because it defaults to 1 in RuntimeSpec, so local runs without a runtime block would drop to a single lane.
khushalsonawat
left a comment
There was a problem hiding this comment.
A few smaller ones. Also, CHANGELOG.md has no line under ## Unreleased for this. The default goes from a fixed 16 to the CPU count, which is worth one line, like the recent harness fixes have.
| ) | ||
|
|
||
|
|
||
| def test_database_lanes_default_to_one_per_cpu(monkeypatch): |
There was a problem hiding this comment.
Low. This tests the helper, but nothing tests that validate_once uses it. I put line 602 back to the old fixed 16 and all 43 tests in this file still passed. Removing the cap at line 725 passed too. A validate_once test with a fake provider whose clone_world_database returns a value, _database_lanes patched to 3, and 10 scenarios, asserting 3 distinct lane* targets, would pin the fix itself.
There was a problem hiding this comment.
Added: the connect-only validate_once test now asserts it asks _database_lanes, and it fails if line 602 goes back to the fixed 16.
| def test_database_lanes_default_to_one_per_cpu(monkeypatch): | ||
| monkeypatch.delenv("ALK_VALIDATION_DB_LANES", raising=False) | ||
| monkeypatch.setattr("os.sched_getaffinity", lambda _pid: {0, 1, 2, 3}, raising=False) | ||
| assert _database_lanes() == 4 |
There was a problem hiding this comment.
Low. 4 is also the ALK_VALIDATION_INSTANCES default and a CI runner's vCPU count, so a hard-coded configured or 4 passes this test. So does an empty override that no longer falls back to the CPU count, since there's no "" case. Faking affinity as {0, 1, 2} and expecting 3, patching os.cpu_count to 64, and adding the "" case would close both.
There was a problem hiding this comment.
Tightened: affinity {0, 1, 2} with os.cpu_count patched to 64, an empty override, a quota of 2, and abc.
| """One lane per CPU by default: each check starts a worker inside its time budget.""" | ||
| configured = os.environ.get("ALK_VALIDATION_DB_LANES") | ||
| cpus = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count() | ||
| return max(1, int(configured or cpus or 1)) |
There was a problem hiding this comment.
Low. ALK_VALIDATION_DB_LANES=abc raises a raw ValueError here, before the try at :609, so validate_and_repair doesn't catch it. It was the same before this PR, and hosted builds can't hit it because the platform doesn't forward the variable. A typed error naming the variable would still be kinder for local runs.
There was a problem hiding this comment.
Now raises a ValueError that names ALK_VALIDATION_DB_LANES and the value.
… bad lane override
|
Thanks, all addressed in 7011c01, plus the CHANGELOG line. The |
| asyncio.run(validate_once(job, tmp_path, authoring, secrets_path=original)) == 4 | ||
| ) | ||
| assert calls == ["provision", "reset", "close"] | ||
| assert asked |
There was a problem hiding this comment.
Medium. This pins the call, not the use: _database_lanes(); database_lanes = 16 at :610 still passes all 43. A clone-path run with lanes patched to 3 and 10 scenarios, asserting 3 lanes, would pin it.
There was a problem hiding this comment.
Agreed. Added test_database_clone_checks_use_the_lane_count: a clone-path run with lanes patched to 3 and 10 scenarios asserts the checks use exactly lane0 to lane2. It fails with database_lanes = 16 at :610 and passes with the fix; the call-only spy is gone.
1. Why
Since 0.3.3, hosted builds with 10 or more scenarios fail runtime validation with
ready_timeouton almost every scenario. The bounded repair cannot help: rounds that rewrite every ready check toreturn Nonestill time out.ce535534lets a world whose database can be cloned check each scenario on its own clone, runningmin(ALK_VALIDATION_DB_LANES, scenario count)lanes at once with a default of 16. Every check starts a freshfi.alk.harness.phase_workerprocess, and that start-up counts against the 15 s ready budget. On a 4 vCPU sandbox, 16 workers starting together spend longer than the budget just importing, so every ready check times out before it runs. Beforece535534, at most 4 checks ran at once.2. What changed
src/fi/alk/harness/authoring_runtime_validation.py_database_lanes(): one lane per CPU available to the process (os.sched_getaffinity, elseos.cpu_count()).ALK_VALIDATION_DB_LANESstill overrides it.validate_onceuses it instead of the fixed 16tests/harness/test_authoring_runtime_validation.pytest_database_lanes_default_to_one_per_cpu: 4 CPUs give 4 lanes; an explicitALK_VALIDATION_DB_LANES=16is honoured3. Behavior callouts
ALK_VALIDATION_DB_LANESkeeps its value.4. Tests
End to end, on hosted E2B sandboxes with the production sandbox size (4 vCPU, 8 GB), the same chat agent (LangGraph, in-memory store), builder session on:
ready_timeoutWorker start-up timed in a container limited to 4 CPUs:
tests/harness/test_authoring_runtime_validation.pytests/harness/test_hosted_authoring_entrypoint.py,tests/harness/test_process_runtime.pyThe end-to-end builds ran the first commit; the second only shortens the helper (same behaviour, covered by the unit test).
5. Commands to run the tests
6. Scope
Out, possible follow-ups: keeping worker start-up out of the 15 s ready budget or reusing a warm worker, a lighter import chain for
phase_worker, and reporting a slow start as its own failure code.Linear
TH-8729, TH-8756