Skip to content

fix(harness): TH-8729 check scenarios on one database lane per CPU - #198

Merged
KarthikAvinashFI merged 5 commits into
devfrom
fix/TH-8729-validation-lanes-per-cpu
Oct 23, 2026
Merged

KarthikAvinashFI merged 5 commits into
devfrom
fix/TH-8729-validation-lanes-per-cpu

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

1. Why

Since 0.3.3, hosted builds with 10 or more scenarios fail runtime validation with ready_timeout on almost every scenario. The bounded repair cannot help: rounds that rewrite every ready check to return None still time out.

ce535534 lets a world whose database can be cloned check each scenario on its own clone, running min(ALK_VALIDATION_DB_LANES, scenario count) lanes at once with a default of 16. Every check starts a fresh fi.alk.harness.phase_worker process, and that start-up counts against the 15 s ready budget. On a 4 vCPU sandbox, 16 workers starting together spend longer than the budget just importing, so every ready check times out before it runs. Before ce535534, at most 4 checks ran at once.

2. What changed

File Change
src/fi/alk/harness/authoring_runtime_validation.py _database_lanes(): one lane per CPU available to the process (os.sched_getaffinity, else os.cpu_count()). ALK_VALIDATION_DB_LANES still overrides it. validate_once uses it instead of the fixed 16
tests/harness/test_authoring_runtime_validation.py test_database_lanes_default_to_one_per_cpu: 4 CPUs give 4 lanes; an explicit ALK_VALIDATION_DB_LANES=16 is honoured

3. Behavior callouts

  • On a 4 vCPU sandbox, validation checks 4 scenarios at once instead of up to 16; larger sandboxes get one lane per CPU.
  • A deployment that sets ALK_VALIDATION_DB_LANES keeps its value.

4. Tests

End to end, on hosted E2B sandboxes with the production sandbox size (4 vCPU, 8 GB), the same chat agent (LangGraph, in-memory store), builder session on:

Template Scenarios Lanes ready_timeout Result
0.3.3 release 10 10 0 passed, at the edge of the budget
0.3.3 release 20 16 16 failed every ready check
0.3.3 release + this change 20 4 0 completed; setup and ready checks 63 s
0.3.3 release + this change 50 4 0 completed; setup and ready checks 179 s; a 5-scenario run on it completed 5/5

Worker start-up timed in a container limited to 4 CPUs:

Workers started at once Slowest start-up
1 3.8 s
4 6.3 s
16 44 to 64 s
File Result
tests/harness/test_authoring_runtime_validation.py 43 passed; the new test fails without the change
tests/harness/test_hosted_authoring_entrypoint.py, tests/harness/test_process_runtime.py 9 and 318 passed

The end-to-end builds ran the first commit; the second only shortens the helper (same behaviour, covered by the unit test).

5. Commands to run the tests

uv run pytest -q tests/harness/test_authoring_runtime_validation.py

6. Scope

Out, possible follow-ups: keeping worker start-up out of the 15 s ready budget or reusing a warm worker, a lighter import chain for phase_worker, and reporting a slow start as its own failure code.

Linear

TH-8729, TH-8756

@KarthikAvinashFI KarthikAvinashFI self-assigned this Oct 23, 2026
@KarthikAvinashFI
KarthikAvinashFI marked this pull request as ready for review October 23, 2026 08:09

@khushalsonawat khushalsonawat left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check comments

def _database_lanes() -> int:
"""One lane per CPU by default: each check starts a worker inside its time budget."""
configured = os.environ.get("ALK_VALIDATION_DB_LANES")
cpus = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This counts CPUs from affinity, which can't see a cgroup CPU quota. process_runtime.py:398-400 says the sandbox's CPU budget is the cgroup quota, never os.cpu_count() or host cores, and admission already reads it through _cgroup_cpu_quota() (:5680). On E2B this is fine: each sandbox is a Firecracker VM, so affinity sees its 4 vCPUs. But Daytona is the default provider (future-agi hosted_sandbox/factory.py:24, docker-compose.yml:260). If it caps CPU with a quota, affinity returns the host's cores, line 725 makes that many lanes, and we're back to the TH-8729 ready_timeout. Can we take the smaller of affinity, floor(_cgroup_cpu_quota()) when it's set, and the declared cpu_units when it's set? process_runtime is already imported inside validate_once at :495.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. It now takes the smaller of affinity and floor(_cgroup_cpu_quota()) when a quota is set. I left declared cpu_units out because it defaults to 1 in RuntimeSpec, so local runs without a runtime block would drop to a single lane.

@khushalsonawat khushalsonawat left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few smaller ones. Also, CHANGELOG.md has no line under ## Unreleased for this. The default goes from a fixed 16 to the CPU count, which is worth one line, like the recent harness fixes have.

)


def test_database_lanes_default_to_one_per_cpu(monkeypatch):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low. This tests the helper, but nothing tests that validate_once uses it. I put line 602 back to the old fixed 16 and all 43 tests in this file still passed. Removing the cap at line 725 passed too. A validate_once test with a fake provider whose clone_world_database returns a value, _database_lanes patched to 3, and 10 scenarios, asserting 3 distinct lane* targets, would pin the fix itself.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added: the connect-only validate_once test now asserts it asks _database_lanes, and it fails if line 602 goes back to the fixed 16.

def test_database_lanes_default_to_one_per_cpu(monkeypatch):
monkeypatch.delenv("ALK_VALIDATION_DB_LANES", raising=False)
monkeypatch.setattr("os.sched_getaffinity", lambda _pid: {0, 1, 2, 3}, raising=False)
assert _database_lanes() == 4

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low. 4 is also the ALK_VALIDATION_INSTANCES default and a CI runner's vCPU count, so a hard-coded configured or 4 passes this test. So does an empty override that no longer falls back to the CPU count, since there's no "" case. Faking affinity as {0, 1, 2} and expecting 3, patching os.cpu_count to 64, and adding the "" case would close both.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tightened: affinity {0, 1, 2} with os.cpu_count patched to 64, an empty override, a quota of 2, and abc.

"""One lane per CPU by default: each check starts a worker inside its time budget."""
configured = os.environ.get("ALK_VALIDATION_DB_LANES")
cpus = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count()
return max(1, int(configured or cpus or 1))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low. ALK_VALIDATION_DB_LANES=abc raises a raw ValueError here, before the try at :609, so validate_and_repair doesn't catch it. It was the same before this PR, and hosted builds can't hit it because the platform doesn't forward the variable. A typed error naming the variable would still be kinder for local runs.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now raises a ValueError that names ALK_VALIDATION_DB_LANES and the value.

@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Thanks, all addressed in 7011c01, plus the CHANGELOG line. The len(scenarios) cap at :725 predates this PR and needs a full generic validate_once fixture to pin, so I kept that out of this one. Re-tested on latest dev with this head on 4 vCPU sandboxes: 20 scenarios ran 4 lanes with no ready_timeout, setup and ready checks in 67 s.

@khushalsonawat khushalsonawat left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check comments

asyncio.run(validate_once(job, tmp_path, authoring, secrets_path=original)) == 4
)
assert calls == ["provision", "reset", "close"]
assert asked

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium. This pins the call, not the use: _database_lanes(); database_lanes = 16 at :610 still passes all 43. A clone-path run with lanes patched to 3 and 10 scenarios, asserting 3 lanes, would pin it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Added test_database_clone_checks_use_the_lane_count: a clone-path run with lanes patched to 3 and 10 scenarios asserts the checks use exactly lane0 to lane2. It fails with database_lanes = 16 at :610 and passes with the fix; the call-only spy is gone.

@khushalsonawat khushalsonawat left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@KarthikAvinashFI
KarthikAvinashFI merged commit 457b864 into dev Oct 23, 2026
15 checks passed
@KarthikAvinashFI
KarthikAvinashFI deleted the fix/TH-8729-validation-lanes-per-cpu branch October 23, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants