Skip to content

test: wait on real conditions in the integration-kit tests (Windows flake) - #18

Merged
sanctrl merged 1 commit into
mainfrom
fix/windows-released-turn-flake
Nov 11, 2026
Merged

sanctrl merged 1 commit into
mainfrom
fix/windows-released-turn-flake

Conversation

@sanctrl

@sanctrl sanctrl commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Fixes a flake that hit test_a_waiting_turn_that_is_released_never_starts once, on windows-latest with Python 3.11 (run 42491213799: assert [] == ['c1']). The fault was in the test; the kit is unchanged.

Why it flaked: the test fired the coalescer, slept 20 ms, then asserted that c1's turn had started. That cycle's model call is sync, so it runs in a worker thread through asyncio.to_thread, on a thread pool created fresh for each test. On Windows before Python 3.13, the event loop's clock moves in 15.6 ms steps (gh-88494), so asyncio.sleep(0.02) can return almost at once. A slow runner's thread start then loses the race.

Now: the test waits on the real conditions.

  • It waits for turns_waiting == 1 before releasing c2.
  • It asserts turns_waiting == 0 right after the release, the same check the TypeScript twin makes.
  • It waits for turns_active == 0 before checking that c2 never started.

Ten sibling tests had the same habit, a fixed sleep standing in for "the work finished" followed by an assert. They now wait through _eventually, which takes a label so a timeout names what it was waiting for. Sleeps before "nothing happened" asserts are unchanged, since a slow machine can't make those fail.

Proof, locally. I simulated the Windows clock and slow thread starts with a scratch pytest plugin (not committed), on top of a fully loaded CPU:

Old test New test
Windows clock + loaded CPU 74 and 82 of 1000 failed, at CI's exact line 2000 / 2000 passed
30 ms thread delay + loaded CPU 100 / 100 failed 500 / 500 passed
  • All 14 converted tests, 300 runs each under both simulations: 4200 / 4200 passed.
  • A mutation check: with the kit's "a released turn is dropped" guards removed, the new test fails every time.
  • The full gate on 3.9, 3.11 and 3.13: ruff and mypy clean; pytest 714 passed, 1 skipped (the live smoke test).

Test only; no SDK change, no release needed.

…eleased-turn flake)

test_a_waiting_turn_that_is_released_never_starts failed once on
windows-latest / Python 3.11 (run 42491213799): assert [] == ['c1'] at the
first check. The test fired the coalescer's timer and slept 20 ms before
asserting c1's turn had started, but that cycle's gate call runs in a
worker thread (a sync call_model goes through asyncio.to_thread, on a
fresh executor). On Windows before Python 3.13 the event loop's clock is
GetTickCount64 (15.6 ms ticks, gh-88494), so asyncio.sleep(0.02) returns
at the next tick: after anywhere from ~0 to 15.6 ms. A slow runner's
thread start loses that race. The kit is correct; the test assumed timing.

Now it waits for the real conditions through _eventually (5 s backstop,
fails naming what it waited for): turns_waiting == 1 before the release,
turns_active == 0 after c1 ends. It also asserts the released admission
is dropped at once (turns_waiting == 0), as the TypeScript twin does.

The same pattern (a fixed sleep standing in for "the turn's end was
processed" / "the waiting turn started" / "the frames were handled"
before a positive assert) is replaced in ten sibling tests in the file.
Sleeps followed by loop.idle() on an already scheduled cycle, deliberate
durations, and sleeps before negative asserts are left as they were.
@sanctrl
sanctrl merged commit f72c4db into main Nov 11, 2026
11 checks passed
@sanctrl
sanctrl deleted the fix/windows-released-turn-flake branch November 11, 2026 08:39
sanctrl added a commit that referenced this pull request Nov 11, 2026
…, a kit slot-order fix, a whole-session network guard, and a weekly fixture check (#19)

* docs: live docs links, and SystemAgentProtectedError no longer calls system agents the support channel

The README links, the package docstring and the PyPI "Documentation" URL
pointed at agentchat.me/docs/sdk/python, /docs/realtime and /docs, which
all answer 404. They now point at the live pages: the SDK guide
(docs.agentchat.me/sdks/python), the integration contract v1
(agentchat.me/contract/v1/: socket frames, tray, wake payloads,
conformance fixtures) and the REST API reference.

The SystemAgentProtectedError docstring said system agents are "the
support channel" and to send them support requests. ChatFather is
retired: a direct message gets only a pointer to the manual, and the
server's block/report refusal carries the same pointer. The docstring now
says when the server answers SYSTEM_AGENT_PROTECTED (block or report a
system agent; delete or rotate the key of a system account through the
public API) and points at the manual instead of a DM.

* test: the network guard covers collection, session fixtures and the Windows proactor connect

The guard was a function-scoped autouse fixture, so it watched only while
a test ran. Two gaps:

- Collection (a test module that connects at import) and session- or
  module-scoped fixtures ran outside it: they set up before the
  function fixture patched anything.
- On Windows' default proactor loop, asyncio connects through ConnectEx
  and never calls socket.connect, and a literal IP address skips
  getaddrinfo, so asyncio.open_connection("203.0.113.7", 443) went out.

Now the guard is installed in pytest_configure and removed in
pytest_unconfigure, so it is on for the whole session. It also wraps
asyncio's BaseProactorEventLoop.sock_connect (importable on every
platform; it refuses before touching the loop). An attempt while a test
runs, including a wider-scoped fixture set up for it, fails that test as
before. An attempt outside any test (collection, a fixture's final
teardown) sets the exit status to "tests failed" and the summary names
it. Live tests are exempt for their whole run, fixtures included.

New tests, each failing against the old guard: the proactor door (an
unbound first step, so it runs on every OS), a swallowed lookup at
collection, a session fixture's setup and final teardown, and on Windows
only, a real literal-IP open_connection on the default loop plus a
loopback round trip through the proactor (in a subprocess: asyncio.run in
an in-process inner session leaves the main thread with no current loop).
The live-exemption test now checks behaviour (a numeric-host lookup,
answered on this machine) instead of function names.

Not covered, and named in the docstring: subprocesses, UDP sendto without
a connect, and direct use of the C _socket module. The SDK does none.

* ci: the contract-fixture comparison also runs weekly, and a scheduled failure opens an issue

The comparison of conformance/v1/manifest.sha256 with the manifest
agentchat.me publishes ran only on push and pull request, so a contract
change on the server was flagged here only at the next push.

weekly.yml (Mondays 06:29 UTC, and Run workflow by hand) runs that
comparison alone, not the 3 x 3 test matrix, through
scripts/check-contract-fixtures.sh, which CI's step now calls too. Its
report job (scheduled runs only, the one job with issues: write, through
GITHUB_TOKEN) opens "Weekly checks failed" or comments on the open one,
and a passing scheduled run comments and closes it. The repo had no other
alert path (no webhooks, no notify steps).

Its own file on purpose: GitHub turns scheduled workflows off in a public
repository after 60 days without activity, and that must never stop CI.

* test: wait for the condition, never a fixed sleep (realtime, conformance engine, kit)

The #18 pattern, applied to the rest of the suite. A fixed sleep before a
positive assert races the work it stands in for: a gap, backoff, handshake,
stability or liveness timer, a handler running as its own task, a worker
thread. On Windows before Python 3.13 the event loop's clock is
GetTickCount64 (15.6 ms ticks) and the loop runs a timer once it is due
within one tick, so asyncio.sleep(0.02) can return after almost no time
and asyncio.sleep(0.05) after less than 20 ms. A slow runner (a late thread
start, a starved loop) then loses the race. CI runs windows-latest on 3.9
and 3.11, where this holds.

tests/_wait.py adds eventually(predicate, what): it polls from the calling
task (no wait_for wrapper, which runs its coroutine as a separate task
before 3.12, so a predicate may look at the loop's tasks) and fails after
5 s naming what it waited for. Predicates use >= so the exact assert that
follows still shows the list when something extra arrives.

Converted (23 sites):
- test_realtime: the three gap-fill tests, the auto drain's cursor ack
  (a 20 x sleep(0.01) poll whose budget collapses on a coarse clock), the
  three reconnect-after-close tests, the stability window.
- test_realtime_engine: the two gap-fill ack tests, the two lapsed-gap
  handler tests, seq 6's own gap window, b's reconnect. The "not a crash"
  test now waits until the gap task and every handler finished
  (no _bg_tasks, no _lanes), so a late KeyError or a second handover of
  seq 4 is still caught: checked by making the gap task raise.
- test_realtime_gate_v1: the hung handler's 50 ms bound, the queue back
  to half, seq-order starts, the re-offer past the adoption window, the
  idle watchdog's close. The heartbeat test kept a quiet socket alive with
  beats every 60 ms against a 150 ms window: 90 ms of slack, minus up to
  two clock ticks. Now 50 ms beats against 200 ms, eight times: 150 ms of
  slack, and the beats still outlast the window twice over.
- test_realtime_resilience: the second answered probe; the disconnect()
  from a state handler (waits until every task is gone).
- test_conformance_engine: the wait_handler op (100 x sleep(0.001), the
  same collapsing budget) waits up to 5 s for the handler to enter.
- test_integration_kit: the sync model call past its deadline. The cycle
  stops waiting at 30 ms whether or not the worker thread has started the
  call, but the test asserted the thread's handle after a 50 ms sleep: the
  #18 mechanism. It now waits for the call to have started.

Left as they were: deliberate durations (a slow handler, the adoption
clock test with its documented 150 ms margins, the liveness push cadence,
the conformance gap_window_lapses op, whose gap timer is armed first and
fires 150 ms earlier on the same loop); sleeps before negative asserts
(nothing reconnects, nothing is recorded, no gap fires), several already
backed by other_tasks() or ControlledSleep; sleeps followed by
loop.idle() on a cycle already scheduled; the fake clock in
test_conformance_v1; the 300 s webhook tolerance window; the
disconnect-promptness bound (0.5 s against ~50 ms of work) and the
dispose-grace bound (1 s of slack).

* fix(kit): a turn gated while a freed slot is handed on queues behind the waiting ones

test_at_most_max_concurrent_turns_run_and_the_rest_wait_for_a_slot failed
about 1 run in 20 on 3.9 (1 in 10 under a 15.6 ms clock) with
['c1', 'c2', 'c1'] instead of ['c1', 'c2', 'c3'], on main's tests too.
The kit, not the test: when a turn ends, _slot_freed frees its slot and
hands it on through _pump, which runs as its own task. _launch checked only
that a slot was free, so a cycle that ran before the pump (here the one
c1's failed turn woke, gating c1 again) took the slot ahead of c3, which
had been waiting for it. The TypeScript kit's pump takes the slot
synchronously and has no such window.

_launch now queues when turns are already waiting, so they start first,
in order, as the docstring says. A new test holds the hand-on open and
checks that the re-gated c1 waits behind c2 (it starts first without the
fix). The slot tests ran 120 times (3.9 and 3.13, normal and 15.6 ms
clock) without a failure.

* ci(weekly): curl timeouts, any directory, and a timed-out run is reported too

- check-contract-fixtures.sh: curl gets --connect-timeout 10 and
  --max-time 30 (a stalled server could hold CI's job for hours), runs
  from any directory, and tells a missing file (diff exit 2) apart from a
  mismatch.
- report: runs for scheduled and manual runs on main, unless the run was
  cancelled by hand. Anything but a pass is reported, so a job that timed
  out opens the issue too; only a pass closes it.

* ci(temporary): run weekly.yml on this pull request once, to prove its job on GitHub's runners

* test(conformance): copy texts.sample.json and manifest.sha256 from the published contract (manual 2.3.1)

The server now publishes the manual as 2.3.1, so the contract's
texts.sample.json carries the new manual version and sha256, and the
manifest lists the file's new hash. Both files are copied byte for byte
from the published contract (core is canonical); every file in
conformance/v1 matches the published manifest, and the manifest equals
https://agentchat.me/contract/v1/conformance/manifest.sha256. No test
reads the manual's version or hash from this fixture, only the hash's
format.

* review fixes: a refused admission hands its slot on; two waits keep their bounds

- receive_loop._launch: a refused admission now frees its slot through
  _slot_freed, which hands it to a waiting turn, instead of only
  discarding it. With waiting turns first in order, a free slot left with
  turns waiting and no pump scheduled would stall every later turn. Not
  reachable today (one cycle at a time, and _launch admits only when none
  wait), but a randomized probe with overlapping callers stalled without
  it and ran clean with it. The docstring and CHANGELOG no longer claim
  the TypeScript kit is FIFO everywhere: its synchronous hand-on covers a
  turn's end, not a refusal inside its pump.
- test_realtime_gate_v1: the hung-handler and idle-watchdog tests now
  also bound the wait (under 1 s), so a regression to the defaults
  (minutes, 100 s) still fails them as the old fixed sleeps did.
- test_network_guard: the in-process test checks that all four doors are
  put back, not two.
- SystemAgentProtectedError: "at most a pointer" (one per peer per day).

* ci(weekly): a step timeout, the issue found without the search index, and no temporary trigger

- The check step has its own 3-minute timeout: a step timeout fails the
  step, so the report always reads "failure" (a job timeout may read as
  cancelled).
- The open issue is found from the open-issue list (limit 200, exact
  title), not the search index, which can lag and open a duplicate.
- The temporary pull_request trigger is gone; the job ran green on this
  PR, against the live manifest after the 2.3.1 copy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant