Repository navigation
test: wait on real conditions in the integration-kit tests (Windows flake) - #18
Merged
Merged
Conversation
…eleased-turn flake) test_a_waiting_turn_that_is_released_never_starts failed once on windows-latest / Python 3.11 (run 42491213799): assert [] == ['c1'] at the first check. The test fired the coalescer's timer and slept 20 ms before asserting c1's turn had started, but that cycle's gate call runs in a worker thread (a sync call_model goes through asyncio.to_thread, on a fresh executor). On Windows before Python 3.13 the event loop's clock is GetTickCount64 (15.6 ms ticks, gh-88494), so asyncio.sleep(0.02) returns at the next tick: after anywhere from ~0 to 15.6 ms. A slow runner's thread start loses that race. The kit is correct; the test assumed timing. Now it waits for the real conditions through _eventually (5 s backstop, fails naming what it waited for): turns_waiting == 1 before the release, turns_active == 0 after c1 ends. It also asserts the released admission is dropped at once (turns_waiting == 0), as the TypeScript twin does. The same pattern (a fixed sleep standing in for "the turn's end was processed" / "the waiting turn started" / "the frames were handled" before a positive assert) is replaced in ten sibling tests in the file. Sleeps followed by loop.idle() on an already scheduled cycle, deliberate durations, and sleeps before negative asserts are left as they were.
4 of 5 tasks
sanctrl
added a commit
that referenced
this pull request
Nov 11, 2026
…, a kit slot-order fix, a whole-session network guard, and a weekly fixture check (#19) * docs: live docs links, and SystemAgentProtectedError no longer calls system agents the support channel The README links, the package docstring and the PyPI "Documentation" URL pointed at agentchat.me/docs/sdk/python, /docs/realtime and /docs, which all answer 404. They now point at the live pages: the SDK guide (docs.agentchat.me/sdks/python), the integration contract v1 (agentchat.me/contract/v1/: socket frames, tray, wake payloads, conformance fixtures) and the REST API reference. The SystemAgentProtectedError docstring said system agents are "the support channel" and to send them support requests. ChatFather is retired: a direct message gets only a pointer to the manual, and the server's block/report refusal carries the same pointer. The docstring now says when the server answers SYSTEM_AGENT_PROTECTED (block or report a system agent; delete or rotate the key of a system account through the public API) and points at the manual instead of a DM. * test: the network guard covers collection, session fixtures and the Windows proactor connect The guard was a function-scoped autouse fixture, so it watched only while a test ran. Two gaps: - Collection (a test module that connects at import) and session- or module-scoped fixtures ran outside it: they set up before the function fixture patched anything. - On Windows' default proactor loop, asyncio connects through ConnectEx and never calls socket.connect, and a literal IP address skips getaddrinfo, so asyncio.open_connection("203.0.113.7", 443) went out. Now the guard is installed in pytest_configure and removed in pytest_unconfigure, so it is on for the whole session. It also wraps asyncio's BaseProactorEventLoop.sock_connect (importable on every platform; it refuses before touching the loop). An attempt while a test runs, including a wider-scoped fixture set up for it, fails that test as before. An attempt outside any test (collection, a fixture's final teardown) sets the exit status to "tests failed" and the summary names it. Live tests are exempt for their whole run, fixtures included. New tests, each failing against the old guard: the proactor door (an unbound first step, so it runs on every OS), a swallowed lookup at collection, a session fixture's setup and final teardown, and on Windows only, a real literal-IP open_connection on the default loop plus a loopback round trip through the proactor (in a subprocess: asyncio.run in an in-process inner session leaves the main thread with no current loop). The live-exemption test now checks behaviour (a numeric-host lookup, answered on this machine) instead of function names. Not covered, and named in the docstring: subprocesses, UDP sendto without a connect, and direct use of the C _socket module. The SDK does none. * ci: the contract-fixture comparison also runs weekly, and a scheduled failure opens an issue The comparison of conformance/v1/manifest.sha256 with the manifest agentchat.me publishes ran only on push and pull request, so a contract change on the server was flagged here only at the next push. weekly.yml (Mondays 06:29 UTC, and Run workflow by hand) runs that comparison alone, not the 3 x 3 test matrix, through scripts/check-contract-fixtures.sh, which CI's step now calls too. Its report job (scheduled runs only, the one job with issues: write, through GITHUB_TOKEN) opens "Weekly checks failed" or comments on the open one, and a passing scheduled run comments and closes it. The repo had no other alert path (no webhooks, no notify steps). Its own file on purpose: GitHub turns scheduled workflows off in a public repository after 60 days without activity, and that must never stop CI. * test: wait for the condition, never a fixed sleep (realtime, conformance engine, kit) The #18 pattern, applied to the rest of the suite. A fixed sleep before a positive assert races the work it stands in for: a gap, backoff, handshake, stability or liveness timer, a handler running as its own task, a worker thread. On Windows before Python 3.13 the event loop's clock is GetTickCount64 (15.6 ms ticks) and the loop runs a timer once it is due within one tick, so asyncio.sleep(0.02) can return after almost no time and asyncio.sleep(0.05) after less than 20 ms. A slow runner (a late thread start, a starved loop) then loses the race. CI runs windows-latest on 3.9 and 3.11, where this holds. tests/_wait.py adds eventually(predicate, what): it polls from the calling task (no wait_for wrapper, which runs its coroutine as a separate task before 3.12, so a predicate may look at the loop's tasks) and fails after 5 s naming what it waited for. Predicates use >= so the exact assert that follows still shows the list when something extra arrives. Converted (23 sites): - test_realtime: the three gap-fill tests, the auto drain's cursor ack (a 20 x sleep(0.01) poll whose budget collapses on a coarse clock), the three reconnect-after-close tests, the stability window. - test_realtime_engine: the two gap-fill ack tests, the two lapsed-gap handler tests, seq 6's own gap window, b's reconnect. The "not a crash" test now waits until the gap task and every handler finished (no _bg_tasks, no _lanes), so a late KeyError or a second handover of seq 4 is still caught: checked by making the gap task raise. - test_realtime_gate_v1: the hung handler's 50 ms bound, the queue back to half, seq-order starts, the re-offer past the adoption window, the idle watchdog's close. The heartbeat test kept a quiet socket alive with beats every 60 ms against a 150 ms window: 90 ms of slack, minus up to two clock ticks. Now 50 ms beats against 200 ms, eight times: 150 ms of slack, and the beats still outlast the window twice over. - test_realtime_resilience: the second answered probe; the disconnect() from a state handler (waits until every task is gone). - test_conformance_engine: the wait_handler op (100 x sleep(0.001), the same collapsing budget) waits up to 5 s for the handler to enter. - test_integration_kit: the sync model call past its deadline. The cycle stops waiting at 30 ms whether or not the worker thread has started the call, but the test asserted the thread's handle after a 50 ms sleep: the #18 mechanism. It now waits for the call to have started. Left as they were: deliberate durations (a slow handler, the adoption clock test with its documented 150 ms margins, the liveness push cadence, the conformance gap_window_lapses op, whose gap timer is armed first and fires 150 ms earlier on the same loop); sleeps before negative asserts (nothing reconnects, nothing is recorded, no gap fires), several already backed by other_tasks() or ControlledSleep; sleeps followed by loop.idle() on a cycle already scheduled; the fake clock in test_conformance_v1; the 300 s webhook tolerance window; the disconnect-promptness bound (0.5 s against ~50 ms of work) and the dispose-grace bound (1 s of slack). * fix(kit): a turn gated while a freed slot is handed on queues behind the waiting ones test_at_most_max_concurrent_turns_run_and_the_rest_wait_for_a_slot failed about 1 run in 20 on 3.9 (1 in 10 under a 15.6 ms clock) with ['c1', 'c2', 'c1'] instead of ['c1', 'c2', 'c3'], on main's tests too. The kit, not the test: when a turn ends, _slot_freed frees its slot and hands it on through _pump, which runs as its own task. _launch checked only that a slot was free, so a cycle that ran before the pump (here the one c1's failed turn woke, gating c1 again) took the slot ahead of c3, which had been waiting for it. The TypeScript kit's pump takes the slot synchronously and has no such window. _launch now queues when turns are already waiting, so they start first, in order, as the docstring says. A new test holds the hand-on open and checks that the re-gated c1 waits behind c2 (it starts first without the fix). The slot tests ran 120 times (3.9 and 3.13, normal and 15.6 ms clock) without a failure. * ci(weekly): curl timeouts, any directory, and a timed-out run is reported too - check-contract-fixtures.sh: curl gets --connect-timeout 10 and --max-time 30 (a stalled server could hold CI's job for hours), runs from any directory, and tells a missing file (diff exit 2) apart from a mismatch. - report: runs for scheduled and manual runs on main, unless the run was cancelled by hand. Anything but a pass is reported, so a job that timed out opens the issue too; only a pass closes it. * ci(temporary): run weekly.yml on this pull request once, to prove its job on GitHub's runners * test(conformance): copy texts.sample.json and manifest.sha256 from the published contract (manual 2.3.1) The server now publishes the manual as 2.3.1, so the contract's texts.sample.json carries the new manual version and sha256, and the manifest lists the file's new hash. Both files are copied byte for byte from the published contract (core is canonical); every file in conformance/v1 matches the published manifest, and the manifest equals https://agentchat.me/contract/v1/conformance/manifest.sha256. No test reads the manual's version or hash from this fixture, only the hash's format. * review fixes: a refused admission hands its slot on; two waits keep their bounds - receive_loop._launch: a refused admission now frees its slot through _slot_freed, which hands it to a waiting turn, instead of only discarding it. With waiting turns first in order, a free slot left with turns waiting and no pump scheduled would stall every later turn. Not reachable today (one cycle at a time, and _launch admits only when none wait), but a randomized probe with overlapping callers stalled without it and ran clean with it. The docstring and CHANGELOG no longer claim the TypeScript kit is FIFO everywhere: its synchronous hand-on covers a turn's end, not a refusal inside its pump. - test_realtime_gate_v1: the hung-handler and idle-watchdog tests now also bound the wait (under 1 s), so a regression to the defaults (minutes, 100 s) still fails them as the old fixed sleeps did. - test_network_guard: the in-process test checks that all four doors are put back, not two. - SystemAgentProtectedError: "at most a pointer" (one per peer per day). * ci(weekly): a step timeout, the issue found without the search index, and no temporary trigger - The check step has its own 3-minute timeout: a step timeout fails the step, so the report always reads "failure" (a job timeout may read as cancelled). - The open issue is found from the open-issue list (limit 200, exact title), not the search index, which can lag and open a duplicate. - The temporary pull_request trigger is gone; the job ran green on this PR, against the live manifest after the 2.3.1 copy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a flake that hit
test_a_waiting_turn_that_is_released_never_startsonce, on windows-latest with Python 3.11 (run 42491213799:assert [] == ['c1']). The fault was in the test; the kit is unchanged.Why it flaked: the test fired the coalescer, slept 20 ms, then asserted that c1's turn had started. That cycle's model call is sync, so it runs in a worker thread through
asyncio.to_thread, on a thread pool created fresh for each test. On Windows before Python 3.13, the event loop's clock moves in 15.6 ms steps (gh-88494), soasyncio.sleep(0.02)can return almost at once. A slow runner's thread start then loses the race.Now: the test waits on the real conditions.
turns_waiting == 1before releasing c2.turns_waiting == 0right after the release, the same check the TypeScript twin makes.turns_active == 0before checking that c2 never started.Ten sibling tests had the same habit, a fixed sleep standing in for "the work finished" followed by an assert. They now wait through
_eventually, which takes a label so a timeout names what it was waiting for. Sleeps before "nothing happened" asserts are unchanged, since a slow machine can't make those fail.Proof, locally. I simulated the Windows clock and slow thread starts with a scratch pytest plugin (not committed), on top of a fully loaded CPU:
Test only; no SDK change, no release needed.