Skip to content

Live docs links, the system-agent docstring, condition waits in tests, a kit slot-order fix, a whole-session network guard, and a weekly fixture check - #19

Merged
sanctrl merged 10 commits into
mainfrom
chore/docs-links-ci
Nov 11, 2026
Merged

sanctrl merged 10 commits into
mainfrom
chore/docs-links-ci

Conversation

@sanctrl

@sanctrl sanctrl commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Why

Six things were wrong or missing:

  • Dead links. The README, the package docstring and the PyPI "Documentation" link pointed at agentchat.me/docs pages that answer 404.
  • A stale docstring. SystemAgentProtectedError called system agents "the support channel" and said to send them support requests. ChatFather is retired: a direct message gets only a pointer to the manual.
  • Timing in tests. Many tests slept a fixed time where they meant "wait until this has happened". CI runs Windows on 3.9 and 3.11. There the event loop's clock ticks every 15.6 ms (before 3.13), so asyncio.sleep(0.02) can return almost at once. test: wait on real conditions in the integration-kit tests (Windows flake) #18 fixed one such test after it flaked.
  • A kit race. Stress runs showed a kit test failing about 1 run in 20, on main's tests too. The cause is in the kit, not the test.
  • Gaps in the network guard. It watched only while a test function ran, so code at collection time and in session-scoped fixtures was not covered. On Windows' default proactor loop, an asyncio connect to a literal IP address bypassed it altogether.
  • The fixture comparison. The check that our conformance fixtures equal the published contract ran only on push and pull request.

What changes

  • Links:

    • README.md: the SDK guide (docs.agentchat.me/sdks/python), the integration contract v1 (agentchat.me/contract/v1/) and the REST API reference.
    • src/agentchatme/__init__.py: the docstring points at the guide and this README.
    • pyproject.toml: the PyPI "Documentation" link is the SDK guide.
  • errors.py (SystemAgentProtectedError): the docstring now says when the server answers SYSTEM_AGENT_PROTECTED, as core does today:

    • blocking or reporting a system agent;
    • deleting or rotating the key of a system account through the public API.

    A system agent is not a support channel. The refusal carries the pointer to the manual.

  • Tests wait for conditions:

    • tests/_wait.py adds eventually(predicate, what), the test: wait on real conditions in the integration-kit tests (Windows flake) #18 pattern. It has a 5 s backstop and fails naming what it waited for.
    • 23 sites are converted, in test_realtime, test_realtime_engine, test_realtime_gate_v1, test_realtime_resilience, test_conformance_engine and test_integration_kit. Each one is listed with its race in the commit message.
    • Left as they were: deliberate durations, sleeps before "nothing happened" asserts, sleeps followed by loop.idle(), and fake clocks.
  • Kit fix (integration/receive_loop.py, _launch):

    • The cause: when a turn ends, its slot is handed on by _pump, which runs as its own task. A cycle that ran first could take the slot ahead of turns already waiting for it. In the stress runs this was the conversation whose turn had just failed, gated again.
    • Now a new turn queues behind the waiting ones, so they start first, in order, as the docstring says. (After a turn ends, the TypeScript kit's pump takes the slot synchronously.)
    • A refused admission now hands its slot on, too. Without that, the FIFO rule could leave a free slot behind waiting turns with no pump scheduled. That is not reachable today, since one cycle runs at a time, but a randomized probe with overlapping callers stalled.
    • A new test holds the hand-on open and fails without the fix.
    • The CHANGELOG has an Unreleased entry.
  • Network guard (tests/conftest.py):

    • It is installed in pytest_configure and removed in pytest_unconfigure, so it is on for the whole session.
    • It also wraps asyncio's BaseProactorEventLoop.sock_connect. On every platform the wrapper refuses before it touches the loop.
    • An attempt during a test, including a fixture set up for it, fails that test as before.
    • An attempt outside any test sets the exit status to "tests failed", and the summary names it.
    • Live tests stay exempt, fixtures included.
    • Not covered, as the docstring says: subprocesses, UDP sendto without a connect, and direct use of _socket.
  • Contract fixtures (manual 2.3.1): the server now publishes the manual as 2.3.1.

    • conformance/v1/texts.sample.json (the manual's version and sha256) and manifest.sha256 are copied byte for byte from core main at 46dc031.
    • Every file in conformance/v1/ matches the live manifest, and the copy is identical to the TypeScript SDK's.
    • No test reads the manual's version or hash from this fixture, only the hash's format.
  • Weekly fixture check:

    • scripts/check-contract-fixtures.sh holds the comparison, with curl timeouts. It works from any directory and tells a missing file apart from a mismatch. CI's step calls it.
    • .github/workflows/weekly.yml runs it every Monday and by hand. It runs only that comparison, not the 3 × 3 matrix.
    • A run on main that does not pass opens "Weekly checks failed", or comments on the open one. A run that passes closes it.
      • This covers scheduled and manual runs, unless a run was cancelled by hand.
      • The check step has its own timeout, so a stall reads as a failure.
      • The open issue is found from the issue list, not the search index, which can lag.
      • Before this, the only alert was GitHub's default email to whoever last edited a cron line, and there was no cron line.
    • The issue is written with GITHUB_TOKEN, and the report job is the only one with issues: write. No new secrets.
    • It lives in its own file because GitHub turns scheduled workflows off after 60 days without activity in a public repo, and that must never stop CI.

Proof

  • Gate: ruff, mypy, and pytest on 3.9, 3.10, 3.11, 3.12 and 3.13, all green (718 passed, 2 skipped on each). The second skip is the Windows-only guard test.

  • Network guard:

    • Its four new tests each fail against the old guard: the proactor door, a swallowed lookup at collection, and a session fixture's setup and final teardown.
    • The Windows-only test runs a real literal-IP open_connection and a loopback round trip on the proactor loop. It runs in CI on windows-latest; I can't run it locally.
  • Converted sites: I instrumented the helper temporarily. Each new wait started with its condition false, apart from two backstop-only waits.

  • Stress: each changed file ran 20 times on 3.9 and 3.13, plus runs under CPU load and under a simulated 15.6 ms clock. The only failure was the kit race.

  • Kit fix:

    • The slot tests ran 120 times (3.9 and 3.13, normal and 15.6 ms clock) with no failure. Before the fix: 1 in 10 under the coarse clock.
    • The new test fails without the fix with ['c1', 'c1'].
  • Report job: its shell ran against a stub gh for a failure, a timeout or cancel, and a pass, each with and without an open issue. The real issue lookup ran read-only against this repo.

  • Weekly job on GitHub: a temporary pull_request trigger let weekly.yml's job run on GitHub's runners. It ran green on runs 42992391953 and 42996593439, the second against the live manifest after the 2.3.1 copy. The trigger is removed.

  • Review: an independent reviewer checked all six goals as met. Fixed from its findings:

    • the refused-admission hand-on;
    • wall-clock upper bounds (under 1 s) on two converted waits whose names promise a bound;
    • all four guard doors are now checked on restore;
    • the step timeout and the issue lookup;
    • wording.

    Left for follow-ups:

    • the TypeScript kit's launch can still overtake waiting turns right after a refusal inside its pump;
    • the cycle watchdog shares its turn-depth counter with the pump's admissions;
    • disconnect() from inside an async handler waits the full 1 s dispose grace.

    All three are on main already.

Test plan

  • CI green on the head commit: check on ubuntu, macOS and Windows × 3.9 / 3.11 / 3.13, plus secret-scan (run 42997221210)
  • Windows legs: test_a_literal_ip_asyncio_connect_on_the_windows_default_loop_fails_the_test ran and passed (719 passed and only the live smoke skipped, on 3.9, 3.11 and 3.13)
  • the temporary weekly run: contract-fixtures green against the live manifest, report skipped
  • the CI manifest step matches the live manifest after the 2.3.1 copy (run 42996593567)
  • after merge: Actions → Weekly → Run workflow on main is green

…system agents the support channel

The README links, the package docstring and the PyPI "Documentation" URL
pointed at agentchat.me/docs/sdk/python, /docs/realtime and /docs, which
all answer 404. They now point at the live pages: the SDK guide
(docs.agentchat.me/sdks/python), the integration contract v1
(agentchat.me/contract/v1/: socket frames, tray, wake payloads,
conformance fixtures) and the REST API reference.

The SystemAgentProtectedError docstring said system agents are "the
support channel" and to send them support requests. ChatFather is
retired: a direct message gets only a pointer to the manual, and the
server's block/report refusal carries the same pointer. The docstring now
says when the server answers SYSTEM_AGENT_PROTECTED (block or report a
system agent; delete or rotate the key of a system account through the
public API) and points at the manual instead of a DM.
…indows proactor connect

The guard was a function-scoped autouse fixture, so it watched only while
a test ran. Two gaps:

- Collection (a test module that connects at import) and session- or
  module-scoped fixtures ran outside it: they set up before the
  function fixture patched anything.
- On Windows' default proactor loop, asyncio connects through ConnectEx
  and never calls socket.connect, and a literal IP address skips
  getaddrinfo, so asyncio.open_connection("203.0.113.7", 443) went out.

Now the guard is installed in pytest_configure and removed in
pytest_unconfigure, so it is on for the whole session. It also wraps
asyncio's BaseProactorEventLoop.sock_connect (importable on every
platform; it refuses before touching the loop). An attempt while a test
runs, including a wider-scoped fixture set up for it, fails that test as
before. An attempt outside any test (collection, a fixture's final
teardown) sets the exit status to "tests failed" and the summary names
it. Live tests are exempt for their whole run, fixtures included.

New tests, each failing against the old guard: the proactor door (an
unbound first step, so it runs on every OS), a swallowed lookup at
collection, a session fixture's setup and final teardown, and on Windows
only, a real literal-IP open_connection on the default loop plus a
loopback round trip through the proactor (in a subprocess: asyncio.run in
an in-process inner session leaves the main thread with no current loop).
The live-exemption test now checks behaviour (a numeric-host lookup,
answered on this machine) instead of function names.

Not covered, and named in the docstring: subprocesses, UDP sendto without
a connect, and direct use of the C _socket module. The SDK does none.
… failure opens an issue

The comparison of conformance/v1/manifest.sha256 with the manifest
agentchat.me publishes ran only on push and pull request, so a contract
change on the server was flagged here only at the next push.

weekly.yml (Mondays 06:29 UTC, and Run workflow by hand) runs that
comparison alone, not the 3 x 3 test matrix, through
scripts/check-contract-fixtures.sh, which CI's step now calls too. Its
report job (scheduled runs only, the one job with issues: write, through
GITHUB_TOKEN) opens "Weekly checks failed" or comments on the open one,
and a passing scheduled run comments and closes it. The repo had no other
alert path (no webhooks, no notify steps).

Its own file on purpose: GitHub turns scheduled workflows off in a public
repository after 60 days without activity, and that must never stop CI.
…nce engine, kit)

The #18 pattern, applied to the rest of the suite. A fixed sleep before a
positive assert races the work it stands in for: a gap, backoff, handshake,
stability or liveness timer, a handler running as its own task, a worker
thread. On Windows before Python 3.13 the event loop's clock is
GetTickCount64 (15.6 ms ticks) and the loop runs a timer once it is due
within one tick, so asyncio.sleep(0.02) can return after almost no time
and asyncio.sleep(0.05) after less than 20 ms. A slow runner (a late thread
start, a starved loop) then loses the race. CI runs windows-latest on 3.9
and 3.11, where this holds.

tests/_wait.py adds eventually(predicate, what): it polls from the calling
task (no wait_for wrapper, which runs its coroutine as a separate task
before 3.12, so a predicate may look at the loop's tasks) and fails after
5 s naming what it waited for. Predicates use >= so the exact assert that
follows still shows the list when something extra arrives.

Converted (23 sites):
- test_realtime: the three gap-fill tests, the auto drain's cursor ack
  (a 20 x sleep(0.01) poll whose budget collapses on a coarse clock), the
  three reconnect-after-close tests, the stability window.
- test_realtime_engine: the two gap-fill ack tests, the two lapsed-gap
  handler tests, seq 6's own gap window, b's reconnect. The "not a crash"
  test now waits until the gap task and every handler finished
  (no _bg_tasks, no _lanes), so a late KeyError or a second handover of
  seq 4 is still caught: checked by making the gap task raise.
- test_realtime_gate_v1: the hung handler's 50 ms bound, the queue back
  to half, seq-order starts, the re-offer past the adoption window, the
  idle watchdog's close. The heartbeat test kept a quiet socket alive with
  beats every 60 ms against a 150 ms window: 90 ms of slack, minus up to
  two clock ticks. Now 50 ms beats against 200 ms, eight times: 150 ms of
  slack, and the beats still outlast the window twice over.
- test_realtime_resilience: the second answered probe; the disconnect()
  from a state handler (waits until every task is gone).
- test_conformance_engine: the wait_handler op (100 x sleep(0.001), the
  same collapsing budget) waits up to 5 s for the handler to enter.
- test_integration_kit: the sync model call past its deadline. The cycle
  stops waiting at 30 ms whether or not the worker thread has started the
  call, but the test asserted the thread's handle after a 50 ms sleep: the
  #18 mechanism. It now waits for the call to have started.

Left as they were: deliberate durations (a slow handler, the adoption
clock test with its documented 150 ms margins, the liveness push cadence,
the conformance gap_window_lapses op, whose gap timer is armed first and
fires 150 ms earlier on the same loop); sleeps before negative asserts
(nothing reconnects, nothing is recorded, no gap fires), several already
backed by other_tasks() or ControlledSleep; sleeps followed by
loop.idle() on a cycle already scheduled; the fake clock in
test_conformance_v1; the 300 s webhook tolerance window; the
disconnect-promptness bound (0.5 s against ~50 ms of work) and the
dispose-grace bound (1 s of slack).
…the waiting ones

test_at_most_max_concurrent_turns_run_and_the_rest_wait_for_a_slot failed
about 1 run in 20 on 3.9 (1 in 10 under a 15.6 ms clock) with
['c1', 'c2', 'c1'] instead of ['c1', 'c2', 'c3'], on main's tests too.
The kit, not the test: when a turn ends, _slot_freed frees its slot and
hands it on through _pump, which runs as its own task. _launch checked only
that a slot was free, so a cycle that ran before the pump (here the one
c1's failed turn woke, gating c1 again) took the slot ahead of c3, which
had been waiting for it. The TypeScript kit's pump takes the slot
synchronously and has no such window.

_launch now queues when turns are already waiting, so they start first,
in order, as the docstring says. A new test holds the hand-on open and
checks that the re-gated c1 waits behind c2 (it starts first without the
fix). The slot tests ran 120 times (3.9 and 3.13, normal and 15.6 ms
clock) without a failure.
…rted too

- check-contract-fixtures.sh: curl gets --connect-timeout 10 and
  --max-time 30 (a stalled server could hold CI's job for hours), runs
  from any directory, and tells a missing file (diff exit 2) apart from a
  mismatch.
- report: runs for scheduled and manual runs on main, unless the run was
  cancelled by hand. Anything but a pass is reported, so a job that timed
  out opens the issue too; only a pass closes it.
…e published contract (manual 2.3.1)

The server now publishes the manual as 2.3.1, so the contract's
texts.sample.json carries the new manual version and sha256, and the
manifest lists the file's new hash. Both files are copied byte for byte
from the published contract (core is canonical); every file in
conformance/v1 matches the published manifest, and the manifest equals
https://agentchat.me/contract/v1/conformance/manifest.sha256. No test
reads the manual's version or hash from this fixture, only the hash's
format.
…heir bounds

- receive_loop._launch: a refused admission now frees its slot through
  _slot_freed, which hands it to a waiting turn, instead of only
  discarding it. With waiting turns first in order, a free slot left with
  turns waiting and no pump scheduled would stall every later turn. Not
  reachable today (one cycle at a time, and _launch admits only when none
  wait), but a randomized probe with overlapping callers stalled without
  it and ran clean with it. The docstring and CHANGELOG no longer claim
  the TypeScript kit is FIFO everywhere: its synchronous hand-on covers a
  turn's end, not a refusal inside its pump.
- test_realtime_gate_v1: the hung-handler and idle-watchdog tests now
  also bound the wait (under 1 s), so a regression to the defaults
  (minutes, 100 s) still fails them as the old fixed sleeps did.
- test_network_guard: the in-process test checks that all four doors are
  put back, not two.
- SystemAgentProtectedError: "at most a pointer" (one per peer per day).
… and no temporary trigger

- The check step has its own 3-minute timeout: a step timeout fails the
  step, so the report always reads "failure" (a job timeout may read as
  cancelled).
- The open issue is found from the open-issue list (limit 200, exact
  title), not the search index, which can lag and open a duplicate.
- The temporary pull_request trigger is gone; the job ran green on this
  PR, against the live manifest after the 2.3.1 copy.
@sanctrl
sanctrl marked this pull request as ready for review November 11, 2026 09:43
@sanctrl
sanctrl merged commit 4a6db13 into main Nov 11, 2026
11 checks passed
@sanctrl
sanctrl deleted the chore/docs-links-ci branch November 11, 2026 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant