UTS: fix flaky integration specs and make polling/timing contracts explicit - #517
Open
sacOO7 wants to merge 2 commits into
Open
UTS: fix flaky integration specs and make polling/timing contracts explicit#517sacOO7 wants to merge 2 commits into
sacOO7 wants to merge 2 commits into
Conversation
…cit in the derived-tests docs Driven by CI evidence from ably-cocoa (the SDK whose runners surfaced these): RSP4b1 fired twice on different lanes from runner-vs-server clock skew, and teardown close-awaits timed out on stalled runners. Spec fixes: - rest/integration/presence.md (RSP4b1): derive the queried time range from the events' server-assigned timestamps instead of client now_millis() — mirrors the convention history.md already documents. - Cleanup AWAIT_STATE closed normalized 10s -> 15s across the 7 files sharing the AFTER-EACH template (in-body spec-timed awaits deliberately untouched). - realtime/integration/proxy/auth_reauth.md: the AUTH-frame assertion now polls the proxy log (value-form) instead of a single snapshot that can observe 0 frames before the AUTH round-trip lands. Derived-tests docs (so translations inherit the contracts without per-SDK skill guidance): - writing-derived-tests.md: poll_until's two forms + assert-on-settled-value rule (a refetch of an eventually-consistent read can under-return); random_id() mapping row; untimed-steps rule (use the harness default wait); spec-written WAIT carve-out (deliberate, translate as a real wait). - integration-testing.md: settled-value polling bullet under Avoiding Flaky Tests.
- writing-derived-tests.md poll_until row: plain-English no-refetch wording (a refetch of an eventually-consistent read can return fewer items) plus the error semantics — an error raised inside the condition aborts the poll and fails the test; never swallow it in the predicate. poll_until_success row: full description (errors mean keep-polling, last error re-raised on timeout) and its exclusive raise-by-design use-cases. - Infra-setup paragraph: port the reference definitions from writing-test-specs.md rather than reconstructing from prose — every clause is load-bearing (past harnesses drifted by refetching and by swallowing predicate errors). - integration-testing.md anti-flake bullet: same plain-English rewording.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related PR
Motivation
CI runs of the derived UTS integration tests in ably-cocoa surfaced three stability defects that live in the specs and docs, not in any SDK:
RSP4b1(rest presence, time-range history) is flaky by construction. The spec bracketed the queried window with client-sidenow_millis(), but the events are timestamped by the server. Runner clock skew larger than the window silently excludes the events. It fired twice on ably-cocoa CI (tvOS 2026-08-15, macOS 2026-08-19, both protocol variants each time).AFTER EACH TESTcleanup template used a 10sAWAIT_STATE closedwhile every in-body await uses 15–30s — a stalled runner fails a passing test in teardown.auth_reauth's AUTH-frame assertion raced the proxy log. A singlegetLog()snapshot can observe 0 frames, because the SDK sends AUTH only after the auth callback returns and any token round-trip completes.Separately, a cross-SDK audit showed why one SDK's translation drifted where others didn't: the
poll_untilcontract (its value-returning form) was defined in the spec-author doc (writing-test-specs.md) but absent from the translator-facing doc (writing-derived-tests.md). Per the UTS principle that specs/docs must be explicit enough that no per-SDK translation guidance is needed, the contracts now live in the translator doc.Spec changes
uts/rest/integration/presence.md(RSP4b1)history.mdalready documents ("Client-side now() must not be used here")objects_faults,auth_reauth×2,channel_faults,connection_open_failures,heartbeat,rest_faults,connection_lifecycle_test)AWAIT_STATE closed10s → 15s. In-body spec-timed awaits (e.g.channel_history_test.md's 10s connect-awaits) deliberately untoucheduts/realtime/integration/proxy/auth_reauth.mdpoll_until(15s), matching how the log actually fillsDocs changes (translator-facing explicitness)
uts/docs/writing-derived-tests.mdpoll_untiltable row now states both forms and the settled-value rule: a value-assigned poll returns the settled result and later assertions run on it, never on a re-read (a refetch of an eventually-consistent read can under-return). Added:random_id()mapping row; "use the harness's default wait for steps the spec doesn't time"; spec-writtenWAITcarve-out (deliberate, translate as a real wait — the anti-sleep rule bans invented waits)uts/docs/integration-testing.mdCross-SDK impact
presence.test.tsRSP4b1 still usesDate.now()(verbatim old-spec port; same latent flake, un-fired on ubuntu runners) andauth_reauth.test.tsstill snapshots the log once.