Repository navigation
test: wait for consumer before checking its read - #1826
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1826 +/- ##
=======================================
Coverage 99.75% 99.75%
=======================================
Files 23 23
Lines 6665 6665
=======================================
Hits 6649 6649
Misses 16 16
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
|
Thanks for the PR, and for tracking this down on ppc64le. It is a real problem with the test and we want it fixed. Two things before it can go in. 1. Please merge
|
| Page size | Slack | Short at the assertion |
|---|---|---|
| 4 KiB (real) | 16384 (same before and after this PR) | 4 of 350 |
| 16 KiB (modelled) | 16384 (before) | 0 of 110 |
| 16 KiB (modelled) | 4 pages (this PR) | 8 of 230 |
| 64 KiB (modelled) | 16384 (before) | 7 of 290 |
| 64 KiB (modelled) | 4 pages (this PR) | 0 of 290 |
| 4, 16 and 64 KiB | either, plus the wait below | 0 of 360 |
A separate script that repeats the test body up to that assertion gave the same picture on real kernels:
- Linux, 4 KiB pages, slack 16384: 1 of 40 short when idle, 2 of 560 with the CPUs saturated.
- macOS arm64, CPUs saturated: 55 of 560 short with slack 16384, 1 of 560 with 65536 (what this PR makes it there, since the page size is 16 KiB).
A model reproduces the kernel's page and slot arithmetic, not a real ppc64le machine. Does the test fail on every run for you, or only some of the time? If it fails every time, something else is going on and I'd like to understand it.
Recommended fix
Wait for the consumer before asserting. Its reads are capped at first_read, so the equality check still holds:
if pauses:
# The child can exit before the consumer has read all of its first page, so give it time.
deadline = time.monotonic() + 5
while len(received) < first_read and time.monotonic() < deadline:
time.sleep(0.01)
assert len(received) == first_readThat is the last row of the table. With it, the slack line does not need to change at all, so when you resolve the conflict you can take main's version of that hunk and add the wait.
Leaving the slack alone also avoids two side effects of scaling it by page size:
first_readbecomescapacity + 131072 - 4 * page, which is 0 on a 64 KiB-page kernel that hands out 2-page pipes (a user overfs.pipe-user-pages-soft). The consumer would then drain nothing and the child would time out. I have not run that case; it is from the arithmetic.- macOS arm64 has 16 KiB pages, so the slack changes there too, although macOS pipes are not sized in pages.
If you can confirm the wait fixes it on ppc64le, that would be very helpful, since none of us can test on that architecture.
tleonhardt
left a comment
There was a problem hiding this comment.
See my other comment regarding changes requested.
On ppc64le,
test_pipeline_writer_relay_lends_only_while_a_producer_waits[True]
failed with:
AssertionError: assert 1114112 == 1163264
The child's last write unblocks once the relay has forwarded about
capacity + 65536 bytes, 1114112 with 64 KiB pages. The consumer may
still need more forwarded to reach first_read, and the test asserted
as soon as the lend was returned, without waiting for it.
Wait up to 5 seconds for the consumer to reach first_read. Its reads
are capped there, so the equality check still holds.
|
I'm well out of my depth here so feel free to take this and rework - its your patch :-) I can confirm with 4.4.3 + #1828 + this updated patch then a ppc64le build does run through okay. https://kojipkgs.fedoraproject.org//work/tasks/9747/152389747/hw_info.log https://kojipkgs.fedoraproject.org//work/tasks/9747/152389747/build.log |
0c5c6b2
into
python-cmd2:fix/relay-consumer-read
|
Thanks a bunch @traylenator. I'll reword this slightly on a branch this merges into in order to make sure you get credit. |
On ppc64le,
test_pipeline_writer_relay_lends_only_while_a_producer_waits[True]
failed with:
The child's last write unblocks once the relay has forwarded about
capacity + 65536 bytes, 1114112 with 64 KiB pages. The consumer may
still need more forwarded to reach first_read, and the test asserted
as soon as the lend was returned, without waiting for it.
Wait up to 5 seconds for the consumer to reach first_read. Its reads
are capped there, so the equality check still holds.