Skip to content

test: wait for consumer before checking its read - #1826

Merged
tleonhardt merged 1 commit into
python-cmd2:fix/relay-consumer-readfrom
traylenator:slack
Nov 17, 2026
Merged

tleonhardt merged 1 commit into
python-cmd2:fix/relay-consumer-readfrom
traylenator:slack

Conversation

@traylenator

@traylenator traylenator commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

On ppc64le,
test_pipeline_writer_relay_lends_only_while_a_producer_waits[True]
failed with:

AssertionError: assert 1114112 == 1163264

The child's last write unblocks once the relay has forwarded about
capacity + 65536 bytes, 1114112 with 64 KiB pages. The consumer may
still need more forwarded to reach first_read, and the test asserted
as soon as the lend was returned, without waiting for it.

Wait up to 5 seconds for the consumer to reach first_read. Its reads
are capped there, so the equality check still holds.

@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.75%. Comparing base (1e8abbb) to head (8b14977).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1826   +/-   ##
=======================================
  Coverage   99.75%   99.75%           
=======================================
  Files          23       23           
  Lines        6665     6665           
=======================================
  Hits         6649     6649           
  Misses         16       16           
Flag Coverage Δ
unittests 99.75% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@tleonhardt

Copy link
Copy Markdown
Member

Thanks for the PR, and for tracking this down on ppc64le. It is a real problem with the test and we want it fixed.

Two things before it can go in.

1. Please merge main and resolve the conflict

#1825 was merged after you branched and changed the line directly above yours (the payload now uses cu._RELAY_CHUNK_SIZE), so this PR conflicts with main in tests/test_utils.py.

2. The failure looks like a race in the test, and the slack change does not remove it

The change does make the failure go away on 64 KiB pages, but I don't think the page-granularity explanation is the cause, and the same failure remains possible on 4 KiB and 16 KiB pages.

What I think is happening

The test asserts len(received) == first_read as soon as the child has exited and the lend has been returned. It never waits for the consumer thread to get there.

The child's last write unblocks once the relay has forwarded about capacity + 65536 bytes. On ppc64le that is 1114112, which is exactly the value in your failure. At that point the consumer still needs 49152 more bytes forwarded to reach first_read (1163264), and the main thread can reach the assertion first.

A consumer that is blocked short of first_read means its pipe is empty, not that a partial-page read freed no space. In every failure I reproduced, the consumer reached first_read within the next second.

With 4 pages of slack, first_read on 64 KiB pages drops to 14 pages. That is less than the roughly 15 pages already sitting in the consumer's pipe when the lend happens, so the consumer no longer depends on the relay to finish its read. That is why the change helps there. On 4 KiB pages the slack is unchanged, and on 16 KiB pages the consumer still needs data forwarded around the time the child exits.

Evidence

I don't have a 64 KiB-page machine, so apart from the 4 KiB rows this comes from a scale model: Linux with 4 KiB pages, with every size the relay and the test use (relay chunk, PIPE_BUF write size, consumer read size, the fixed slack) divided by 4 or 16, so that a 4 KiB page stands in for a 16 KiB or 64 KiB one and the pipe keeps its 16 slots. Idle, everything passes. With the CPUs saturated, the pauses=True case came up short at that assertion this often:

Page size Slack Short at the assertion
4 KiB (real) 16384 (same before and after this PR) 4 of 350
16 KiB (modelled) 16384 (before) 0 of 110
16 KiB (modelled) 4 pages (this PR) 8 of 230
64 KiB (modelled) 16384 (before) 7 of 290
64 KiB (modelled) 4 pages (this PR) 0 of 290
4, 16 and 64 KiB either, plus the wait below 0 of 360

A separate script that repeats the test body up to that assertion gave the same picture on real kernels:

  • Linux, 4 KiB pages, slack 16384: 1 of 40 short when idle, 2 of 560 with the CPUs saturated.
  • macOS arm64, CPUs saturated: 55 of 560 short with slack 16384, 1 of 560 with 65536 (what this PR makes it there, since the page size is 16 KiB).

A model reproduces the kernel's page and slot arithmetic, not a real ppc64le machine. Does the test fail on every run for you, or only some of the time? If it fails every time, something else is going on and I'd like to understand it.

Recommended fix

Wait for the consumer before asserting. Its reads are capped at first_read, so the equality check still holds:

        if pauses:
            # The child can exit before the consumer has read all of its first page, so give it time.
            deadline = time.monotonic() + 5
            while len(received) < first_read and time.monotonic() < deadline:
                time.sleep(0.01)
            assert len(received) == first_read

That is the last row of the table. With it, the slack line does not need to change at all, so when you resolve the conflict you can take main's version of that hunk and add the wait.

Leaving the slack alone also avoids two side effects of scaling it by page size:

  • first_read becomes capacity + 131072 - 4 * page, which is 0 on a 64 KiB-page kernel that hands out 2-page pipes (a user over fs.pipe-user-pages-soft). The consumer would then drain nothing and the child would time out. I have not run that case; it is from the arithmetic.
  • macOS arm64 has 16 KiB pages, so the slack changes there too, although macOS pipes are not sized in pages.

If you can confirm the wait fixes it on ppc64le, that would be very helpful, since none of us can test on that architecture.

@tleonhardt tleonhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my other comment regarding changes requested.

On ppc64le,
test_pipeline_writer_relay_lends_only_while_a_producer_waits[True]
failed with:

    AssertionError: assert 1114112 == 1163264

The child's last write unblocks once the relay has forwarded about
capacity + 65536 bytes, 1114112 with 64 KiB pages. The consumer may
still need more forwarded to reach first_read, and the test asserted
as soon as the lend was returned, without waiting for it.

Wait up to 5 seconds for the consumer to reach first_read. Its reads
are capped there, so the equality check still holds.
@traylenator traylenator changed the title test: scale pipe relay slack with page size test: wait for consumer before checking its read Nov 17, 2026
@traylenator

traylenator commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

I'm well out of my depth here so feel free to take this and rework - its your patch :-)

I can confirm with 4.4.3 + #1828 + this updated patch then a ppc64le build does run through okay.

https://kojipkgs.fedoraproject.org//work/tasks/9747/152389747/hw_info.log

https://kojipkgs.fedoraproject.org//work/tasks/9747/152389747/build.log

@tleonhardt
tleonhardt changed the base branch from main to fix/relay-consumer-read November 17, 2026 22:00
@tleonhardt
tleonhardt merged commit 0c5c6b2 into python-cmd2:fix/relay-consumer-read Nov 17, 2026
47 of 49 checks passed
@tleonhardt

Copy link
Copy Markdown
Member

Thanks a bunch @traylenator. I'll reword this slightly on a branch this merges into in order to make sure you get credit.

@traylenator
traylenator deleted the slack branch November 18, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants