ssh, internal: flush the worker's queued output on every call - #1217
ssh, internal: flush the worker's queued output on every call#1217yosuke-wolfssl wants to merge 3 commits into
Conversation
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
b9d4cc3 to
a650544
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
a650544 to
7b1ec65
Compare
7b1ec65 to
d95204d
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Fenrir result: Approved ✅
No new issues found in the changed files.
Advisory only — this automated result does not count as a GitHub approval.
Fenrir's latest completed scan found no issues; clearing the prior automated change request.
d95204d to
b598ecc
Compare
b598ecc to
e2839fa
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
e2839fa to
e5524ca
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
e5524ca to
d15c978
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 2
1 finding(s) posted as inline comments (see file-level comments below)
Required changes (1)
wolfsshd WIN32 forced-command drain loop becomes an unbounded CPU spin
File: apps/wolfsshd/wolfsshd.c:1131
Function: SHELL_Subsystem
Category: Resource leaks
wolfSSH_worker() now leaves ssh->error == WS_WANT_WRITE on an idle receive with queued output (src/ssh.c:3731-3746), so this unpaced loop never terminates while the peer stops reading the non-blocking socket (set at wolfsshd.c:3412) — a remote 100% CPU spin per session.
Recommendation: Wait for socket writability with a bounded select/timeout between passes and give up after a cap instead of looping on WS_WANT_WRITE.
Referenced code: apps/wolfsshd/wolfsshd.c:1131-1135 (5 lines)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
d15c978 to
c547220
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 1
Required changes (1)
wolfsshd Windows drain loop becomes a delay-free busy-wait on the worker's new WS_WANT_WRITE
File: apps/wolfsshd/wolfsshd.c:1131
Function: SHELL_Subsystem
Category: Incorrect error handling
wolfSSH_worker() now leaves ssh->error == WS_WANT_WRITE when an idle receive is followed by a blocked flush (previously the removed second DoReceive() left WS_WANT_READ). This drain loop has no select(), sleep, or iteration cap, so a peer that stops reading pins a CPU core on the non-blocking connection socket for as long as it keeps the socket unwritable.
Recommendation: Wait for write readiness with select() on the write set, or bound the loop with a deadline and a pause between retries.
Referenced code: apps/wolfsshd/wolfsshd.c:1131-1136 (6 lines)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
Note for last Fenrir-bot comments and reviewers:Confirmed, and deferred deliberately. The mechanism is as described: that drain has no Both are folded into one follow-up with Not fixing it in this PR because nothing can verify it. |
fe68694 to
35f752f
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
35f752f to
cd5c5e8
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
cd5c5e8 to
f177bae
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
- The echoserver and Espressif shell loops and the Windows wolfsshd shell loop treat a WS_WANT_WRITE from wolfSSH_worker() as non-fatal.
- wolfSSH_TriggerKeyExchange() writes ssh->error only when SendKexInit() fails. It runs from HighwaterCheck() inside wolfSSH_SendPacket(), so writing WS_SUCCESS there erased what the pass the mark fired on had already reported. - test_TriggerKeyExchangeKeepsError() seeds ssh->error and checks a rekey that starts cleanly leaves it alone.
- wolfSSH_worker() calls wolfSSH_SendPacket() whenever ssh->outputBuffer holds bytes and the session is not disconnected, in place of doing so only for WS_SUCCESS, WS_WANT_READ, WS_CHAN_RXD or WS_EOF. ssh->error keeps the receive's code when the receive failed, and the close's when a WS_CHANNEL_CLOSED pass hard-failed its flush; a function-scope sendRet also masks the WS_REKEYING report. Drops the second DoReceive(), its WS_WINDOW_FULL case, the WOLFSSH_TEST_BLOCK fork, and the separate WS_CHANNEL_CLOSED flush. - BundlePacket() resets ssh->outputBuffer.length to ssh->packetStartIdx when the framing fails, and wolfSSH_shutdown() reports WS_WANT_WRITE when its read for the peer's close leaves output queued. - SendPacketFlush() records its code in ssh->error on every transport failure path; wolfSSH_SendPacket() says so and what a later write to that field owes it. - wolfssh/ssh.h drops WS_WINDOW_FULL from wolfSSH_worker() and names the two exceptions to a status surviving a failed flush: a hard-failed WS_CHANNEL_CLOSED, and WS_REKEYING. Comments in ssh.c, unit.c, the echoserver, the Espressif copy and portfwd name the channel's own state or the failing call instead of restating a contract. - Sixteen unit tests and the extended TestWorkerReportsDisconnect cover what ret and ssh->error hold after a receive, send, buffer, callback or framing failure, against channel data, extended data, a half-close, a rekey, a close, and a teardown read. The #ifndef WOLFSSH_TEST_BLOCK guards around TestWorkerReadsWhenSendWouldBlock go with the send-first fork.
f177bae to
c9f9857
Compare
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #1217
Scan targets checked: wolfssh-bugs, wolfssh-src
Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Reported findings require changes before merge.
Problem
A read-only application on a non-blocking socket stalls permanently.
A channel read credits the window,
ChannelCreditWindow()bundles aCHANNEL_WINDOW_ADJUSTintossh->outputBuffer, and the socket write blocks. The credit is not re-parked, sowolfSSH_SendPacket()is the only thing that can discharge it, and the peer has spent its window and goes silent waiting for it. The application callswolfSSH_worker(), aswolfssh/ssh.hdirects — but the worker gated its flush onDoReceive()'s return, and an idle socket makesDoReceive()returnWS_FATAL_ERROR, which the gate did not list. No write is attempted, on that call or any later one.The fix (
src/ssh.c)wolfSSH_worker()flushes whenever output is queued and the session is live:ssh->errorkeeps the receive's code when the receive failed, and the close's when aWS_CHANNEL_CLOSEDpass hard-failed its flush. Every other status keeps the send's.WS_REKEYINGis withheld when the flush failed, so a dead transport is not reported as a rekey to drive.DoReceive(), itsWS_WINDOW_FULLarm, theWOLFSSH_TEST_BLOCKordering fork, and the separateWS_CHANNEL_CLOSEDflush — all four existed to work around the gate.The gate ignores
retdeliberately. An idle receive and a hard one both surface asWS_FATAL_ERROR, so a narrower gate would key onssh->error == WS_WANT_READand reintroduce this same class of bug: a status nobody thought to list stops the flush.ssh->errordiscipline (src/internal.c)SendPacketFlush()now records its code on every transport failure path, not onlyWS_WANT_WRITE— otherwise a hard send failure left the idle receive'sWS_WANT_READin place and callers would select for read on a dead socket. Two rules follow, both stated onwolfSSH_SendPacket(): a later write tossh->erroron the same pass must be conditional on the flush having succeeded, and success is never written into the field.wolfSSH_TriggerKeyExchange()violated the second, and runs fromHighwaterCheck()insidewolfSSH_SendPacket(), so a clean flush could zero the field for any caller.What the wider flush exposed
Flushing on every pass reaches two places the old gate shielded:
BundlePacket()leftoutputBuffer.lengthpastpacketStartIdxwhen framing failed, so an unpadded, un-MAC'd, unencrypted partial packet stayed queued. The old gate excludedWS_FATAL_ERROR, so nothing went out; the new one would have put it on the wire. Its failure arm now drops that packet alone.wolfSSH_shutdown()mappedWS_CHAN_RXDandWS_EOFfrom its close-read toWS_SUCCESS. If the receive highwater fires on that read it queues a KEXINIT, and a blocked socket left the teardown reporting success with bytes owed. It now reportsWS_WANT_WRITE, whichssh.halready documents for the call.Consumers
Three shell loops treated anything but
WS_WANT_READas fatal —examples/echoserver/echoserver.c:1205,apps/wolfsshd/wolfsshd.c:1364,ide/Espressif/.../echoserver.c:1170— one line each. For these three the teardown is pre-existing: master's worker already returnsWS_WANT_WRITEvia its secondDoReceive()path, and master'sSendPacketFlush()already records it, so they already drop a live session when the send buffer fills mid-transfer. This change only makes it easier to reach.ssh.hstates the worker's contract and its two exceptions;WS_WINDOW_FULLcomes off the return list, since no path reaches it. No public API change.Tests
Sixteen new unit tests cover the flush on an idle receive, the owed flush across calls, and what
retandssh->errorhold after a receive, send, buffer, callback or framing failure — each against channel data, extended data, a half-close, a rekey, or a channel close. One arm is knowingly uncovered: aDoReceive()returning plainWS_SUCCESSwith output queued, which needs a non-channel packet builder nothing else uses.Verification
unit.test159 passed / 0 failed;regress.testpassed. Clean under gcc-13-Werroracross 6 configurations, plus lint.-DWOLFSSH_TEST_BLOCK,scripts/sftp.test) passes atWOLFSSH_BLOCK_PROB70/50/30 in 61s/33s/13s.scp.testandget-put.testskip under that macro by their own design.Known limitations, not addressed here
Five caller loops mishandle an owed flush, and they are deferred together because none has a test that can verify a fix — nothing drives a shell session under
-DWOLFSSH_TEST_BLOCK.examples/echoserver/echoserver.c:806select()at:999watches read fds,NULLtimeoutide/Espressif/.../echoserver.c:794select()at:984apps/wolfsshd/wolfsshd.c:1364select()whenwolfSSH_stream_peek()has dataexamples/echoserver/echoserver.c:1496apps/wolfsshd/wolfsshd.c:1131The last two are a regression this PR creates, not one it inherits. On master an idle receive attempts no flush, so
ssh->errorisWS_WANT_READand both loops exit after one pass; here the flush runs every pass and re-setsWS_WANT_WRITE, so they spin until the peer's window opens. One iswolfsshd, which ships.An earlier revision fixed those two with a
tcp_select_write()helper. It was pulled out: fixing two of five left the class half-done and grew the diff without closing the stall. The follow-up writes the shell-session harness first, then fixes all five.