Skip to content

Net::FTP: the data connection releases the GVL too, and the backlog shrinks - #927

Merged
tannevaled merged 1 commit into
mainfrom
ftp-data-gvl
Nov 13, 2026
Merged

tannevaled merged 1 commit into
mainfrom
ftp-data-gvl

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Net::FTP: the data connection releases the GVL too, and the backlog shrinks

#921 bounded the control connection; #924 released the GVL around it and added a
gate. Both left the data connection holding it — and the gate said so, in
four seeded backlog entries I wrote down rather than fixed, so that the gate
could land without a sprawling change nobody would review.

This is that change. dialData, acceptData, sendData, retrlines and
retrbinary now run their blocking call inside ftpIO, and the four entries are
deleted from testdata/gvl-blocking-backlog.txt. That list may only shrink,
and an entry that stops reporting fails the gate too, so a fix cannot leave its
allowance behind for the next regression to hide in.

⛔ acceptData is the one issue 916 is actually about

In active mode the client listens and the server dials back, so a Ruby
program that serves itself has both ends in one process. Holding the GVL in that
Accept parks every other Ruby thread — including the one that would make the
call it is waiting for
.

ruby/spec's net-ftp fixtures are exactly that shape. Their spec_helper.rb puts
Net::FTP into active mode, and rbgo has no Net::FTP.default_passive, so today
that helper is skipped in silence and the suite never reaches this path at all
(written up in the issue's latest comment). The order matters and it is the
opposite of the obvious one: release the GVL here first, add
default_passive second, re-measure third — otherwise today's fast errors turn
into fresh hangs.

Only the blocking call goes inside each closure: ftpConnErr raises, and a panic
crossing a released GVL would unwind without re-acquiring it, so every raise
stays outside.

⛔ No behavioural test, and the reason is not laziness

The in-repo FTP test server is Go-side. Nothing about rbgo's emulated GVL
blocks a Go goroutine, so that server cannot reproduce this deadlock at all: a
test built on it would pass whether or not this change is present. A real
regression test needs both ends in Ruby — a dummy FTP server in a Ruby
Thread, which is what ruby/spec's fixture is — and that is worth doing properly
rather than approximating with something that looks like proof and is not.

What does guard the property is the gate itself:
TestEveryBlockingSocketCallReleasesTheGVL fails in about a second if any of
these five is unwrapped again, and the backlog deletion means it also fails if
one silently stops being reported.

The measurement that would show this moving — the library/net-ftp sweep — is
deferred: this machine has been carrying a 1-minute load average between 23
and 86, and a per-file timeout measured under that cannot tell a hang from a slow
file. Reporting a number taken there would be worse than reporting none.

Refs #916, #921, #924

(#916 stays open on purpose — the measurement has not moved. #921 and #924 are
the merged pull requests this one continues.)

🤖 Generated with Claude Code

…hrinks

#921 bounded the control connection and #924 released the GVL around it. Both
left the DATA connection holding it, and the gate #924 added said so in its
seeded backlog -- four entries I wrote down rather than fixed, so the gate
could land without a sprawling change.

This is the change. dialData, acceptData, sendData, retrlines and retrbinary
now run their blocking call inside ftpIO, and the four entries are DELETED from
testdata/gvl-blocking-backlog.txt. That list may only shrink, and an entry that
stops reporting fails the gate too, so a fix cannot leave its allowance behind.

⛔ acceptData is the one #916 is actually about. In ACTIVE mode the client
listens and the server dials back, so a Ruby program that serves itself has
both ends in one process -- and holding the GVL in that Accept parks every
other Ruby thread, INCLUDING the one that would make the call it is waiting
for. ruby/spec's net-ftp fixtures are exactly that shape, and their
spec_helper.rb puts Net::FTP into active mode (see #916's comment: rbgo has no
Net::FTP.default_passive, so today that helper is skipped in silence and the
suite never reaches this path at all).

Only the blocking call goes inside each closure. ftpConnErr raises, and a panic
crossing a released GVL would unwind without re-acquiring it, so every raise
stays outside.

⛔ NO BEHAVIOURAL TEST, AND THE REASON IS NOT LAZINESS. The in-repo FTP test
server is Go-side, so it cannot reproduce this deadlock at all: nothing about
rbgo's emulated GVL blocks a Go goroutine, and a test built on it would pass
whether or not the fix is present. A real regression test needs BOTH ends in
Ruby -- a dummy FTP server in a Ruby Thread, which is what ruby/spec's fixture
is -- and that is worth doing properly rather than approximating with something
that looks like proof and is not.

What does guard the property is the gate: TestEveryBlockingSocketCallReleasesTheGVL
fails in about a second if any of these five is unwrapped again, and the
backlog deletion means it also fails if one silently stops being reported.

The measurement that would show this moving -- the library/net-ftp sweep -- is
deferred: this machine has been carrying a 1-minute load average between 23 and
86, and a per-file timeout measured under that cannot tell a hang from a slow
file.

Refs #916

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 94fb102 into main Nov 13, 2026
26 checks passed
@tannevaled
tannevaled deleted the ftp-data-gvl branch November 13, 2026 14:20
tannevaled added a commit that referenced this pull request Nov 15, 2026
…938)

Every figure here was already measured and merged; none of it needed a new run.
What it needed was someone to ask which published sentences the merges had made
false, which is a different question from "what did we fix".

## The four

1. `library/socket` **8 killed -> 7**, and the headline **75 -> 9** is **75 -> 8**.
   #935 measured it in the sentence "nine files killed by the clock across
   `library/socket` + `library/timeout` before, eight after".

2. A withdrawn figure. The README said the remaining hangs were "275 passing
   against MRI's 371". That control scored **7** on those 190 files, because
   `library/socket/spec_helper.rb` calls `MSpec.enable_feature` and the shim
   defined no `MSpec` (#936). A number counted against a broken control is not a
   smaller number, it is not a number; it is removed rather than adjusted.

3. The corpus map's "the rest is concentrated: matrix 233, bigdecimal 180, ..."
   list was compiled while `library/socket` contributed a flat **zero on both
   sides**. Repaired, socket is MRI 1790 / rbgo 224 -- about **-1560**, larger
   than every module on that list put together (#937). The two totals are
   deliberately **not summed**: different runs, different MRI patch releases
   (4.0.7 and 4.0.6), and a sum across two measurements is how two errors cancel.
   The published gap is marked a **floor** instead.

4. `docs/SHARED-ENGINES.md` still concluded, at the top of its `net-ftp` section,
   that **"#916 is not closed, because the measurement did not move ... still 39
   killed"**, and told the reader `net-ftp` was not available for extraction.
   That was true on 2026-11-13 and is the opposite of the outcome: #927 released
   the GVL on the **data** connection and #928 added `Net::FTP.default_passive`,
   and net-ftp is **0 killed, -128**. The section now records why the inference
   was wrong -- "the fix did not move the measurement" was read as "the fix is
   incomplete" when the mechanism was wired at only some of its doors -- and
   lifts the extraction blocker while naming the -128 the new home inherits.

## Why a doc gate did not catch any of them

Seven README drift gates run in this repository's root package and all seven are
green both before and after this change: they check the conformance total
against `BASELINE`, the Ruby version against the runtime, the Go floor, the
numeric surface, the wasm lane, and the shim line numbers the prose cites. Not
one of them looks at a figure in a dated prose table, because there is nothing
to compare such a figure against -- the run that produced it is gone.

That is the honest limit, and it is stated in the README rather than papered
over: the dated map is explicitly "not a gate", and the only number CI enforces
is per-file. The gates caught the thing they can catch -- when a 22-line
insertion moved two cited line numbers, every lane went red.

Refs #916, #933, #935, #936, #937

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant