MySQL editor stop races the server to the next statement (flaky consumer_gone_kills_the_rest_of_the_batch) #3766

What happened?

editor_session::consumer_gone_kills_the_rest_of_the_batch failed once in the linux / Docker tests (tablepro-driver-mysql, mysql-8.0-mariadb-10.11) job on #3714 and passed on the re-run. It passed on that PR's earlier run and on #3727's runs. The failure, on the MySQL flavour:

thread 'editor_session::consumer_gone_kills_the_rest_of_the_batch' (29682) panicked at crates/drivers/mysql/tests/integration/editor_session.rs:122:9:
assertion `left == right` failed: Mysql
  left: 0
 right: 1

The asserts at lines 117 and 118 passed: the run ended Cancelled and the session was Ready. Line 122 then found t empty, so the DELETE FROM t behind the SELECT ran after the reader had left.

The test and the driver both rely on timing:

• The script goes to the server whole, as one multi-statement batch (linux/crates/drivers/mysql/src/editor_run.rs:36). When the SELECT ends, the server starts the DELETE on its own.
• The stop is KILL QUERY on a control connection that the first stop opens: connect, log in, then the KILL (linux/crates/drivers/mysql/src/cancel.rs:38). The KILL names the connection, not the statement (cancel.rs:90).
• Only timing makes the KILL land while the SELECT is still running. The test relies on 100 MB of rows not fitting in the socket buffers (linux/crates/drivers/mysql/tests/integration/editor_session.rs:15). The driver relies on reading nothing until the KILL lands (linux/crates/drivers/mysql/src/editor_session.rs:191).
• The whole stop gets cancel_grace: 1 s in tests (linux/crates/test-support/src/conformance/test_liveness.rs:13), 5 s in the app (linux/crates/core/src/liveness_policy.rs:31).
• When the KILL errors, or lands after the SELECT has ended, run_script reads the batch to its end (editor_run.rs:51). If that ends within the grace, the run reports Cancelled with the session ready (editor_session.rs:204), the same as a stop that landed, and the DELETE has run.

So this is a race in the driver, not only in the test. Stop takes the same path (editor_session.rs:185), so a script the user stopped, or whose tab closed, can run its later statements and still read as cancelled.

Steps to reproduce

Not reproduced on demand yet. From linux/, run the test in a loop while the machine is under CPU load:

TABLEPRO_TEST_MYSQL=8.0 TABLEPRO_TEST_MARIADB=10.11 cargo nextest run --profile ci -p tablepro-driver-mysql --run-ignored only -E 'test(=editor_session::consumer_gone_kills_the_rest_of_the_batch)'

Expected behavior

Once the reader is gone, no statement after the running one starts, however slow the runner is. A larger grace or a bigger SELECT only moves the window.

• Send an editor script one statement per round trip, as batches already does for a script with DELIMITER lines (editor_run.rs:40). "No later batch is sent" (editor_run.rs:54) then covers the DELETE, and the KILL has only the running statement to stop.
• A run whose KILL did not land does not report the same result as one that did.
• The test then needs no 100 MB SELECT: the reader leaves at the first rows, and the DELETE must never reach the server.

Done when: no assert in the test depends on socket buffer size, cancel_grace or runner load, and it passes on both flavours.

Database type

MySQL / MariaDB

TablePro version

main at b8db6fb.

Platform

Linux

OS version & hardware

GitHub-hosted runner, MySQL 8.0 and MariaDB 10.11 containers.

Screenshots / Logs

Failing job: https://github.com/TableProApp/TablePro/actions/runs/39573071989/job/119136002401 (attempt 1 of run 39573071989; attempt 2 passed).