Repository navigation
Stage 4: integrate reserved toolbar handoffs, job control, and resize recovery - #1761
Conversation
Keep nested prompts and selection on the shared reservation while transferring input ownership. Release physical rows before job-control suspension and before measuring the first pager frame, then reacquire using current terminal geometry. Restore bindings and terminal state after interrupted prompts, failed pager teardown, and reacquisition errors. Add transition regressions for completion, secrets, typeahead, resize, fallback, and signals. Validation: 2647 passed, 6 skipped; make check and make docs-test pass. Acceptance and dynamic harness gates pass at 12, 24, and 40 rows. Real-terminal and Windows qualification remain open.
Defer clearing the main-screen band until the first prepared pager frame is ready to commit, while exposing full terminal geometry during layout. Clear retained cells before managed output, external handoffs, recovery, or shutdown so they cannot leak into history. Add regressions for startup visibility and deferred cleanup. Validation: 2652 passed, 6 skipped; make check and make docs-test pass.
Resolve SIGTSTP conditionally before scheduling job-control suspension, preserving POSIX behavior without referencing a signal absent from Windows. Verified with mypy --platform win32 and the full test suite.
…hat raises an exception
Let terminal-attached POSIX pipelines share cmd2's process group so Ctrl-Z and fg transfer input ownership together. Avoid forwarding SIGINT back to that group, while retaining isolated groups for nonterminal pipelines and Windows. Add real PTY coverage for repeated suspend/resume, shell input, resizing, and command recovery, plus process-group and SIGINT regression tests.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## reserved_row_toolbar #1761 +/- ##
========================================================
+ Coverage 99.75% 99.82% +0.06%
========================================================
Files 35 35
Lines 8153 9059 +906
========================================================
+ Hits 8133 9043 +910
+ Misses 20 16 -4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
Bash 5.1 and later turn bracketed paste off with "\x1b[?2004l\r" before running a command, so the reply to "stty size" follows a bare carriage return rather than "\r\n". The test's predicate required the latter and timed out on Linux CI while passing on macOS, whose bash 3.2 has no bracketed paste. Match the reply at any line boundary instead.
The helper pager announced readiness and resumption with print. Its SIGCONT handler could therefore run while the main thread was still inside the PAGER_READY print, since the test stops the job the instant that marker appears, and Python raised "reentrant call inside <_io.BufferedWriter>". The pager died with a traceback whose source line contained PAGER_RESUMED, which satisfied the first wait by accident, and the second timed out. Write the markers with os.write, which is safe from a signal handler, and match them as whole lines so a traceback can never satisfy the wait.
…paths Three paths had no test: declining to suspend where the platform or the stop signal is missing, refusing a nested session whose layout has no toolbar window, and invalidating a nested prompt's bridge when the terminal is handed to a guest program. Each new test fails if its line is removed.
…control test The shell takes the terminal back as soon as its direct child stops, but the helper pager, a grandchild, may not have processed its stop signal yet. It sits in a one-byte terminal read, and the tty read loop hands a woken reader any byte already queued before it checks for the pending stop, so the pager consumed the first keystroke of the probe command and the shell ran "rintf". Free-threaded builds under load widened the window enough to hit on CI. Wait until both cmd2 and the pager report the stopped state before typing.
|
@kmvanbrunt @bambu The flicker-free bottom bar is now in a more mature state where it is more completely integrated with I'll keep reviewing and testing. But if others have time for manual testing on other platforms use: uv run examples/getting_started.pyThe bottom bar should now be fully persistent unless you invoke a shell application which is a TUI and takes over the whole screen and it should remain flicker-free. Though if for some reason you shrink the terminal vertical height to a pathological level of only 2 rows, it will revert to a default Please report any bugs you find along with the following:
NOTE: This is not the "done" state. I'm anticipating a good deal of cleanup along with adding support for multiple reserved rows. |
…ad pipes A shell command piped to an interactive consumer, such as `shell git log | less`, hung: the child inherited the raw pipe descriptor, bypassing the writer that lends the terminal per write, so the consumer stopped on its first terminal access while the producer blocked on the full pipe. do_shell now spawns its child in the pipeline's process group and lends the terminal for the child's lifetime, so the consumer keeps the terminal and Ctrl-C and Ctrl-Z reach both processes as in a shell pipeline. If the pipeline exits before the child can join its group, the child runs in our own group as before. Because such a producer can outlive the consumer that led the group, ProcReader.send_sigint() falls back to the leader's pid as the group id once the leader has been reaped, rather than doing nothing. A pipe started off the main thread could not install job-control signal handlers and failed after Popen, leaving the child unreaped. The terminal pipeline path is now taken only on the main thread; elsewhere the pipeline keeps running in its own session.
Exercise ProcReader.terminal_group for readers without a terminal and for a finished pipeline, the shell command's retry in its own group when the pipeline's group is gone before it can join, and a PermissionError from Popen that has nothing to do with the join, which must still propagate.
A signal that interrupts a blocking pipe write, such as the job-control stop ProcReader relays for Ctrl-Z, returned a partial count, so PipelineWriter ended its lend and the terminal went back to cmd2's group for the instant before the next write re-lent it. A consumer that had just resumed a terminal read could hit that window and be stopped again with SIGTTIN, racing with the next Ctrl-Z in the job-control test on a loaded runner. Finish the buffer under a single lend instead. When a terminal condition times out, the job-control tests now also report the pty's foreground group and the state and wait channel of every process under the outer shell, so a silent transcript still shows which process was waiting for whom.
The watcher thread stopped cmd2's job by sending SIGTSTP to the whole process with os.kill(). A process-directed signal may be taken by any thread that does not block it, and only the main thread runs Python signal handlers, so when it went elsewhere while the main thread slept in a system call, the handler never ran: the pipeline stayed stopped, the terminal was already back with cmd2's group, and the outer shell never saw the job stop. CI caught this with a shell producer sleeping in waitpid, and the process listing showed the main thread still there with the pipeline stopped. Use pthread_kill() on the main thread, which queues the signal on that thread and interrupts its system call.
Only the main thread runs Python signal handlers, and the job-control stop the pipeline watcher relays can wake another thread, however it is addressed: CI on macOS showed the pipeline stopped, the terminal already returned, and cmd2's main thread still asleep in the shell command's waitpid, and the earlier Linux failures had it asleep in a pipe write. A thread that never returns from its system call never runs the handler, so the job is never suspended. Wait in short polls at both points instead, as wait_for_exit already does: PipelineWriter writes through a non-blocking pipe and polls for room, and the shell command waits in slices while its child belongs to a terminal pipeline. The job-control test now delivers the relay to the watcher thread itself in the two cases that failed, which reproduced the hang deterministically. The reserved-terminal pager test waits for the bar repainted after a handoff, as the command display renders on its own thread and can land it a frame later than the resume, which Windows CI hit.
Making the pipe non-blocking regressed shell commands piped to a terminal consumer: do_shell() hands the same open file description to the child, which writes to it directly and failed with EAGAIN once the pipe was full. Leave the descriptor blocking. PipelineWriter still returns to Python regularly, as the job-control relay requires, by polling for room and writing at most PIPE_BUF bytes at a time, which cannot block once the pipe reports it is writable. The job-control test's timeout diagnostics now cope with a sandbox that cannot run ps.
A toolbar callback that did not return within the shutdown timeout made toolbar.stop() raise, and the error escaped cmdloop(). Had it not, the next prompt would have refused to run on a terminal the display still held. Report the display that would not stop, as one that cannot start already is, and wait at the main prompt for it to release the terminal. Ctrl-C during that wait ends the loop.
When a terminal pipeline exits before a shell command can join its group, do_shell() retries in cmd2's own group. The terminal was still lent to the dead pipeline, so the command stopped with SIGTTIN on its first terminal read and the wait for it never returned. End the lend before retrying.
SIGTTOU was blocked on the main thread for the whole life of a terminal pipeline. A signal mask survives fork and exec, so every child started meanwhile, whether a shell producer or a subprocess run by command code, kept SIGTTOU blocked for life and could change terminal modes from the background. The consumer now owns the terminal only during a lend, so block SIGTTOU for the lend alone, and spawn a shell producer, which starts inside one, with it unblocked.
The test raced cmd2's fixed 0.2s startup lend: a pager that set its terminal modes after it ended was stopped with SIGTTOU and, on macOS, failed with EINTR. Skipping coverage made the pager start faster, but a busy free-threaded macOS runner still missed the window. Have the test application hold the startup check open until the pager reports, so the test checks that the pager owns the terminal throughout the check rather than how fast it starts. Record whether the pager already held the terminal, so a failure shows which side of the window it hit.
…ends _job_control saved app.suspend_to_background, the class's bound method, and on exit wrote it back as an instance attribute. The command display installs its thread-safe Ctrl-Z handler only where the application has none of its own, so after a fallback from reserved rendering, or a later legacy loop, Ctrl-Z during a command went back to upstream's handler and could leave the terminal cooked after fg. Delete the attribute instead when nothing had replaced the method.
do_shell() lends the terminal for as long as a shell producer runs, and a pipe write from another thread lends and returns meanwhile. The first lend to end cleared availability and took the terminal back, so the producer stopped with SIGTTIN on its next terminal read and nothing resumed it. Count lends, and hand the terminal back after the last.
The stuck display left the terminal in raw mode, where Ctrl-C is only a key, and the binding that would raise SIGINT runs on the event loop the toolbar callback is blocking. So the wait before the next prompt could not be interrupted, despite saying it could. Cook the terminal before waiting, so its driver sends SIGINT again. Test it with a real Ctrl-C on a pseudo-terminal rather than a simulated KeyboardInterrupt.
Waiting for a terminal pipeline now hands the terminal back, which can fail, for example after a hangup. _restore_output then left the dead pipeline current and _redirecting set, so ppaged() stopped paging and Ctrl-C went to a process group that was gone. Restore both in a finally.
The wait for a toolbar display that will not stop cooked the terminal and printed its notice before entering the try that catches Ctrl-C. With the terminal cooked, Ctrl-C can arrive at once, and on Linux CI it landed while the notice was still being written, escaping cmdloop() with a traceback. Cook the terminal and print the notice inside the try.
A pipe process always ran in its own session, so it never received the terminal. Piped to an interactive program such as less, Ctrl-Z and fg did not suspend and resume it together with cmd2, and the program could not take the keyboard as a shell's foreground job would. When a pipe's output goes to cmd2's controlling terminal and the pipe is started on the main thread, run the pipe process in a new process group and lend it the terminal as it starts, while cmd2 writes to it, and while cmd2 waits for it. Between writes the terminal returns to cmd2, so command code can still read the keyboard. ProcReader watches the group on a thread, relays its job-control stops to cmd2's job, and forwards Ctrl-C without signalling cmd2's own group again. A shell command piped to such a program joins the pipeline's job for as long as it runs. Pipes started off the main thread, or whose output cmd2 captures, still run in their own session. The terminal tests drive cmd2 under an interactive bash on a pseudo-terminal, using pyte to read the screen, and coverage now follows the applications those tests start. Extracted from the reserved-row toolbar work (PR #1761) without the toolbar.
|
Independent automated check of With the inherited TERM=dumb uv run python -Xutf8 -m pytest tests/test_prompt_toolkit_bridge.py::TestCursorPositionReports::test_cpr_uses_row_one_coordinate_contract --no-cov -n 0 -qIt fails because Could the fixtures that explicitly model a CPR-capable terminal declare that The focused pipeline-job-control/command-toolbar/pager run also passed all |
# Conflicts: # cmd2/cmd2.py # cmd2/command_toolbar.py # cmd2/reserved_toolbar.py # cmd2/utils.py # tests/test_cmd2.py # tests/test_pipeline_job_control.py # tests/test_utils.py
# Conflicts: # cmd2/cmd2.py
|
@Neallin-917 Test fixtures (609961e, on Production (dcbad7b): I agree the bridge's handling of cursor position reports doesn't need to change. While checking that, though, I found that mode selection did. It decided capability purely by backend identity, so A dumb terminal is now an unmet prerequisite, judged the way prompt-toolkit judges cursor position reports: from the output's own terminal type, or from Both changes are merged into this branch. |
Brings consolidated_toolbar's display error handling, unfinished-line handling in the legacy stdout proxy, and the locking documentation. The Ctrl-Z stop fix was already here; its comment and the test parametrized over both rendering modes are kept.
One Windows check left to close Stage 4Everything automated is green at Steps, in Windows Terminal with PowerShell:
To count rows exactly, run Please report the Windows build, Windows Terminal version, Python version, and for each of steps 3 to 7 either "as expected" or what you saw, with a screenshot of anything wrong. If you have time, the same sequence during |
…dy pending A redraw requested by command output stays marked pending until a frame commits, and the command display's suppressed frames never commit. A frame which then failed to prepare asked for a retry through the same coalescing request, which dropped it as a duplicate, although the scheduling that went with the earlier request was what had just run that frame. The built-in pager depends on that retry: its first frame enters the alternate screen, which cannot be recorded in a frame. In reserved mode the pager stayed blank until the next toolbar refresh, and with no refresh_interval, the default, until a key was pressed. Schedule the retry directly. A frame whose preparation keeps raising is retried only a few times, so a layout callback which always fails cannot become a redraw loop.
In reserved mode the pager's first erase deletes whole lines starting at the cursor's row. A command which wrote 'Results: ' without a newline and then called ppaged() had that line deleted before the pager drew anything. Move past it first, as is already done before a nested prompt.
The prompt documentation still said select() shows no toolbar, which is only true of legacy rendering now, and the changelog had no entries for what Stage 4 changes or for the two built-in pager fixes.
Final review of the Stage 4 diff (2026-11-14)Reviewed the diff against Fixed
Found and left open: the pager never enters the alternate screen in reserved modeCounting escape sequences for one pager session: legacy rendering sends one alternate-screen enter and one leave. Reserved rendering sends no enter and one leave, on this branch and on This is not introduced by this PR and I have not changed it here: a fix has to perform the switch outside a frame without undoing "keep the toolbar visible while the first page prepares", and it needs checking in real terminals, since pyte does not model the alternate screen. It is direct input for Stage 5's pager investigation. The harness's functional driver now reports it as a known limitation so it cannot be forgotten. Review findings not acted on, for a maintainer decision
|
Windows check: resultsResults for the one Windows check left, at the current head How it was run: a script opened real Windows Terminal windows with PowerShell, ran Environment
Steps
Both Terminal versions gave the same results for every step. Further passes, Windows Terminal 1.25
What happens in the half second after a resizeNot a failure, but worth knowing. Sampling the console every 10 to 15 ms after one-row resizes:
The delay matches prompt-toolkit polling the terminal size every 0.5 s by default, which cmd2 does not change. I have not confirmed that as the cause. Not checked
|
Summary
Complete Stage 4 of the reserved-row toolbar work: transfer terminal and input ownership correctly across nested prompts, paging, external applications, suspend/resume, and resize. This PR targets
reserved_row_toolbar, which already contains Stage 3; it retains the current one-row reservation and opt-in mode defaults.What changed
read_input(),read_secret(), andselect()borrow the existing reservation after the command reader stops. Temporary applications receive their own bridge, retain the current toolbar content/styles, and restore original bindings when they finish, are interrupted, or fall back. This avoids a second input reader and preserves completion, secret masking, and typeahead.SIGTSTPconditionally so Windows type checking and unsupported-platform behavior remain valid.help -v | less, Ctrl-Z, andfgcould leavelessrunning while cmd2 was stopped, allowing it to steal shell input. Avoid forwarding SIGINT back into cmd2's own group; retain isolated groups for nonterminal pipelines and the existing Windows process-group behavior.pyin the getting-started example and add an exception-raising script for manual testing. Add nested-input, pager, cleanup, polling, partial-output, and real-PTY job-control regressions.Review focus
The implementation retains the existing alternating main/command application lifecycle and serialized-output architecture. Review ownership restoration and failure ordering, especially nested suspension, alternate-screen exit, native Windows geometry, and partial output. The
ChoiceInput._create_application()integration and polling tests depend on prompt-toolkit internals; qualification remains at exactly 3.0.53. Broader dependency/backend qualification follows in Stage 8.Roadmap
The additional investigations requested after Stage 4 move the former Stage 5 release qualification to Stage 8.
reserved_row_toolbarvia #1758cmd2/toolbar/package; align focused tests with production modules; keep cross-module integration suites; separate mechanical moves from justified behavior-preserving refactorsValidation
At head
91463a27(2026-11-14), which also containsmainthrough 4.4.2 andconsolidated_toolbar(#1745) merged in throughreserved_row_toolbar:make check,make test(3,807 passed, 7 skipped) andmake docs-testpass on macOS. A Linux container run of the suite passes with none failing. PR CI is 30/30, including Windows and the free-threaded lanes.read_input/read_secret/select, the built-in pager, editor, shell commands, pipes, redirection, Ctrl-C, resize, unfinished lines of output, and failing or blocked toolbar callbacks. Reserved rendering: 97 passed with 5 known limitations (two-line toolbars, which are Stage 7; an emulator limitation around a pending line wrap; and the alternate-screen defect described in the review comment below). Legacy rendering: 102 passed.A final review of the diff produced three fixes on this branch (the built-in pager not appearing without a refresh timer, an unfinished line deleted by the pager, and documentation) and one defect left open for Stage 5: in reserved mode the pager never enters the alternate screen. Details are in the review comment of 2026-11-14.
What the merges brought into this branch since the last update:
main: 4.4.2 routedrun_editor()arounddo_shell(), which bypassed the toolbar suspension. The suspension now sits on_run_shell_command().get_bottom_toolbar()no longer floods the terminal with tracebacks in either rendering mode (478 from one command before, one notice now), and legacy rendering no longer erases lines of output left unfinished.Since that evidence was taken the head has moved to
712f353b, a merge ofreserved_row_toolbarwhich brings four more fixes from #1745 to how legacy rendering continues unfinished lines of output. It changescmd2/command_toolbar.pyand its tests only, and nothing in reserved rendering. At712f353b:make check,make test(3,822 passed, 7 skipped) andmake docs-testpass on macOS, PR CI is 30/30, and the functional driver's unfinished-line scenarios pass in both rendering modes. The observer gates and the Linux container were not rerun at this head.Maintainer-reported manual results at earlier heads: nested prompts on Mac and Windows; embedded paging and external tool/interpreter recovery; Mac job-control edge cases; clean shell return; and the exact pipeline sequence in Mac iTerm2.
Native Windows check: done (2026-11-14, at
91463a27). The "Exact minimum-height recovery" row of the checklist below was run in Windows Terminal 1.25.3131.0 and Windows Terminal Preview 1.26.3133.0 on Windows 11 build 26300.9974 with Python 3.14.10. Every step was as expected, empty and with partly typed input, and while a command was running; the 2 → 3-row resize settled correctly more than 20 times. The full results are in the comment of that date. A resize takes up to half a second to show, which is prompt-toolkit's polling of the terminal size: on Windows that poll, every 0.5 seconds by default, is the only way it learns of a resize. Not covered: dragging with a real mouse, the scrollback after shrinking, and terminals other than Windows Terminal, which belong to Stage 8.Manual reviewer checklist
Run from the repository root with
uv run python examples/stage4_manual.py.stage4_manual.pyis a super-set of thegetting_started.pyapplication which adds commands for testing nested prompts viaread_input,read_secret, andselect. The example uses AUTO with a live timestamp. Record the commit, OS, terminal/version, shell, Python and prompt-toolkit versions, actual selected backend/mode, and measured viewport dimensions. AUTO can fall back: a visible native toolbar alone does not prove reserved rendering.scripts/toolbar_inventory.pycan help identify the environment; inspect its usage first because it includes interactive probes.Perform applicable checks on Mac/Linux and Windows Terminal. POSIX Ctrl-Z/fg is not expected in PowerShell. External TUIs intentionally hide the reserved toolbar while they own the terminal; it must return when they exit. Exact physical row counts matter, especially at the minimum height.
echo helloandwork 5; edit input and use completion/historynested; use Tab and Up, resize, then accept; repeat and press Ctrl-C or EOFsecretwith dummy text only; runchooseand use arrows/Enter; repeat with cancellation and resizenested, quickly enteralphafollowed by a newline andecho afterfollowed by a newlinecat cmd2/cmd2.py; use Space/PgDn, b/PgUp,/search,n, resize andqcat examples/scripts/raise_exception.pyin a normal-height terminalhelp -v | less; resize inside it, navigate and quit withqeditwith a disposable file and a configured editor; resize and exitpy/ipy, print text and exit normally; run a normalrun_pyscriptandrun_pyscript examples/scripts/raise_exception.pyhandoff, thenhandoff fail; resize at the guest input and complete itwork 10, press Ctrl-Z; run a shell command, resize,fg; repeatfginside the embedded pager; navigate and quit afterwardhelp -v | less, Ctrl-Z; type a complete shell command such asprintf 'shell owns input\n';fg;q; thenecho afterin cmd2quiet, thenpartial; resize repeatedly through 2/3/4 rows during the pausePARTIALENDremains intact and output never lands in the toolbar bandOptional commands for managed-input and handoff checks
Run
uv run python examples/stage4_manual.py. These commands execute inside the normal command lifecycle. Callingread_input()from the externalpyshell would instead exercise an already-suspended handoff.Fault-injection cases such as failing pager erasure, failed reacquisition and reservation abandonment are covered by automated tests; reviewers should inspect those outcomes rather than make production callbacks fail ad hoc. Record PASS/FAIL/not available per manual row, with terminal details and screenshots/diagnostic traces for failures. The broader release terminal matrix and configurable multiple reserved rows remain later-stage work.