Skip to content

Stage 4: integrate reserved toolbar handoffs, job control, and resize recovery - #1761

Merged
tleonhardt merged 56 commits into
reserved_row_toolbarfrom
stage4-handoffs-integration
Nov 15, 2026
Merged

tleonhardt merged 56 commits into
reserved_row_toolbarfrom
stage4-handoffs-integration

Conversation

@tleonhardt

@tleonhardt tleonhardt commented Sep 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Complete Stage 4 of the reserved-row toolbar work: transfer terminal and input ownership correctly across nested prompts, paging, external applications, suspend/resume, and resize. This PR targets reserved_row_toolbar, which already contains Stage 3; it retains the current one-row reservation and opt-in mode defaults.

What changed

  • Managed nested input: read_input(), read_secret(), and select() borrow the existing reservation after the command reader stops. Temporary applications receive their own bridge, retain the current toolbar content/styles, and restore original bindings when they finish, are interrupted, or fall back. This avoids a second input reader and preserves completion, secret masking, and typeahead.
  • Embedded pager ownership and loading: release the reservation before the first pager layout measures available space, expose full physical geometry, and avoid subtracting the toolbar height twice when deciding whether output fits. Keep the existing main-screen toolbar visible while preparing a large pager's first frame; clear it when that frame commits or when intervening output/external ownership requires it. Restore the main-screen reservation after leaving the pager, including cleanup when exit-time erasure fails.
  • Nested and external handoffs: make release/reacquisition idempotent across nested suspension and alternate-screen ownership. Reacquire from fresh geometry, invalidate stale prompt/output accounting, and restore compatibility bindings when acquisition fails. If both the guest and cleanup fail, preserve the guest's exception and retain the cleanup error for diagnostics.
  • POSIX job control: release rows inside prompt-toolkit's cooked-mode/input-detached handoff before stopping the foreground job, then reacquire on resume. Suspending while a pager already owns the screen does not install a reservation inside its alternate buffer. Look up SIGTSTP conditionally so Windows type checking and unsupported-platform behavior remain valid.
  • Interactive pipelines: terminal-attached POSIX pipelines now share cmd2's foreground process group. Previously help -v | less, Ctrl-Z, and fg could leave less running while cmd2 was stopped, allowing it to steal shell input. Avoid forwarding SIGINT back into cmd2's own group; retain isolated groups for nonterminal pipelines and the existing Windows process-group behavior.
  • Minimum-height recovery: include the reservation generation in the painter's cached-band identity. A 3 → 2 → 3 resize can erase the old cells and reacquire identical coordinates; the first new paint must redraw the entire band rather than reuse that stale cache. Unchanged content remains a no-op within a stable generation.
  • Examples and regression coverage: enable py in the getting-started example and add an exception-raising script for manual testing. Add nested-input, pager, cleanup, polling, partial-output, and real-PTY job-control regressions.

Review focus

The implementation retains the existing alternating main/command application lifecycle and serialized-output architecture. Review ownership restoration and failure ordering, especially nested suspension, alternate-screen exit, native Windows geometry, and partial output. The ChoiceInput._create_application() integration and polling tests depend on prompt-toolkit internals; qualification remains at exactly 3.0.53. Broader dependency/backend qualification follows in Stage 8.

Roadmap

The additional investigations requested after Stage 4 move the former Stage 5 release qualification to Stage 8.

Stage Work Status / exit criterion
3 Ordinary prompt/command lifecycle, mode selection, serialization, stable toolbar and history Completed and merged into reserved_row_toolbar via #1758
4 Nested prompts, pager/external handoffs, job control, resizing and cleanup This PR; CI, review, final evidence and the native Windows check of the 2 → 3-row transition are complete. Ready to merge
5 Spike: is the embedded pager still useful and worth maintaining? Compare capabilities, startup latency, portability and maintenance cost; review a keep/simplify/replace/remove recommendation before changing behavior
6 Toolbar package organization, test naming, and simplification Investigate an internal cmd2/toolbar/ package; align focused tests with production modules; keep cross-module integration suites; separate mechanical moves from justified behavior-preserving refactors
7 Configurable 0–3 reserved toolbar lines 0 disables; 1 preserves current behavior; 2/3 use explicit newlines for subsequent reserved rows. Clip each line at terminal width with visible truncation; verify geometry, fallback and every ownership transition. Review mode consolidation separately rather than assuming removal of legacy/forced controls
8 Final integrated release qualification Qualify the final code across required terminals/backends and dependency versions; record history/flicker evidence, compatibility and rollback. Default promotion remains a separate maintainer decision

Validation

At head 91463a27 (2026-11-14), which also contains main through 4.4.2 and consolidated_toolbar (#1745) merged in through reserved_row_toolbar:

  • make check, make test (3,807 passed, 7 skipped) and make docs-test pass on macOS. A Linux container run of the suite passes with none failing. PR CI is 30/30, including Windows and the free-threaded lanes.
  • Observer harness: 23/23 controls, 24/24 mutations discriminate, 7/7 history self-tests.
  • Acceptance and dynamic gates recaptured at this head at 12, 24 and 40 rows: all PASS. Zero toolbar disappearances, zero erases addressed to the band, complete exactly-once history, no toolbar rows in history. The legacy control fails continuity as designed and AUTO selects reserved rendering.
  • A new pseudo-terminal functional driver checks the settled screen for output ordering, typeahead, read_input/read_secret/select, the built-in pager, editor, shell commands, pipes, redirection, Ctrl-C, resize, unfinished lines of output, and failing or blocked toolbar callbacks. Reserved rendering: 97 passed with 5 known limitations (two-line toolbars, which are Stage 7; an emulator limitation around a pending line wrap; and the alternate-screen defect described in the review comment below). Legacy rendering: 102 passed.
  • The exact minimum-height sequence from the checklist below (24 → 2 → 3 → 4 → 24, empty and with partly typed input) is one of that driver's scenarios and passes in both modes on macOS. That is emulated POSIX evidence, not Windows evidence.
  • Earlier results still stand: the in-memory mutation removing the generation from the painter cache fails both painter regressions, and the real-pty pipeline tests exercise repeated Ctrl-Z/fg, shell input, resizing, quit/interrupt and toolbar restoration.

A final review of the diff produced three fixes on this branch (the built-in pager not appearing without a refresh timer, an unfinished line deleted by the pager, and documentation) and one defect left open for Stage 5: in reserved mode the pager never enters the alternate screen. Details are in the review comment of 2026-11-14.

What the merges brought into this branch since the last update:

  • From main: 4.4.2 routed run_editor() around do_shell(), which bypassed the toolbar suspension. The suspension now sits on _run_shell_command().
  • From Consolidated toolbar during command execution #1745: a raising get_bottom_toolbar() no longer floods the terminal with tracebacks in either rendering mode (478 from one command before, one notice now), and legacy rendering no longer erases lines of output left unfinished.

Since that evidence was taken the head has moved to 712f353b, a merge of reserved_row_toolbar which brings four more fixes from #1745 to how legacy rendering continues unfinished lines of output. It changes cmd2/command_toolbar.py and its tests only, and nothing in reserved rendering. At 712f353b: make check, make test (3,822 passed, 7 skipped) and make docs-test pass on macOS, PR CI is 30/30, and the functional driver's unfinished-line scenarios pass in both rendering modes. The observer gates and the Linux container were not rerun at this head.

Maintainer-reported manual results at earlier heads: nested prompts on Mac and Windows; embedded paging and external tool/interpreter recovery; Mac job-control edge cases; clean shell return; and the exact pipeline sequence in Mac iTerm2.

Native Windows check: done (2026-11-14, at 91463a27). The "Exact minimum-height recovery" row of the checklist below was run in Windows Terminal 1.25.3131.0 and Windows Terminal Preview 1.26.3133.0 on Windows 11 build 26300.9974 with Python 3.14.10. Every step was as expected, empty and with partly typed input, and while a command was running; the 2 → 3-row resize settled correctly more than 20 times. The full results are in the comment of that date. A resize takes up to half a second to show, which is prompt-toolkit's polling of the terminal size: on Windows that poll, every 0.5 seconds by default, is the only way it learns of a resize. Not covered: dragging with a real mouse, the scrollback after shrinking, and terminals other than Windows Terminal, which belong to Stage 8.

Manual reviewer checklist

Run from the repository root with uv run python examples/stage4_manual.py. stage4_manual.py is a super-set of the getting_started.py application which adds commands for testing nested prompts via read_input, read_secret, and select. The example uses AUTO with a live timestamp. Record the commit, OS, terminal/version, shell, Python and prompt-toolkit versions, actual selected backend/mode, and measured viewport dimensions. AUTO can fall back: a visible native toolbar alone does not prove reserved rendering. scripts/toolbar_inventory.py can help identify the environment; inspect its usage first because it includes interactive probes.

Perform applicable checks on Mac/Linux and Windows Terminal. POSIX Ctrl-Z/fg is not expected in PowerShell. External TUIs intentionally hide the reserved toolbar while they own the terminal; it must return when they exit. Exact physical row counts matter, especially at the minimum height.

Check How to exercise it Expected result
Ordinary prompts and commands Press Enter repeatedly; run echo hello and work 5; edit input and use completion/history Stable reserved band, live timestamp, usable prompt, no duplicated/lost input or output
Managed text input In the optional test app below run nested; use Tab and Up, resize, then accept; repeat and press Ctrl-C or EOF Band remains reserved; completion/history work; only the active prompt consumes input; main prompt recovers
Secret input and selection Run secret with dummy text only; run choose and use arrows/Enter; repeat with cancellation and resize Secret remains masked; selection works; toolbar and main input recover without stale bindings
Typeahead At nested, quickly enter alpha followed by a newline and echo after followed by a newline Accepted input and following command are each consumed once by the correct owner
Large embedded pager cat cmd2/cmd2.py; use Space/PgDn, b/PgUp, / search, n, resize and q No prolonged blank-toolbar interval while loading; pager uses full geometry with its native bar; main reserved band and prompt return on exit
Short pager output cat examples/scripts/raise_exception.py in a normal-height terminal Short output prints normally without an unnecessary full-screen pager
External pager On POSIX run help -v | less; resize inside it, navigate and quit with q External pager owns the full terminal; toolbar returns at current dimensions with usable input and intact output
External editor Use edit with a disposable file and a configured editor; resize and exit Full terminal available to the editor; reservation restored afterward
Embedded Python/IPython and scripts Run py/ipy, print text and exit normally; run a normal run_pyscript and run_pyscript examples/scripts/raise_exception.py External interpreter ownership is intentional; success and exception paths restore toolbar/input. Record unavailable optional IPython explicitly
Nested public suspension Optional app: handoff, then handoff fail; resize at the guest input and complete it No early reacquisition at an inner exit; toolbar restored after outer exit, including failure
POSIX prompt/command suspend At an idle or partially typed prompt, and during work 10, press Ctrl-Z; run a shell command, resize, fg; repeat Shell has full scrolling/input while stopped; resume restores correct geometry and preserves editable input/output
POSIX pager suspend Repeat Ctrl-Z, shell command, resize and fg inside the embedded pager; navigate and quit afterward No reservation installed into a guest screen; pager remains usable and restores main-screen toolbar on exit
POSIX pipeline ownership regression help -v | less, Ctrl-Z; type a complete shell command such as printf 'shell owns input\n'; fg; q; then echo after in cmd2 Shell receives all keystrokes; less and cmd2 stop/resume together; reserved bar and command input return. Repeat the cycle
Exact minimum-height recovery Idle main prompt: measure and resize 24 → 2 → 3 → 4 → 24; repeat with partially typed input. Also start at 2 or 3 rows and grow At 2 rows reservation releases; at 3 there are 2 usable rows plus a fully painted toolbar on row 3; 4+ remains correct. No Enter/Ctrl-L needed to repair it
Quiet and partial-output resize Optional app: quiet, then partial; resize repeatedly through 2/3/4 rows during the pause Automatic resize detection; full bar returns when eligible; PARTIALEND remains intact and output never lands in the toolbar band
Width and handoff recovery Narrow/widen at main/nested prompts and in a pager; repeat the height-boundary sequence after returning from an external tool Current width/viewport used, no stale band, prompt overlap, missing input or permanent fallback
Compatibility and disabled mode In a disposable copy of the example change constructor mode AUTO to LEGACY, then OFF; repeat basic commands and paging. Test RESERVED on a qualified backend Legacy remains usable; OFF has no toolbar; RESERVED really exercises the new path. Unsupported forced prerequisites report an error rather than pretending to qualify
History and final shell cleanup After the transitions, scroll through history; quit, type a shell command and generate more than one screen of output; relaunch No toolbar leakage or unexplained lost/duplicated command output; normal shell echo/cursor and full-height scrolling; fresh loop behaves normally

Optional commands for managed-input and handoff checks

Run uv run python examples/stage4_manual.py. These commands execute inside the normal command lifecycle. Calling read_input() from the external py shell would instead exercise an already-suspended handoff.

import time
from getting_started import BasicApp


class ManualApp(BasicApp):
    def do_nested(self, arg):
        value = self.read_input(
            "Value> ", choices=["alpha", "beta"], history=["alpha", "beta"]
        )
        self.poutput(repr(value))

    def do_secret(self, arg):
        self.read_secret("Dummy secret> ")
        self.poutput("Secret accepted; value not displayed")

    def do_choose(self, arg):
        self.poutput(self.select(["alpha", "beta"], "Choose> "))

    def do_quiet(self, arg):
        time.sleep(10)

    def do_partial(self, arg):
        self.stdout.write("PARTIAL")
        self.stdout.flush()
        time.sleep(10)
        self.stdout.write("END\n")
        self.stdout.flush()

    def do_handoff(self, arg):
        with self.suspend_bottom_toolbar():
            with self.suspend_bottom_toolbar():
                input("Guest owns terminal; resize, then Enter> ")
            if arg.strip() == "fail":
                raise RuntimeError("Intentional guest failure")


if __name__ == "__main__":
    ManualApp().cmdloop()

Fault-injection cases such as failing pager erasure, failed reacquisition and reservation abandonment are covered by automated tests; reviewers should inspect those outcomes rather than make production callbacks fail ad hoc. Record PASS/FAIL/not available per manual row, with terminal details and screenshots/diagnostic traces for failures. The broader release terminal matrix and configurable multiple reserved rows remain later-stage work.

Keep nested prompts and selection on the shared reservation while transferring input ownership. Release physical rows before job-control suspension and before measuring the first pager frame, then reacquire using current terminal geometry.

Restore bindings and terminal state after interrupted prompts, failed pager teardown, and reacquisition errors. Add transition regressions for completion, secrets, typeahead, resize, fallback, and signals.

Validation: 2647 passed, 6 skipped; make check and make docs-test pass. Acceptance and dynamic harness gates pass at 12, 24, and 40 rows. Real-terminal and Windows qualification remain open.
Defer clearing the main-screen band until the first prepared pager frame is ready to commit, while exposing full terminal geometry during layout. Clear retained cells before managed output, external handoffs, recovery, or shutdown so they cannot leak into history.

Add regressions for startup visibility and deferred cleanup. Validation: 2652 passed, 6 skipped; make check and make docs-test pass.
Resolve SIGTSTP conditionally before scheduling job-control suspension, preserving POSIX behavior without referencing a signal absent from Windows. Verified with mypy --platform win32 and the full test suite.
Let terminal-attached POSIX pipelines share cmd2's process group so Ctrl-Z and fg transfer input ownership together. Avoid forwarding SIGINT back to that group, while retaining isolated groups for nonterminal pipelines and Windows.

Add real PTY coverage for repeated suspend/resume, shell input, resizing, and command recovery, plus process-group and SIGINT regression tests.
@codecov

codecov Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.82%. Comparing base (0c8c09f) to head (712f353).
⚠️ Report is 131 commits behind head on reserved_row_toolbar.

Additional details and impacted files
@@                   Coverage Diff                    @@
##           reserved_row_toolbar    #1761      +/-   ##
========================================================
+ Coverage                 99.75%   99.82%   +0.06%     
========================================================
  Files                        35       35              
  Lines                      8153     9059     +906     
========================================================
+ Hits                       8133     9043     +910     
+ Misses                       20       16       -4     
Flag Coverage Δ
unittests 99.82% <100.00%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Bash 5.1 and later turn bracketed paste off with "\x1b[?2004l\r" before
running a command, so the reply to "stty size" follows a bare carriage
return rather than "\r\n". The test's predicate required the latter and
timed out on Linux CI while passing on macOS, whose bash 3.2 has no
bracketed paste. Match the reply at any line boundary instead.
The helper pager announced readiness and resumption with print. Its SIGCONT
handler could therefore run while the main thread was still inside the
PAGER_READY print, since the test stops the job the instant that marker
appears, and Python raised "reentrant call inside <_io.BufferedWriter>".
The pager died with a traceback whose source line contained PAGER_RESUMED,
which satisfied the first wait by accident, and the second timed out.

Write the markers with os.write, which is safe from a signal handler, and
match them as whole lines so a traceback can never satisfy the wait.
…paths

Three paths had no test: declining to suspend where the platform or the
stop signal is missing, refusing a nested session whose layout has no
toolbar window, and invalidating a nested prompt's bridge when the terminal
is handed to a guest program. Each new test fails if its line is removed.
@tleonhardt tleonhardt self-assigned this Sep 12, 2026
…control test

The shell takes the terminal back as soon as its direct child stops, but the
helper pager, a grandchild, may not have processed its stop signal yet. It
sits in a one-byte terminal read, and the tty read loop hands a woken reader
any byte already queued before it checks for the pending stop, so the pager
consumed the first keystroke of the probe command and the shell ran "rintf".
Free-threaded builds under load widened the window enough to hit on CI.

Wait until both cmd2 and the pager report the stopped state before typing.
@tleonhardt

tleonhardt commented Sep 12, 2026 •

Copy link
Copy Markdown
Member Author

@kmvanbrunt @bambu The flicker-free bottom bar is now in a more mature state where it is more completely integrated with cmd2. Compared to the version yesterday evening, <Ctrl>+Z process suspesion and fg resume work properly, terminal resizing should be robust, it works with nested prompts and various other edge cases bugs have been fixed.

I'll keep reviewing and testing. But if others have time for manual testing on other platforms use:

uv run examples/getting_started.py

The bottom bar should now be fully persistent unless you invoke a shell application which is a TUI and takes over the whole screen and it should remain flicker-free. Though if for some reason you shrink the terminal vertical height to a pathological level of only 2 rows, it will revert to a default prompt-toolkit toolbar; though when you grow the size it will return to the reserved toolbar.

Please report any bugs you find along with the following:

  • Operating system
  • Termainl program
  • Python version
  • Steps to reproduce

NOTE: This is not the "done" state. I'm anticipating a good deal of cleanup along with adding support for multiple reserved rows.

…ad pipes

A shell command piped to an interactive consumer, such as `shell git log | less`,
hung: the child inherited the raw pipe descriptor, bypassing the writer that
lends the terminal per write, so the consumer stopped on its first terminal
access while the producer blocked on the full pipe. do_shell now spawns its
child in the pipeline's process group and lends the terminal for the child's
lifetime, so the consumer keeps the terminal and Ctrl-C and Ctrl-Z reach both
processes as in a shell pipeline. If the pipeline exits before the child can
join its group, the child runs in our own group as before.

Because such a producer can outlive the consumer that led the group,
ProcReader.send_sigint() falls back to the leader's pid as the group id once
the leader has been reaped, rather than doing nothing.

A pipe started off the main thread could not install job-control signal
handlers and failed after Popen, leaving the child unreaped. The terminal
pipeline path is now taken only on the main thread; elsewhere the pipeline
keeps running in its own session.
Exercise ProcReader.terminal_group for readers without a terminal and for a
finished pipeline, the shell command's retry in its own group when the
pipeline's group is gone before it can join, and a PermissionError from Popen
that has nothing to do with the join, which must still propagate.
A signal that interrupts a blocking pipe write, such as the job-control stop
ProcReader relays for Ctrl-Z, returned a partial count, so PipelineWriter ended
its lend and the terminal went back to cmd2's group for the instant before the
next write re-lent it. A consumer that had just resumed a terminal read could
hit that window and be stopped again with SIGTTIN, racing with the next Ctrl-Z
in the job-control test on a loaded runner. Finish the buffer under a single
lend instead.

When a terminal condition times out, the job-control tests now also report the
pty's foreground group and the state and wait channel of every process under
the outer shell, so a silent transcript still shows which process was waiting
for whom.
The watcher thread stopped cmd2's job by sending SIGTSTP to the whole process
with os.kill(). A process-directed signal may be taken by any thread that does
not block it, and only the main thread runs Python signal handlers, so when it
went elsewhere while the main thread slept in a system call, the handler never
ran: the pipeline stayed stopped, the terminal was already back with cmd2's
group, and the outer shell never saw the job stop. CI caught this with a shell
producer sleeping in waitpid, and the process listing showed the main thread
still there with the pipeline stopped. Use pthread_kill() on the main thread,
which queues the signal on that thread and interrupts its system call.
Only the main thread runs Python signal handlers, and the job-control stop the
pipeline watcher relays can wake another thread, however it is addressed: CI
on macOS showed the pipeline stopped, the terminal already returned, and cmd2's
main thread still asleep in the shell command's waitpid, and the earlier Linux
failures had it asleep in a pipe write. A thread that never returns from its
system call never runs the handler, so the job is never suspended.

Wait in short polls at both points instead, as wait_for_exit already does:
PipelineWriter writes through a non-blocking pipe and polls for room, and the
shell command waits in slices while its child belongs to a terminal pipeline.
The job-control test now delivers the relay to the watcher thread itself in
the two cases that failed, which reproduced the hang deterministically.

The reserved-terminal pager test waits for the bar repainted after a handoff,
as the command display renders on its own thread and can land it a frame
later than the resume, which Windows CI hit.
Making the pipe non-blocking regressed shell commands piped to a terminal
consumer: do_shell() hands the same open file description to the child, which
writes to it directly and failed with EAGAIN once the pipe was full. Leave the
descriptor blocking. PipelineWriter still returns to Python regularly, as the
job-control relay requires, by polling for room and writing at most PIPE_BUF
bytes at a time, which cannot block once the pipe reports it is writable.

The job-control test's timeout diagnostics now cope with a sandbox that cannot
run ps.
A toolbar callback that did not return within the shutdown timeout made
toolbar.stop() raise, and the error escaped cmdloop(). Had it not, the
next prompt would have refused to run on a terminal the display still
held. Report the display that would not stop, as one that cannot start
already is, and wait at the main prompt for it to release the terminal.
Ctrl-C during that wait ends the loop.
When a terminal pipeline exits before a shell command can join its
group, do_shell() retries in cmd2's own group. The terminal was still
lent to the dead pipeline, so the command stopped with SIGTTIN on its
first terminal read and the wait for it never returned. End the lend
before retrying.
SIGTTOU was blocked on the main thread for the whole life of a terminal
pipeline. A signal mask survives fork and exec, so every child started
meanwhile, whether a shell producer or a subprocess run by command code,
kept SIGTTOU blocked for life and could change terminal modes from the
background. The consumer now owns the terminal only during a lend, so
block SIGTTOU for the lend alone, and spawn a shell producer, which
starts inside one, with it unblocked.
The test raced cmd2's fixed 0.2s startup lend: a pager that set its
terminal modes after it ended was stopped with SIGTTOU and, on macOS,
failed with EINTR. Skipping coverage made the pager start faster, but a
busy free-threaded macOS runner still missed the window. Have the test
application hold the startup check open until the pager reports, so the
test checks that the pager owns the terminal throughout the check rather
than how fast it starts. Record whether the pager already held the
terminal, so a failure shows which side of the window it hit.
…ends

_job_control saved app.suspend_to_background, the class's bound method,
and on exit wrote it back as an instance attribute. The command display
installs its thread-safe Ctrl-Z handler only where the application has
none of its own, so after a fallback from reserved rendering, or a later
legacy loop, Ctrl-Z during a command went back to upstream's handler and
could leave the terminal cooked after fg. Delete the attribute instead
when nothing had replaced the method.
do_shell() lends the terminal for as long as a shell producer runs, and
a pipe write from another thread lends and returns meanwhile. The first
lend to end cleared availability and took the terminal back, so the
producer stopped with SIGTTIN on its next terminal read and nothing
resumed it. Count lends, and hand the terminal back after the last.
The stuck display left the terminal in raw mode, where Ctrl-C is only a
key, and the binding that would raise SIGINT runs on the event loop the
toolbar callback is blocking. So the wait before the next prompt could
not be interrupted, despite saying it could. Cook the terminal before
waiting, so its driver sends SIGINT again. Test it with a real Ctrl-C on
a pseudo-terminal rather than a simulated KeyboardInterrupt.
Waiting for a terminal pipeline now hands the terminal back, which can
fail, for example after a hangup. _restore_output then left the dead
pipeline current and _redirecting set, so ppaged() stopped paging and
Ctrl-C went to a process group that was gone. Restore both in a finally.
The wait for a toolbar display that will not stop cooked the terminal
and printed its notice before entering the try that catches Ctrl-C.
With the terminal cooked, Ctrl-C can arrive at once, and on Linux CI it
landed while the notice was still being written, escaping cmdloop()
with a traceback. Cook the terminal and print the notice inside the try.
tleonhardt added a commit that referenced this pull request Sep 26, 2026
A pipe process always ran in its own session, so it never received the
terminal. Piped to an interactive program such as less, Ctrl-Z and fg
did not suspend and resume it together with cmd2, and the program could
not take the keyboard as a shell's foreground job would.

When a pipe's output goes to cmd2's controlling terminal and the pipe is
started on the main thread, run the pipe process in a new process group
and lend it the terminal as it starts, while cmd2 writes to it, and while
cmd2 waits for it. Between writes the terminal returns to cmd2, so
command code can still read the keyboard. ProcReader watches the group on
a thread, relays its job-control stops to cmd2's job, and forwards Ctrl-C
without signalling cmd2's own group again. A shell command piped to such
a program joins the pipeline's job for as long as it runs. Pipes started
off the main thread, or whose output cmd2 captures, still run in their
own session.

The terminal tests drive cmd2 under an interactive bash on a
pseudo-terminal, using pyte to read the screen, and coverage now follows
the applications those tests start.

Extracted from the reserved-row toolbar work (PR #1761) without the
toolbar.
@tleonhardt
tleonhardt marked this pull request as draft September 27, 2026 18:42
@Neallin-917

Copy link
Copy Markdown

Independent automated check of 7f6b8dc on macOS 27 arm64, CPython 3.12.13,
prompt-toolkit 3.0.53 (Codex-assisted): I found a test-fixture environment
dependency, not a reason to change production terminal-capability detection.

With the inherited TERM=dumb, make test reports 52 failed / 2,683 passed /
8 skipped. This single case reproduces it without xdist:

TERM=dumb uv run python -Xutf8 -m pytest tests/test_prompt_toolkit_bridge.py::TestCursorPositionReports::test_cpr_uses_row_one_coordinate_contract --no-cov -n 0 -q

It fails because request_cursor_position() returns False. The harness
creates Vt100_Output without a terminal override; its fake stream reports
isatty() == True, but prompt-toolkit still disables CPR for TERM=dumb.
Changing only that command's TERM to xterm-256color passes, twice checked.
The same head with TERM=xterm-256color make test passes 2,735 / 8 skipped.

Could the fixtures that explicitly model a CPR-capable terminal declare that
terminal capability, keeping separate dumb-terminal/fallback coverage intact?
The existing colour-environment fixture does not isolate TERM.

The focused pipeline-job-control/command-toolbar/pager run also passed all
110 cases, and make docs-test passed. make check hits the known 75 Mypy 2.4
errors covered by the open #1768; I have not made a competing fix. These are
automated/controlling-PTY results, not native Terminal/iTerm visual testing,
Windows qualification, or proof that the 3.14t CI timeout is resolved.

# Conflicts:
#	cmd2/cmd2.py
#	cmd2/command_toolbar.py
#	cmd2/reserved_toolbar.py
#	cmd2/utils.py
#	tests/test_cmd2.py
#	tests/test_pipeline_job_control.py
#	tests/test_utils.py
@tleonhardt

tleonhardt commented Nov 3, 2026 •

Copy link
Copy Markdown
Member Author

@Neallin-917
Thanks for the careful check and the exact reproduction. I confirmed it on the current head: with TERM=dumb, make test failed 52 tests on this branch, 41 of which come from reserved_row_toolbar itself. All of them are in test_prompt_toolkit_bridge.py, test_reserved_terminal.py and test_reserved_lifecycle.py.

Test fixtures (609961e, on reserved_row_toolbar): rather than giving each harness its own term=, tests/conftest.py now clears TERM and PROMPT_TOOLKIT_NO_CPR in an autouse fixture, next to the one that already neutralizes the color environment. The suite then models the same terminal CI does by leaving TERM unset, and harnesses added later are covered too. The fallback coverage stays intact: tests of terminals that don't answer cursor position reports set that up explicitly (enable_cpr=False, or PROMPT_TOOLKIT_NO_CPR through monkeypatch, which runs after the autouse fixture). With this, TERM=dumb make test and PROMPT_TOOLKIT_NO_CPR=1 make test both pass (3,757 passed, 7 skipped on this branch), and so does your single-test command.

Production (dcbad7b): I agree the bridge's handling of cursor position reports doesn't need to change. While checking that, though, I found that mode selection did. It decided capability purely by backend identity, so AUTO still selected reserved rendering for a Vt100_Output in a dumb terminal. In a pty with TERM=dumb, cmd2 installed scroll margins (ESC[1;23r) and painted the toolbar with cursor addressing, which a real dumb terminal, such as an Emacs shell, shows as text. Meanwhile prompt-toolkit's own prompt falls back to plain line input there, and LEGACY draws nothing.

A dumb terminal is now an unmet prerequisite, judged the way prompt-toolkit judges cursor position reports: from the output's own terminal type, or from TERM if it names none. AUTO falls back to LEGACY, and RESERVED raises with the reason. This is a deliberate, narrow exception to deciding capability "never by TERM", and it's documented in cmd2/toolbar_mode.py and docs/features/prompt.md.

Both changes are merged into this branch.

Brings consolidated_toolbar's display error handling, unfinished-line handling in
the legacy stdout proxy, and the locking documentation. The Ctrl-Z stop fix was
already here; its comment and the test parametrized over both rendering modes
are kept.
@tleonhardt

Copy link
Copy Markdown
Member Author

One Windows check left to close Stage 4

Everything automated is green at 16d24282 (see Validation in the description). The remaining gate is the idle-prompt resize from 2 to 3 rows in Windows Terminal, which was corrupted before 37645b59 and was reported fixed by it. It needs a result from a real Windows machine at the current head.

Steps, in Windows Terminal with PowerShell:

  1. From the repository root on this branch: uv run python examples/stage4_manual.py
  2. Confirm the toolbar shows a live timestamp on the bottom row. Do not type anything.
  3. Drag the window's bottom edge up until only 2 rows are visible. Expected: the prompt is still visible; the toolbar may be drawn by the fallback or not at all.
  4. Grow it to exactly 3 rows. Expected: the toolbar is fully painted on row 3, with the prompt on one of the 2 rows above it. No leftover fragments, and no need to press Enter or Ctrl-L to repair it.
  5. Grow to 4 rows, then back to a normal height. Expected: toolbar on the bottom row each time, nothing stale left higher up.
  6. Type echo ok and press Enter. Expected: it runs, and the toolbar stays on the bottom row.
  7. Repeat steps 3 to 6 with a few characters typed at the prompt and not submitted. Expected: the typed text survives every resize.

To count rows exactly, run shell powershell -c "$Host.UI.RawUI.WindowSize.Height" at the prompt between resizes; the toolbar is hidden while that runs and comes back afterwards.

Please report the Windows build, Windows Terminal version, Python version, and for each of steps 3 to 7 either "as expected" or what you saw, with a screenshot of anything wrong. If you have time, the same sequence during work 10 (a running command) is the next most useful thing to try.

…dy pending

A redraw requested by command output stays marked pending until a frame commits,
and the command display's suppressed frames never commit. A frame which then
failed to prepare asked for a retry through the same coalescing request, which
dropped it as a duplicate, although the scheduling that went with the earlier
request was what had just run that frame.

The built-in pager depends on that retry: its first frame enters the alternate
screen, which cannot be recorded in a frame. In reserved mode the pager stayed
blank until the next toolbar refresh, and with no refresh_interval, the default,
until a key was pressed.

Schedule the retry directly. A frame whose preparation keeps raising is retried
only a few times, so a layout callback which always fails cannot become a redraw
loop.
In reserved mode the pager's first erase deletes whole lines starting at the
cursor's row. A command which wrote 'Results: ' without a newline and then
called ppaged() had that line deleted before the pager drew anything. Move past
it first, as is already done before a nested prompt.
The prompt documentation still said select() shows no toolbar, which is only
true of legacy rendering now, and the changelog had no entries for what Stage 4
changes or for the two built-in pager fixes.
@tleonhardt

Copy link
Copy Markdown
Member Author

Final review of the Stage 4 diff (2026-11-14)

Reviewed the diff against reserved_row_toolbar, then checked each finding against the running application in a pseudo-terminal rather than taking it on reading alone. Three fixes came out of it and are pushed (9e19d17f, 3e958a57, 91463a27).

Fixed

  • The built-in pager stayed blank in reserved mode until the next toolbar refresh, or until a key was pressed when no refresh_interval is set (the default). With a 1 second interval the first page took 1 second to appear; with none it never appeared on its own. The pager's first frame cannot be prepared (it enters the alternate screen, which a frame cannot record) and relies on a retry. That retry was dropped as a duplicate of a redraw request left pending by earlier command output, which only a committed frame clears, and the command display's suppressed frames never commit. Present since Stage 3; the examples all set a refresh interval, which hid it. Now 0.02 s regardless. (9e19d17f)
  • A line of output left unfinished before ppaged() was deleted in reserved mode. The review attributed this to the new handoff forgetting the bridge's record of the line. The byte stream shows a different cause, also present before this PR: the pager's first erase deletes whole lines from the cursor's row before the pager draws. The line is now finished first, as it already is before a nested prompt. (3e958a57)
  • Docs and changelog. docs/features/prompt.md still said select() shows no toolbar, and the changelog had nothing for this stage. (91463a27)

Found and left open: the pager never enters the alternate screen in reserved mode

Counting escape sequences for one pager session: legacy rendering sends one alternate-screen enter and one leave. Reserved rendering sends no enter and one leave, on this branch and on reserved_row_toolbar back to Stage 3. The recorder refuses the switch inside a frame, prompt-toolkit has by then marked itself as switched, and the retried frame is painted on the main screen. The pager therefore draws over the visible history instead of beside it, and its exit sends a leave that was never matched.

This is not introduced by this PR and I have not changed it here: a fix has to perform the switch outside a frame without undoing "keep the toolbar visible while the first page prepares", and it needs checking in real terminals, since pyte does not model the alternate screen. It is direct input for Stage 5's pager investigation. The harness's functional driver now reports it as a known limitation so it cannot be forgotten.

Review findings not acted on, for a maintainer decision

  • Ctrl-Z after the reservation is abandoned mid-command. _job_control teardown removes the application's suspend_to_background override, and the running CommandToolbar did not install its own because the reserved one was present. For the rest of that command Ctrl-Z would use upstream's version, which is the Linux cooked-terminal bug suspend_process_group exists for. Plausible on reading; needs an abandoned reservation and a Ctrl-Z in the same command. Not reproduced.
  • ReservedOutput.enter_alternate_screen called twice without a quit overwrites _alternate_handoff_owned, so the reservation would not be reacquired. Narrow, and tied up with the alternate-screen issue above.
  • A resize while a large pager prepares its first page may leave the deferred toolbar text on the main screen, because clear_deferred_band() only erases when the geometry still matches. Not reproduced.
  • select() differs by mode. The managed path builds ChoiceInput with the session's style and enable_suspend; the fallback calls choice() with neither. Confirmed on reading. Either both should pass them or neither.
  • A session with its own bottom_toolbar passed to _read_raw_input() now has that toolbar hidden and the main session's content shown in the reserved rows. Intended, or should such a session keep the exclusive path?
  • Cleanups: the getattr(self.bridge, "before_managed_write", None) lookup on every managed write, the redundant available_rows branch in page(), and job control being two independent monkeypatches of suspend_to_background coordinated by install order.

@tleonhardt
tleonhardt marked this pull request as ready for review November 14, 2026 23:42
@tleonhardt

tleonhardt commented Nov 15, 2026 •

Copy link
Copy Markdown
Member Author

Windows check: results

Results for the one Windows check left, at the current head 91463a27. Steps 3 to 7 are all as expected, in Windows Terminal and in Windows Terminal Preview. The 2 to 3 row resize was repeated more than 20 times and settled correctly every time.

How it was run: a script opened real Windows Terminal windows with PowerShell, ran uv run python examples/stage4_manual.py, changed the window's height one row at a time by moving its bottom edge, typed into the console, and took screenshots of the window, which were then inspected. Row counts were read from the console after each resize. It was not dragged with a mouse or watched by eye in real time.

Environment

  • Windows 11 Pro 26H2, build 26300.9974, ARM64
  • Windows Terminal 1.25.3131.0, and Windows Terminal Preview 1.26.3133.0, with Windows PowerShell 5.1, 120 columns by 30 rows
  • Python 3.14.10 x64 (emulated), prompt-toolkit 3.0.53

Steps

Step Result
2. Live timestamp on the bottom row as expected
3. Down to 2 rows as expected: prompt on row 1, toolbar on row 2, drawn by the fallback (reverse video)
4. Up to exactly 3 rows as expected: toolbar fully painted on row 3 in the reserved style, prompt on row 1, row 2 empty, no fragments, no key needed
5. Up to 4 rows, then back to 30 as expected: toolbar on the bottom row each time, nothing stale above it
6. echo ok as expected: it ran, toolbar stayed on the bottom row
7. Steps 3 to 6 with text typed and not submitted as expected: ech survived every resize, and typing o ok and Enter then ran echo ok. Repeated with abc, which survived too

Both Terminal versions gave the same results for every step.

Further passes, Windows Terminal 1.25

  • Back and forth between 2 and 3 rows four times, then 4, 3, 2, 3: right each time.
  • Each size set in one move instead of a row at a time: right.
  • A slow drag, a quarter of a second at each row: right.
  • While work 15 runs (also in Preview): at 2 rows there is no toolbar and the output keeps printing; at 3 rows the toolbar is back on row 3, at 4 on row 4, and at 30 on the bottom row. After the command finished the prompt came back and echo ok ran.

What happens in the half second after a resize

Not a failure, but worth knowing. Sampling the console every 10 to 15 ms after one-row resizes:

  • The old layout stays for up to about 0.5 s (measured 0.0 to 0.5 s over 31 resizes in each Terminal version) before the application repaints for the new size. After 2 to 3 rows that means the fallback toolbar is still on row 2 with row 3 empty; after 3 to 4, the toolbar is on row 3 with row 4 empty; after shrinking, there is no toolbar until the repaint.
  • The repaint passes through one blank frame of 10 to 60 ms.
  • About half a second later the prompt row is sometimes blank once more for 10 to 50 ms.

The delay matches prompt-toolkit polling the terminal size every 0.5 s by default, which cmd2 does not change. I have not confirmed that as the cause.

Not checked

  • Dragging with a real mouse.
  • What is left in the scrollback above the visible rows after shrinking.
  • conhost.exe and other terminals: the request was for Windows Terminal.

@tleonhardt
tleonhardt merged commit 54c3366 into reserved_row_toolbar Nov 15, 2026
30 checks passed
@tleonhardt
tleonhardt deleted the stage4-handoffs-integration branch November 15, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants