Skip to content

test: address some flaky tests and a language server stop error - #1955

Merged
eitsupi merged 11 commits into
mainfrom
test/stop-session-display-diagnostics
Nov 17, 2026
Merged

eitsupi merged 11 commits into
mainfrom
test/stop-session-display-diagnostics

Conversation

@eitsupi

@eitsupi eitsupi commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

The vscode-R test suites currently contain many flaky tests, and CI fails with high probability for reasons unrelated to the change under test. This PR addresses some of the ones found so far. Where a failure message was not enough to find the cause, it adds diagnostics without changing the assertions; where the cause was found, it fixes it.

  • R packages from P3M may lack system dependencies Installing unigd from P3M fails due to lack of X11. xquartz is needed.
  • Interactive Stop Session ("Stop Session adds one notice to its own notebook and source execution can choose a live replacement", macOS): after the stop notice appears, it sometimes times out waiting for every display to report running: false. On timeout it now reports each remaining display (cell, generation, running/connected), the view's session status, the output queue's state, and when the stop notice and each display update reached the notebook.
  • Help panel navigation (Linux and macOS, different tests): after navigation completes, clock.countTimers() sometimes finds one pending fake timer. Fake timers replace setTimeout for the whole extension host, so it now reports each pending timer's type, delay, and callback source. This shows whether the timer belongs to Help or to unrelated code.
  • Retained jgd plot tab ("a retained jgd tab redraws plots made while hidden without resizing R", macOS): after a hidden tab is revealed, the canvas was sometimes 207x116 while its container had returned to 966x542. On failure the test now inspects again after one second and reports both inspections, any resize requests R received, and a webview timeline of replays (with container and canvas sizes), resize reports, visibility messages, and container resizes.
  • Activation terminal-persistent-reload (macOS): two problems surfaced in the window reload. First, VS Code saves the terminal layout through a debounced call that the reload does not flush, so when resizes kept postponing that save the R terminal was missing from the saved layout and was not revived; the test then read a new default shell as terminals[0]. The test now waits until the pty host trace log shows an accepted layout that includes the R terminal before reloading, and after the reload it selects the terminal by its recorded PID. Second, the test sent its R code once, as soon as the terminal reported a PID, but VS Code can drop input while it replays a revived terminal's buffer. It now resends an idempotent probe until R answers, then sends the state check, and reports R errors from that check. On failure it keeps the pty host trace logs, the phase files, a timeline of sent input, and the terminal's process state (with sample on macOS).
  • R language server stop (Windows, Interactive editor suite): R's languageserver exits as soon as it has answered shutdown, so the client can fail to write the following exit notification and see the connection reset. On Windows this showed users a "connection to server is erroring" notification whenever a server was stopped, for example when an Interactive notebook was closed, and made the stop fail with ECONNABORTED. The client now treats this as a completed stop only when the server answered shutdown and its own process exited before any forced termination, and it logs connection errors during a requested stop to the output channel instead of notifying. A reset while the server is running is still reported. Stops of clients removed from the language service are now tracked, logged on failure, and awaited on disposal.

The Stop Session test intermittently times out on macOS while waiting for
every display to report that R stopped. Report the remaining displays, the
view's status, the output queue, and when the stop notice and each display
update reached the notebook, so the failure shows whether the update was
late, missing, or never scheduled.
@eitsupi
eitsupi marked this pull request as ready for review November 16, 2026 23:56
@eitsupi
eitsupi requested a review from renkun-ken November 16, 2026 23:56
Help panel tests intermittently find one pending fake timer after the
navigation under test has completed. Fake timers replace setTimeout for the
whole extension host, so report each pending timer's type, delay and
callback source to tell a Help timer apart from one scheduled by unrelated
code.
@eitsupi eitsupi changed the title test(interactive): report display state when Stop Session times out test: report diagnostics for intermittent Interactive and Help test failures Nov 17, 2026
@eitsupi
eitsupi requested a review from Fred-Wu November 17, 2026 00:15
@eitsupi
eitsupi marked this pull request as draft November 17, 2026 00:48
The persistent terminal reload case sent its state check once, as soon as
the restored terminal reported a PID. VS Code can drop input while it
replays a revived terminal's buffer, so a single early send may never reach
R. Resend an idempotent probe until R answers, then send the state check,
and report R errors from that check instead of timing out.

On failure, keep the pty host trace logs, the phase and probe files, a
timeline of sent input, and the terminal's process state so the next CI
failure shows whether input reached R.

@renkun-ken renkun-ken left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 28e48371714ffa151331dd1ec36f57dc99b2f180; no blocking findings.

The added diagnostics preserve the zero-pending-timer assertions and the existing all-displays-stopped condition. The Interactive diagnostic callback is evaluated only on timeout, and the notebook listener is disposed in finally. I verified the timer heap/jobs layout against the locked Sinon 22.1.0 / fake-timers 15.4.0 implementation and the view/queue fields against the current source.

Validation passed: extension build and TypeScript compilation, focused lint/format checks, and diff whitespace checks. On macOS with VS Code 1.145.2 and arf 0.7.2-rc.1, all 17 Help navigation tests plus the native execution / Stop Session pair passed (19 tests). Separate failure-path checks verified pending timer reporting, display/view/queue serialization, timeline events, and missing-view / paused / disposed queue cases.

CI note: nine of ten hosted checks currently pass. The failing macOS Extension job stops in test:activation with R terminal did not report its retained state, before the changed extension suites run. That activation failure remains outstanding.

@eitsupi eitsupi changed the title test: report diagnostics for intermittent Interactive and Help test failures test: diagnose intermittent Interactive, Help, and activation test failures Nov 17, 2026
@eitsupi
eitsupi requested a review from renkun-ken November 17, 2026 00:52
@eitsupi
eitsupi marked this pull request as ready for review November 17, 2026 00:52
@eitsupi

eitsupi commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

@renkun-ken Excuse me, but I've added a commit, could you please take another look?

@renkun-ken renkun-ken left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new commit 5d9633bc062ebc968a877dc4123214577d241ffe. The input probe and R error reporting preserve the existing persistence assertions, and the artifact path now includes the nested trace logs. One correction is needed in the new macOS process sampling; see the inline P2 finding.

Validation: JavaScript syntax check and both activation environment tests passed. The revised terminal-persistent-reload case passed locally on macOS / VS Code 1.145.2, retaining the same terminal and R PID, workspace marker and sess identity, then reconnecting only after explicit activation. An exact-source diagnostic check confirmed that the current sampler makes zero sample calls when R is the terminal PID and has no descendants.

The latest macOS CI job still fails, now at Window reload must retain the terminal process (11983 !== 11810). Its artifact identifies the initial R PID as 11810, while the selected post-reload PID 11983 belongs to a newly launched /bin/bash; no after-phase probe was sent. Terminal restoration/selection therefore deserves investigation alongside input delivery. The new retry has not yet produced a passing macOS hosted activation run.

Comment thread scripts/test-activation.cjs Resolved
VS Code saves the terminal layout through a debounced call that a window
reload does not flush. On macOS CI, resizes kept postponing that save until
the test reloaded, so the R terminal was missing from the saved layout, was
not revived, and the test read a new default shell as terminals[0].

Before reloading, wait until the pty host trace log shows an accepted layout
that includes the R terminal's persistent process. After reloading, select
the terminal by the PID recorded before the reload, and report the open
terminals if none matches.

Also sample the terminal PID itself on macOS, since the R Terminal profile
runs R directly rather than as a child process.
@eitsupi
eitsupi requested a review from renkun-ken November 17, 2026 01:34
@eitsupi
eitsupi marked this pull request as draft November 17, 2026 01:43
R's languageserver exits once it has answered shutdown, so the client can
fail to write the exit notification and see the connection reset. On
Windows this showed a "connection to server is erroring" notification and
failed the stop with ECONNABORTED during the Interactive editor suite.

Fix that order in a test with a real owned child process: the client must
stop without a user-facing error when the server exits by itself, still
fail when the server keeps running, and still report a reset while the
server is running.
R's languageserver exits as soon as it has answered shutdown, so writing
the exit notification can fail and the connection can be reset. On Windows
this showed users a "connection to server is erroring" notification
whenever a server was stopped, such as when an Interactive notebook was
closed, and made the stop reject with ECONNABORTED.

Treat that as a completed stop only when the server answered shutdown,
the error is a known disconnect, and every owned server process exited
before any forced termination. During a requested stop, log disconnect
errors to the output channel instead of notifying. A reset while the
server is running is still reported.

Track stops of clients removed from the language service, log their
failures, and wait for them on disposal. A rebind no longer skips starting
the new server when stopping the old one fails.
@eitsupi eitsupi changed the title test: diagnose intermittent Interactive, Help, and activation test failures test: address some flaky tests and a language server stop error Nov 17, 2026
@eitsupi
eitsupi marked this pull request as ready for review November 17, 2026 02:31
…n fails

The retained jgd tab test intermittently found the canvas at 207x116 while
its container had returned to 966x542 after the tab was revealed. The
failure did not show whether the viewer drew at a temporary size, whether
it asked R to resize, or whether a later replay corrected the canvas.

Record each replay's start and end with the container and canvas sizes,
the viewer's resize reports, visibility messages, and container resizes.
When an assertion after the reveal fails, inspect again after one second
and attach both inspections, the resize calls R received, and the timeline
to the failure.
@eitsupi
eitsupi force-pushed the test/stop-session-display-diagnostics branch from e165ddc to 444166d Compare November 17, 2026 02:41

@renkun-ken renkun-ken left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed c29af3f697cebc8b28a89161e3bbd56066a3cdec. The previous terminal-sampling finding is fixed: the R PID is now included. The layout-save wait and PID-based restored-terminal selection also address the Bash-terminal failure identified in the last review.

I found one new P2 issue in the retained-jgd test diagnostics: updating AssertionError.message does not update the stack that Mocha's spec reporter prints, so the new failure details are absent from CI output. See the inline comment.

Validation with the locked dependencies passed: extension build, TypeScript compilation, focused lint/format checks, all 14 language-service tests, an additional real-socket-close shutdown case, and the retained-jgd test. The updated persistent-terminal reload case, JavaScript syntax check and both activation environment tests also passed. The diagnostic loss was reproduced separately with Mocha 12.0.3. All ten hosted checks are green at this head.

Comment thread src/test/plot/plotRestoration.test.ts Resolved
@eitsupi
eitsupi force-pushed the test/stop-session-display-diagnostics branch from e9e4ba5 to 7fc4d86 Compare November 17, 2026 06:35
@eitsupi
eitsupi requested a review from renkun-ken November 17, 2026 06:36
@eitsupi
eitsupi force-pushed the test/stop-session-display-diagnostics branch from 5c445db to 8c592fd Compare November 17, 2026 07:09
@eitsupi

eitsupi commented Nov 17, 2026 •

Copy link
Copy Markdown
Member Author

@renkun-ken CI continues to fail on main. Today's daily release also failed.
https://github.com/REditorSupport/vscode-R/actions/runs/44010454932

I want to merge as soon as possible. (For minor issues, please commit directly or refer them to follow-up.)

@renkun-ken renkun-ken left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 8c592fd13b0afe86a5e06902c0d01ff23c0bc336; no blocking findings.

The plot-diagnostics correction is verified: Mocha 12 now prints the new diagnostic details while retaining the original assertion and stack through cause. Adding ERR_STREAM_DESTROYED preserves the existing shutdown-acknowledgement and owned-process-exit gates. The macOS CI log also confirms that the new XQuartz setup step installed successfully.

Validation passed: extension build, TypeScript compilation, focused lint/format and diff checks, the retained-jgd test, and two temporary shutdown fixtures using ERR_STREAM_DESTROYED (an exited server stops successfully; a still-running server continues to reject). Separate failure-path checks confirmed the diagnostic output and the missing-acknowledgement/no-owned-process rejection cases. All ten hosted checks are green at this head.

@eitsupi
eitsupi merged commit 97100ef into main Nov 17, 2026
22 of 24 checks passed
@eitsupi
eitsupi deleted the test/stop-session-display-diagnostics branch November 17, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants