Repository navigation
test: address some flaky tests and a language server stop error - #1955
Conversation
The Stop Session test intermittently times out on macOS while waiting for every display to report that R stopped. Report the remaining displays, the view's status, the output queue, and when the stop notice and each display update reached the notebook, so the failure shows whether the update was late, missing, or never scheduled.
Help panel tests intermittently find one pending fake timer after the navigation under test has completed. Fake timers replace setTimeout for the whole extension host, so report each pending timer's type, delay and callback source to tell a Help timer apart from one scheduled by unrelated code.
The persistent terminal reload case sent its state check once, as soon as the restored terminal reported a PID. VS Code can drop input while it replays a revived terminal's buffer, so a single early send may never reach R. Resend an idempotent probe until R answers, then send the state check, and report R errors from that check instead of timing out. On failure, keep the pty host trace logs, the phase and probe files, a timeline of sent input, and the terminal's process state so the next CI failure shows whether input reached R.
renkun-ken
left a comment
There was a problem hiding this comment.
Reviewed 28e48371714ffa151331dd1ec36f57dc99b2f180; no blocking findings.
The added diagnostics preserve the zero-pending-timer assertions and the existing all-displays-stopped condition. The Interactive diagnostic callback is evaluated only on timeout, and the notebook listener is disposed in finally. I verified the timer heap/jobs layout against the locked Sinon 22.1.0 / fake-timers 15.4.0 implementation and the view/queue fields against the current source.
Validation passed: extension build and TypeScript compilation, focused lint/format checks, and diff whitespace checks. On macOS with VS Code 1.145.2 and arf 0.7.2-rc.1, all 17 Help navigation tests plus the native execution / Stop Session pair passed (19 tests). Separate failure-path checks verified pending timer reporting, display/view/queue serialization, timeline events, and missing-view / paused / disposed queue cases.
CI note: nine of ten hosted checks currently pass. The failing macOS Extension job stops in test:activation with R terminal did not report its retained state, before the changed extension suites run. That activation failure remains outstanding.
|
@renkun-ken Excuse me, but I've added a commit, could you please take another look? |
renkun-ken
left a comment
There was a problem hiding this comment.
Reviewed the new commit 5d9633bc062ebc968a877dc4123214577d241ffe. The input probe and R error reporting preserve the existing persistence assertions, and the artifact path now includes the nested trace logs. One correction is needed in the new macOS process sampling; see the inline P2 finding.
Validation: JavaScript syntax check and both activation environment tests passed. The revised terminal-persistent-reload case passed locally on macOS / VS Code 1.145.2, retaining the same terminal and R PID, workspace marker and sess identity, then reconnecting only after explicit activation. An exact-source diagnostic check confirmed that the current sampler makes zero sample calls when R is the terminal PID and has no descendants.
The latest macOS CI job still fails, now at Window reload must retain the terminal process (11983 !== 11810). Its artifact identifies the initial R PID as 11810, while the selected post-reload PID 11983 belongs to a newly launched /bin/bash; no after-phase probe was sent. Terminal restoration/selection therefore deserves investigation alongside input delivery. The new retry has not yet produced a passing macOS hosted activation run.
VS Code saves the terminal layout through a debounced call that a window reload does not flush. On macOS CI, resizes kept postponing that save until the test reloaded, so the R terminal was missing from the saved layout, was not revived, and the test read a new default shell as terminals[0]. Before reloading, wait until the pty host trace log shows an accepted layout that includes the R terminal's persistent process. After reloading, select the terminal by the PID recorded before the reload, and report the open terminals if none matches. Also sample the terminal PID itself on macOS, since the R Terminal profile runs R directly rather than as a child process.
R's languageserver exits once it has answered shutdown, so the client can fail to write the exit notification and see the connection reset. On Windows this showed a "connection to server is erroring" notification and failed the stop with ECONNABORTED during the Interactive editor suite. Fix that order in a test with a real owned child process: the client must stop without a user-facing error when the server exits by itself, still fail when the server keeps running, and still report a reset while the server is running.
R's languageserver exits as soon as it has answered shutdown, so writing the exit notification can fail and the connection can be reset. On Windows this showed users a "connection to server is erroring" notification whenever a server was stopped, such as when an Interactive notebook was closed, and made the stop reject with ECONNABORTED. Treat that as a completed stop only when the server answered shutdown, the error is a known disconnect, and every owned server process exited before any forced termination. During a requested stop, log disconnect errors to the output channel instead of notifying. A reset while the server is running is still reported. Track stops of clients removed from the language service, log their failures, and wait for them on disposal. A rebind no longer skips starting the new server when stopping the old one fails.
…n fails The retained jgd tab test intermittently found the canvas at 207x116 while its container had returned to 966x542 after the tab was revealed. The failure did not show whether the viewer drew at a temporary size, whether it asked R to resize, or whether a later replay corrected the canvas. Record each replay's start and end with the container and canvas sizes, the viewer's resize reports, visibility messages, and container resizes. When an assertion after the reveal fails, inspect again after one second and attach both inspections, the resize calls R received, and the timeline to the failure.
e165ddc to
444166d
Compare
renkun-ken
left a comment
There was a problem hiding this comment.
Reviewed c29af3f697cebc8b28a89161e3bbd56066a3cdec. The previous terminal-sampling finding is fixed: the R PID is now included. The layout-save wait and PID-based restored-terminal selection also address the Bash-terminal failure identified in the last review.
I found one new P2 issue in the retained-jgd test diagnostics: updating AssertionError.message does not update the stack that Mocha's spec reporter prints, so the new failure details are absent from CI output. See the inline comment.
Validation with the locked dependencies passed: extension build, TypeScript compilation, focused lint/format checks, all 14 language-service tests, an additional real-socket-close shutdown case, and the retained-jgd test. The updated persistent-terminal reload case, JavaScript syntax check and both activation environment tests also passed. The diagnostic loss was reproduced separately with Mocha 12.0.3. All ten hosted checks are green at this head.
e9e4ba5 to
7fc4d86
Compare
5c445db to
8c592fd
Compare
|
@renkun-ken CI continues to fail on main. Today's daily release also failed. I want to merge as soon as possible. (For minor issues, please commit directly or refer them to follow-up.) |
renkun-ken
left a comment
There was a problem hiding this comment.
Reviewed 8c592fd13b0afe86a5e06902c0d01ff23c0bc336; no blocking findings.
The plot-diagnostics correction is verified: Mocha 12 now prints the new diagnostic details while retaining the original assertion and stack through cause. Adding ERR_STREAM_DESTROYED preserves the existing shutdown-acknowledgement and owned-process-exit gates. The macOS CI log also confirms that the new XQuartz setup step installed successfully.
Validation passed: extension build, TypeScript compilation, focused lint/format and diff checks, the retained-jgd test, and two temporary shutdown fixtures using ERR_STREAM_DESTROYED (an exited server stops successfully; a still-running server continues to reject). Separate failure-path checks confirmed the diagnostic output and the missing-acknowledgement/no-owned-process rejection cases. All ten hosted checks are green at this head.
The vscode-R test suites currently contain many flaky tests, and CI fails with high probability for reasons unrelated to the change under test. This PR addresses some of the ones found so far. Where a failure message was not enough to find the cause, it adds diagnostics without changing the assertions; where the cause was found, it fixes it.
running: false. On timeout it now reports each remaining display (cell, generation,running/connected), the view's session status, the output queue's state, and when the stop notice and each display update reached the notebook.clock.countTimers()sometimes finds one pending fake timer. Fake timers replacesetTimeoutfor the whole extension host, so it now reports each pending timer's type, delay, and callback source. This shows whether the timer belongs to Help or to unrelated code.terminal-persistent-reload(macOS): two problems surfaced in the window reload. First, VS Code saves the terminal layout through a debounced call that the reload does not flush, so when resizes kept postponing that save the R terminal was missing from the saved layout and was not revived; the test then read a new default shell asterminals[0]. The test now waits until the pty host trace log shows an accepted layout that includes the R terminal before reloading, and after the reload it selects the terminal by its recorded PID. Second, the test sent its R code once, as soon as the terminal reported a PID, but VS Code can drop input while it replays a revived terminal's buffer. It now resends an idempotent probe until R answers, then sends the state check, and reports R errors from that check. On failure it keeps the pty host trace logs, the phase files, a timeline of sent input, and the terminal's process state (withsampleon macOS).shutdown, so the client can fail to write the followingexitnotification and see the connection reset. On Windows this showed users a "connection to server is erroring" notification whenever a server was stopped, for example when an Interactive notebook was closed, and made the stop fail withECONNABORTED. The client now treats this as a completed stop only when the server answeredshutdownand its own process exited before any forced termination, and it logs connection errors during a requested stop to the output channel instead of notifying. A reset while the server is running is still reported. Stops of clients removed from the language service are now tracked, logged on failure, and awaited on disposal.