Skip to content

feat(tools): add Test Explorer run, debug and stats tools - #105

Merged
CalvinAllen merged 1 commit into
mainfrom
feat/tools/test-explorer-tools
Sep 3, 2026
Merged

feat(tools): add Test Explorer run, debug and stats tools#105
CalvinAllen merged 1 commit into
mainfrom
feat/tools/test-explorer-tools

Conversation

@CalvinAllen

Copy link
Copy Markdown
Contributor

Resolves #101
Resolves #102

Adds seven MCP tools over Test Explorer.

Tool Behavior
test_run_all Run every test in the solution
test_debug_all Debug every test in the solution
test_run Run the tests in one class or method
test_debug Debug the tests in one class or method
test_cancel Cancel the run in progress
test_status Run state plus current counts
test_stats Passed / failed / skipped / not-run counts

One deviation from the issues, with reason

Both issues called for referencing Microsoft.VisualStudio.TestWindow.Interfaces.dll from the VS install directory via $(DevEnvDir). That doesn't work. I verified before writing any code:

$ dotnet msbuild ...MCPServer.csproj -getProperty:DevEnvDir -getProperty:VsInstallRoot
{ "DevEnvDir": "*Undefined*", "VsInstallRoot": "" }

$(DevEnvDir) is only set when building from a Visual Studio / developer command prompt context. This repo builds with dotnet build both locally and in CI (vsix-build.yml step 3), so a HintPath reference would have broken the build outright. The assembly is also absent from the Microsoft.VisualStudio.SDK metapackage, and nuget.org carries nothing newer than 11.0.61030 (2012), which long predates ITestExplorerStatsService.

TestExplorerInterop therefore resolves the two services by MEF contract name and reads them reflectively. The alternative — an MSBuild target shelling out to vswhere — would have made a Visual Studio installation a hard build prerequisite, which it currently isn't. The tradeoff is losing compile-time type safety across six members; the reflection is confined to one file with the rationale in its <remarks>.

Design notes

Non-blocking, consistent with #96. The run tools start the run and return immediately. Completion is observed by subscribing to IOperationState.StateChanged, not by blocking, so the UI thread is never held — this is specifically not a reintroduction of the Build(true) problem that #96 fixed. test_status collapses the raw TestOperationStates values into NoRunObserved / Discovering / Running / Canceling / Completed / Canceled.

A discovery pass doesn't erase a run outcome. Editing code after a run triggers discovery; the state machine keeps the last execution result rather than overwriting it. Covered by a test.

"Not initialized" is not "no tests". ITestExplorerStatsService reports zeros before Test Explorer starts. test_stats checks IsTestExplorerStatsServiceRunning and returns an explicit message instead, so an agent can't misread an unopened window as a solution with no tests.

Caret positioning. TestExplorer.RunAllTestsInContext acts on wherever the caret is, so test_run / test_debug resolve the name through the existing workspace symbol search, open the file, and place the caret before issuing the command. Resolution prefers exact fully-qualified match, then exact simple name, then trailing segment, and reports when the match was ambiguous.

Command probing. ExecuteCommand throws when a command is disabled, which for Test Explorer is normal before discovery finishes. Commands are checked with Command.IsAvailable first so that's an ordinary result rather than an exception.

test_cancel caveat. Test Explorer exposes no documented cancel command ID — RunAllTests, DebugAllTests, RepeatLastRun and friends are documented, cancel is not. It probes TestExplorer.CancelTestRun then TestExplorer.CancelTests and uses whichever the running VS recognizes, reporting cleanly if neither does. Flagging it as the one piece here that would benefit from a check against a live Test Explorer.

Testing

dotnet build -c Release clean, 0 warnings. 36/36 tests pass (26 added).

Rather than only testing the pure helpers, the tests stand up a real MEF CompositionContainer holding stand-in parts declared in the genuine Microsoft.VisualStudio.TestWindow.Extensibility namespace and shaped like the shipped ones — including implementing StateChanged explicitly, as OperationBroker does. That exercises the contract-name lookup, interface discovery, property reads and event subscription, which is where the risk in this approach actually lives.

Not covered automatically: the DTE command invocations and caret positioning, which need a live VS instance.

Not included

The list of individual failed / not-run test names. As documented in #95, that data is internal and gated behind InternalsVisibleTo on Microsoft's strong-name key. test_stats' description says counts only, so the limitation is visible to the agent rather than silently surprising.

Adds seven MCP tools over Test Explorer: test_run_all, test_debug_all,
test_run, test_debug, test_cancel, test_status and test_stats.

The extension cannot reference Microsoft.VisualStudio.TestWindow.Interfaces.dll
at compile time. It is absent from the Microsoft.VisualStudio.SDK metapackage,
nuget.org carries nothing newer than 11.0.61030 (2012), and $(DevEnvDir) is
undefined under dotnet build, which is how this repo builds locally and in CI.
TestExplorerInterop therefore resolves ITestExplorerStatsService and
IOperationState by MEF contract name and reads them reflectively. The surface
needed is one bool, one struct of four ints and one event, and it is identical
in VS 2022 17.14 and VS 2026 18.0.

Run tools start the run and return immediately, matching the non-blocking
contract the build tools were corrected to in #96. Completion is observed by
subscribing to IOperationState rather than by blocking, so the UI thread is
never held; test_status reports the collapsed run state and current counts.

test_stats distinguishes "Test Explorer not initialized" from "no tests", so
an uninitialized window cannot be misread as a solution with zero tests.

The *InContext commands act on the caret, so test_run and test_debug resolve
the requested class or method through the workspace symbol search, open the
file and position the caret before issuing the command.

Test Explorer exposes no documented cancel command, so test_cancel probes
candidate names and uses the first the running Visual Studio recognises.

Commands are probed with Command.IsAvailable before execution, because
ExecuteCommand throws when a command is disabled, which for Test Explorer is
the normal state before discovery finishes.
@CalvinAllen
CalvinAllen merged commit 1847f22 into main Sep 3, 2026
2 checks passed
@CalvinAllen
CalvinAllen deleted the feat/tools/test-explorer-tools branch September 3, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

feat(tools): add test statistics tool feat(tools): add test run and debug tools with completion tracking

1 participant