Publish the stress tests; answer the ThreadJob and PoshRSJob question

README.md

@@ -62,10 +62,10 @@ Wrap any of them in `Invoke-PkRun { ... }` to get graceful Ctrl+C. (For `Invoke-
 
 For running PowerShell code in parallel, most of these are fine and some are simpler. ParallelKit is mainly about running external programs, and about what happens when a run is stopped partway through.
 
-- **PowerShell 7's `ForEach-Object -Parallel`.** The easy answer for plain parallel PowerShell code if you can use PowerShell 7. ParallelKit was built for scripts and machines stuck on 5.1, though it runs on 7 too, where it still adds the external-program handling and graceful Ctrl+C.
+- **PowerShell 7's `ForEach-Object -Parallel`.** The easy answer for plain parallel PowerShell code if you can use PowerShell 7. ParallelKit was built for scripts and machines stuck on 5.1, though it runs on 7 too, where it still adds the external-program handling and graceful Ctrl+C. `ForEach-Object -Parallel` also refuses a scriptblock passed by `$using:`, rather than letting it run where it isn't safe.
 - **`Start-Job`.** Built in, but each job is a whole new `powershell.exe` process, so every job costs noticeable startup time and memory, and what it returns comes back as serialized copies (properties only, no methods). 5.1's `Start-Job` has no throttle either, so you write the wait-for-a-free-slot loop yourself. Fine for a handful of long jobs, poor for thousands of short ones.
-- **ThreadJob (`Start-ThreadJob`).** Runs jobs as runspaces in the same process, with a `-ThrottleLimit`. On 5.1 it's the best fit for plain parallel scriptblocks, if you can install a module from the PowerShell Gallery (it ships with PowerShell 7, not 5.1).
-- **PoshRSJob (`Start-RSJob`).** A community module from the Gallery, also runspace-based, with a `-Throttle` limit and a job-style interface. For this purpose it has the same strengths and gaps as ThreadJob.
+- **ThreadJob (`Start-ThreadJob`).** Runs jobs as runspaces in the same process, with a `-ThrottleLimit`. On 5.1 it's the best fit for plain parallel scriptblocks, if you can install a module from the PowerShell Gallery (`Microsoft.PowerShell.ThreadJob`; it ships with PowerShell 7, not 5.1).
+- **PoshRSJob (`Start-RSJob`).** A community module from the Gallery, also runspace-based, with a `-Throttle` limit and a job-style interface. It hasn't had a release since 2018, it can report a job as Completed before its output is ready (run `Wait-RSJob` before `Receive-RSJob`), and it's hit harder than ThreadJob by the [scriptblock problem](#faq). Where both are an option, ThreadJob is the better pick.
 - **A runspace pool by hand** (`[runspacefactory]::CreateRunspacePool`). That's what `Invoke-PkRunspaceJobs` is built on. It works, but you end up writing the polling loop, error collection, cancellation and cleanup yourself, which is most of what this file is.
 - **Workflows (`foreach -parallel`).** Still in 5.1, but they use a restricted version of the language, start slowly, and were dropped from PowerShell 6 and later. Not worth starting anything new on.
 - **`Start-Process` in a loop.** Starts programs without waiting, but can only send their output to files, and the throttling and waiting are up to you. Using `System.Diagnostics.Process` directly gets output into memory, but reading stdout and then stderr the simple way deadlocks once a program fills the pipe you aren't reading. The kit reads both in the background.
@@ -267,7 +267,7 @@ Callbacks: a plain `{ ... }` callback sees your variables and functions from whe
 
 `Invoke-PkProcessWithTimeout` and `Start-PkProcess` also take `-WorkingDirectory`.
 
-All four are safe in worker code.
+All of these are safe in worker code.
 
 ## Messages
 
@@ -299,7 +299,8 @@ Three things to know:
 - **Your script carries on after an abort.** The engines return with `Aborted = $true` and the rest of your script runs. If an abort should end it, say so: `if ($batch.Aborted) { exit 1 }`.
 
 `Invoke-PkRun` also turns off the console's QuickEdit mode for the length of the run (`-DisableQuickEdit $false` to opt out). With QuickEdit on in the classic console window, one stray click starts a text selection, and the script freezes at its next console write until the selection is cleared. That looks exactly like a hang. It's turned back on afterwards if it was on before.
-Other pieces: `Request-PkAbort` triggers the same abort from code. `Test-PkAbortSignal` asks whether one is under way. `Register-PkCancellationSource` adds your own `CancellationTokenSource` to the set an abort cancels (a `-Cts` you pass to `Invoke-PkRunspaceJobs` is added automatically; a `-CancelToken` you pass to the dispatcher isn't cancelled by an abort, but the dispatcher stops either way). `Get-PkAbortTriggerLabel` says which kind of abort it was.
+
+Other pieces: `Request-PkAbort` triggers the same abort from code. `Test-PkAbortSignal` asks whether one is under way. `Register-PkCancellationSource` adds your own `CancellationTokenSource` to the set an abort cancels, and `Unregister-PkCancellationSource` takes it back out (a `-Cts` you pass to `Invoke-PkRunspaceJobs` is added automatically; a `-CancelToken` you pass to the dispatcher isn't cancelled by an abort, but the dispatcher stops either way). `Get-PkAbortTriggerLabel` says which kind of abort it was.
 
 ## Choosing limits
 
@@ -327,10 +328,23 @@ Other pieces: `Request-PkAbort` triggers the same abort from code. `Test-PkAbort
 powershell -NoProfile -File .\ParallelKit.Harness.ps1
 ```
 
-It uses real runspaces and real child processes (`cmd.exe`, `ping.exe`), with nothing mocked, so it takes about a minute. It prints a `[PASS]`/`[FAIL]` line per check and exits 0 only if everything passed. A few checks send real Ctrl+C, Ctrl+Break and window-close events to child PowerShell processes in hidden consoles of their own, never to yours. Where a machine can't deliver those, they print `[SKIP]` with the reason instead; set `PK_HARNESS_NO_SKIP=1` to count skips as failures. A few `message handler failed` lines in the output are expected: some checks break the message handler on purpose to test the fallback.
+It uses real runspaces and real child processes (`cmd.exe`, `ping.exe`), with nothing mocked, so it takes about two minutes. It prints a `[PASS]`/`[FAIL]` line per check and exits 0 only if everything passed. A few checks send real Ctrl+C, Ctrl+Break and window-close events to child PowerShell processes in hidden consoles of their own, never to yours. Where a machine can't deliver those, they print `[SKIP]` with the reason instead; set `PK_HARNESS_NO_SKIP=1` to count skips as failures. A few `message handler failed` lines in the output are expected: some checks break the message handler on purpose to test the fallback.
 
 GitHub runs the same harness on Windows PowerShell 5.1 and on PowerShell 7 for every pull request and every change to `main`.
 
+`Testing/` holds the longer stress tests and checks behind some of what this README says. They aren't part of the test suite and aren't needed to use the kit:
+
+- `Testing/cmdlet-hang/` stresses the kit's worker code, for the [two rules](#the-two-rules-for-worker-code).
+- `Testing/third-party/` asks the same question of ThreadJob and PoshRSJob. `results/matrix.log` holds the results behind the FAQ answer, from `Microsoft.PowerShell.ThreadJob` 2.2.0 (the `ThreadJob` package below brings it along) and PoshRSJob 1.7.4.4. To run it yourself, save the modules first, in PowerShell 7.4 or later (which has `Save-PSResource`), then run `matrix.ps1`:
+
+  ```powershell
+  New-Item -ItemType Directory -Force .\Testing\third-party\modules
+  Save-PSResource -Name ThreadJob, PoshRSJob -Path .\Testing\third-party\modules
+  powershell -NoProfile -File .\Testing\third-party\matrix.ps1
+  ```
+
+- `Testing/hosts/check-host.ps1` checks the kit in a host you choose, such as an editor's terminal or a scheduled task.
+
 ## FAQ
 
 **Can I use cmdlets in worker code?**
@@ -339,7 +353,16 @@ Yes. What a worker can't do is run a scriptblock that came from another runspace
 
 **Do ThreadJob and PoshRSJob have the same scriptblock problem?**
 
-It hasn't been tested with them. The cause is in PowerShell itself, not in this kit: a scriptblock stays tied to the runspace that made it. So any runspace-based tool that lets a scriptblock cross from one runspace to another can hit it. This kit refuses to launch when it sees one.
+Yes, both. Neither one rebuilds or refuses a scriptblock you pass to a job (by `-ArgumentList` or `$using:`), so it still runs against your script's session state, just from the job's thread. [The two rules](#the-two-rules-for-worker-code) apply to them as well; they just don't check for you. `ForEach-Object -Parallel` does refuse a `$using:` scriptblock, and in [PowerShell issue #12378](https://github.com/PowerShell/PowerShell/issues/12378) a PowerShell team member wrote of that check: "this should also be disallowed for Start-ThreadJob."
+
+<details>
+<summary>What the tests showed</summary>
+
+In tests with 8 jobs, a scriptblock passed as a job value crashed PowerShell or broke the calling script (its own functions vanished mid-run) in nearly every run where the script was doing anything else at the same time. ThreadJob only got away with it when the script did nothing but sit in `Wait-Job` until the jobs finished. PoshRSJob never did, since `Start-RSJob` is itself PowerShell code that's still running while the first jobs start. A delegate made in your script was handed back to your script's thread to run. With ThreadJob that gave wrong results in some runs while the script was busy, and a deadlock in nearly every run while it waited in `Wait-Job`. With PoshRSJob it failed every time. Passed as text and rebuilt in the job, the same code ran clean every time.
+
+The test scripts and results are in [`Testing/third-party`](Testing/third-party).
+
+</details>
 
 **Does it work in PowerShell 7, or on Linux or macOS?**
 
@@ -379,7 +402,7 @@ No. Programs start without a window, in a hidden console of their own, so Ctrl+C
 
 **Was this written by AI?**
 
-It was built with Claude Code, as the top of this page says. The worker rules and abort behaviour come from hangs and bugs reproduced in real runs of the original project, and the test harness runs real processes and runspaces, with nothing mocked, on every change.
+It was built with Claude Code, as the top of this page says. The worker rules and abort behaviour come from hangs and bugs reproduced in real runs, and the test harness runs real processes and runspaces, with nothing mocked, on every change.
 
 ## Bugs and questions
 

Testing/cmdlet-hang/inner.ps1

@@ -0,0 +1,136 @@
+# One stress trial, run in a fresh powershell.exe by run.ps1.
+# Runs -Rounds separate Invoke-PkRunspaceJobs calls (a new runspace pool, so
+# fresh runspaces, each round). In each round all -Workers workers wait on a
+# gate, then are released at the same instant into the -Mode body - the
+# "first cmdlet call in two fresh runspaces at once" case this was written to
+# test. Progress goes to -ProgressFile so the runner can tell which
+# round a hung trial was in.
+param(
+    [Parameter(Mandatory)][string]$KitPath,
+    [Parameter(Mandatory)][string]$Mode,
+    [int]$Workers = 8,
+    [int]$Rounds = 10,
+    [Parameter(Mandatory)][string]$ProgressFile
+)
+$ErrorActionPreference = 'Stop'
+Set-StrictMode -Version Latest
+. $KitPath
+
+function Write-Progress2([string]$line) { [System.IO.File]::AppendAllText($ProgressFile, $line + "`r`n") }
+
+$tempDir = Join-Path $env:TEMP ('pk-cmdlet-hang-' + [guid]::NewGuid().ToString('N'))
+$null = New-Item -ItemType Directory -Path $tempDir
+
+# Bodies run in the worker right after the gate opens. Each is the first use
+# of its cmdlets in that runspace. Control uses no cmdlets at all.
+$bodies = @{
+    Control    = '$n = 0; for ($i = 0; $i -lt 1000; $i++) { $n += $i }; $null = [System.IO.Directory]::Exists($TempDir); $null = [DateTime]::Now'
+    Core       = '$null = 1 | Out-Null; $null = @(1, 2, 3) | ForEach-Object { $_ * 2 } | Where-Object { $_ -gt 2 }; $null = Write-Output 1'
+    Utility    = 'Start-Sleep -Milliseconds 1; $null = Get-Date; $null = 1..5 | Measure-Object -Sum; $null = 1..5 | Select-Object -First 2; $null = 1..5 | Sort-Object -Descending'
+    Management = '$null = Test-Path -LiteralPath $TempDir; $null = Get-ChildItem -LiteralPath $env:SystemRoot -Filter *.exe; $null = Join-Path $TempDir x; $null = Get-Item -LiteralPath $TempDir'
+    FileIO     = '$p = Join-Path $TempDir ("w" + $Index + ".log"); "line" | Out-File -FilePath $p -Append; Add-Content -Path $p -Value more; $null = Get-Content -Path $p'
+}
+$bodies['Mixed'] = $bodies['Utility'] + '; ' + $bodies['Management'] + '; ' + $bodies['FileIO'] + '; ' + $bodies['Core']
+
+# Positive control: a known hang (README rule 2, a deadlock confirmed
+# from a memory dump), to prove this rig can see one. Worker 0 builds a
+# Lazy whose factory is a scriptblock delegate that itself hands another
+# scriptblock delegate to AddOrUpdate, and publishes it. Then every worker
+# forces .Value on 200 of them, in different orders, so a factory often runs
+# on a thread that isn't its own runspace's while that runspace waits on the
+# same Lazy.
+# The Delegate* modes differ only in what the factory does (FACTORY_BODY):
+#   DelegatePlain  - .NET calls only
+#   DelegateCmdlet - cmdlets that take no scriptblock
+#   DelegateWhere  - cmdlets that take a scriptblock (Where-Object/ForEach-Object)
+#   DelegateNested - a nested scriptblock delegate (AddOrUpdate), no cmdlets
+$factoryBodies = @{
+    DelegatePlain  = '$n = 0; for ($j = 0; $j -lt 50; $j++) { $n += $j }; $null = [DateTime]::Now'
+    DelegateCmdlet = '$null = Get-Date; $null = Test-Path -LiteralPath $TempDir; $null = 1 | Out-Null'
+    DelegateWhere  = '$null = @(1, 2, 3) | Where-Object { $_ -gt 1 }; $null = @(1, 2) | ForEach-Object { $_ }'
+    DelegateNested = '$null = $dict.AddOrUpdate(''k'', ''a'', [Func[string, string, string]] { param($k, $old) $old + ''b'' })'
+}
+$delegateBody = @'
+$count = 200
+if ($Index -eq 0) {
+    $dict = [System.Collections.Concurrent.ConcurrentDictionary[string, string]]::new()
+    $factory = [Func[object]] {
+        FACTORY_BODY
+        42
+    }
+    for ($i = 0; $i -lt $count; $i++) {
+        $Shared[[string]$i] = [System.Lazy[object]]::new($factory, [System.Threading.LazyThreadSafetyMode]::ExecutionAndPublication)
+    }
+    $Gate2.Set()
+} else {
+    $Gate2.Wait()
+}
+for ($n = 0; $n -lt $count; $n++) {
+    if ($Index % 2 -eq 0) { $i = $n } else { $i = $count - 1 - $n }
+    $null = $Shared[[string]$i].Value
+}
+'@
+
+# Job-parameter modes: a scriptblock made here on the main thread (so bound
+# to the main runspace) is passed as a job value and run with & in every
+# worker at once, while the main thread sits in the kit's poll loop.
+#   JobParamPlain  - .NET calls only
+#   JobParamCmdlet - cmdlets that take no scriptblock
+#   JobParamWhere  - Where-Object/ForEach-Object with an inner scriptblock
+#   JobParamText   - control: the JobParamWhere text, [scriptblock]::Create'd
+#                    in the worker (the proposed workaround)
+$jobFns = @{
+    JobParamPlain  = { param($i) $n = 0; for ($j = 0; $j -lt 50; $j++) { $n += $j }; $n + $i }
+    JobParamCmdlet = { param($i) $null = Get-Date; $null = Test-Path -LiteralPath $env:SystemRoot; $null = 1 | Out-Null; $i }
+    JobParamWhere  = { param($i) @(1, 2, 3) | Where-Object { $_ -gt 1 } | ForEach-Object { $_ + $i } }
+}
+$jobFns['JobParamText'] = $jobFns['JobParamWhere'].ToString()
+$jobParamBody = 'if ($Fn -is [string]) { $Fn = [scriptblock]::Create($Fn) }; for ($k = 0; $k -lt 500; $k++) { $null = & $Fn $k }'
+$fn = $null
+
+if ($factoryBodies.ContainsKey($Mode)) { $body = $delegateBody.Replace('FACTORY_BODY', $factoryBodies[$Mode]) }
+elseif ($bodies.ContainsKey($Mode)) { $body = $bodies[$Mode] }
+elseif ($jobFns.ContainsKey($Mode)) { $body = $jobParamBody; $fn = $jobFns[$Mode] }
+else { throw "Unknown -Mode '$Mode'. Use one of: $(@($bodies.Keys + $factoryBodies.Keys + $jobFns.Keys) -join ', ')." }
+
+$workerText = 'param($Ready, $Gate, $Gate2, $Shared, $Index, $TempDir, $Fn)' + "`r`n" +
+    '$null = $Ready.Signal(); $Gate.Wait()' + "`r`n" + $body
+$worker = [scriptblock]::Create($workerText)
+
+$exitCode = 0
+try {
+    for ($round = 1; $round -le $Rounds; $round++) {
+        Write-Progress2 "round $round start"
+        $ready  = [System.Threading.CountdownEvent]::new($Workers)
+        $gate   = [System.Threading.ManualResetEventSlim]::new($false)
+        $gate2  = [System.Threading.ManualResetEventSlim]::new($false)
+        $shared = [System.Collections.Concurrent.ConcurrentDictionary[string, object]]::new()
+        $jobs = [System.Collections.Generic.List[hashtable]]::new()
+        for ($i = 0; $i -lt $Workers; $i++) {
+            $jobs.Add(@{ Ready = $ready; Gate = $gate; Gate2 = $gate2; Shared = $shared; Index = $i; TempDir = $tempDir; Fn = $fn })
+        }
+        $state = @{ Started = [System.Diagnostics.Stopwatch]::StartNew(); GateTimedOut = $false }
+        # Open the gate once every worker is waiting at it. If one never gets
+        # there in 15s, open it anyway and say so - that's a startup stall,
+        # not the race this test is after.
+        $poll = {
+            if (-not $gate.IsSet) {
+                if ($ready.CurrentCount -eq 0) { $gate.Set() }
+                elseif ($state.Started.Elapsed.TotalSeconds -gt 15) { $state.GateTimedOut = $true; $gate.Set() }
+            }
+        }.GetNewClosure()
+        $crashes = [System.Collections.Generic.List[string]]::new()
+        $out = Invoke-PkRunspaceJobs -MaxParallel $Workers -JobParams $jobs -WorkerScript $worker `
+            -PollCallback $poll -JobCrashedHandler { param($err) $crashes.Add([string]$err) }.GetNewClosure()
+        if ($state.GateTimedOut) { Write-Progress2 "round $round gate-timeout" }
+        foreach ($c in $crashes) { Write-Progress2 "round $round crash: $c" }
+        if ($crashes.Count -gt 0) { $exitCode = 3 }
+        Write-Progress2 ("round $round done {0:N0}ms" -f $state.Started.Elapsed.TotalMilliseconds)
+    }
+} catch {
+    Write-Progress2 "error: $_"
+    $exitCode = 4
+} finally {
+    try { Remove-Item -LiteralPath $tempDir -Recurse -Force } catch { }  # best effort; temp dir only
+}
+exit $exitCode