diff --git a/README.md b/README.md index 54cce31..41a952a 100644 --- a/README.md +++ b/README.md @@ -62,10 +62,10 @@ Wrap any of them in `Invoke-PkRun { ... }` to get graceful Ctrl+C. (For `Invoke- For running PowerShell code in parallel, most of these are fine and some are simpler. ParallelKit is mainly about running external programs, and about what happens when a run is stopped partway through. -- **PowerShell 7's `ForEach-Object -Parallel`.** The easy answer for plain parallel PowerShell code if you can use PowerShell 7. ParallelKit was built for scripts and machines stuck on 5.1, though it runs on 7 too, where it still adds the external-program handling and graceful Ctrl+C. +- **PowerShell 7's `ForEach-Object -Parallel`.** The easy answer for plain parallel PowerShell code if you can use PowerShell 7. ParallelKit was built for scripts and machines stuck on 5.1, though it runs on 7 too, where it still adds the external-program handling and graceful Ctrl+C. `ForEach-Object -Parallel` also refuses a scriptblock passed by `$using:`, rather than letting it run where it isn't safe. - **`Start-Job`.** Built in, but each job is a whole new `powershell.exe` process, so every job costs noticeable startup time and memory, and what it returns comes back as serialized copies (properties only, no methods). 5.1's `Start-Job` has no throttle either, so you write the wait-for-a-free-slot loop yourself. Fine for a handful of long jobs, poor for thousands of short ones. -- **ThreadJob (`Start-ThreadJob`).** Runs jobs as runspaces in the same process, with a `-ThrottleLimit`. On 5.1 it's the best fit for plain parallel scriptblocks, if you can install a module from the PowerShell Gallery (it ships with PowerShell 7, not 5.1). -- **PoshRSJob (`Start-RSJob`).** A community module from the Gallery, also runspace-based, with a `-Throttle` limit and a job-style interface. For this purpose it has the same strengths and gaps as ThreadJob. +- **ThreadJob (`Start-ThreadJob`).** Runs jobs as runspaces in the same process, with a `-ThrottleLimit`. On 5.1 it's the best fit for plain parallel scriptblocks, if you can install a module from the PowerShell Gallery (`Microsoft.PowerShell.ThreadJob`; it ships with PowerShell 7, not 5.1). +- **PoshRSJob (`Start-RSJob`).** A community module from the Gallery, also runspace-based, with a `-Throttle` limit and a job-style interface. It hasn't had a release since 2018, it can report a job as Completed before its output is ready (run `Wait-RSJob` before `Receive-RSJob`), and it's hit harder than ThreadJob by the [scriptblock problem](#faq). Where both are an option, ThreadJob is the better pick. - **A runspace pool by hand** (`[runspacefactory]::CreateRunspacePool`). That's what `Invoke-PkRunspaceJobs` is built on. It works, but you end up writing the polling loop, error collection, cancellation and cleanup yourself, which is most of what this file is. - **Workflows (`foreach -parallel`).** Still in 5.1, but they use a restricted version of the language, start slowly, and were dropped from PowerShell 6 and later. Not worth starting anything new on. - **`Start-Process` in a loop.** Starts programs without waiting, but can only send their output to files, and the throttling and waiting are up to you. Using `System.Diagnostics.Process` directly gets output into memory, but reading stdout and then stderr the simple way deadlocks once a program fills the pipe you aren't reading. The kit reads both in the background. @@ -267,7 +267,7 @@ Callbacks: a plain `{ ... }` callback sees your variables and functions from whe `Invoke-PkProcessWithTimeout` and `Start-PkProcess` also take `-WorkingDirectory`. -All four are safe in worker code. +All of these are safe in worker code. ## Messages @@ -299,7 +299,8 @@ Three things to know: - **Your script carries on after an abort.** The engines return with `Aborted = $true` and the rest of your script runs. If an abort should end it, say so: `if ($batch.Aborted) { exit 1 }`. `Invoke-PkRun` also turns off the console's QuickEdit mode for the length of the run (`-DisableQuickEdit $false` to opt out). With QuickEdit on in the classic console window, one stray click starts a text selection, and the script freezes at its next console write until the selection is cleared. That looks exactly like a hang. It's turned back on afterwards if it was on before. -Other pieces: `Request-PkAbort` triggers the same abort from code. `Test-PkAbortSignal` asks whether one is under way. `Register-PkCancellationSource` adds your own `CancellationTokenSource` to the set an abort cancels (a `-Cts` you pass to `Invoke-PkRunspaceJobs` is added automatically; a `-CancelToken` you pass to the dispatcher isn't cancelled by an abort, but the dispatcher stops either way). `Get-PkAbortTriggerLabel` says which kind of abort it was. + +Other pieces: `Request-PkAbort` triggers the same abort from code. `Test-PkAbortSignal` asks whether one is under way. `Register-PkCancellationSource` adds your own `CancellationTokenSource` to the set an abort cancels, and `Unregister-PkCancellationSource` takes it back out (a `-Cts` you pass to `Invoke-PkRunspaceJobs` is added automatically; a `-CancelToken` you pass to the dispatcher isn't cancelled by an abort, but the dispatcher stops either way). `Get-PkAbortTriggerLabel` says which kind of abort it was. ## Choosing limits @@ -327,10 +328,23 @@ Other pieces: `Request-PkAbort` triggers the same abort from code. `Test-PkAbort powershell -NoProfile -File .\ParallelKit.Harness.ps1 ``` -It uses real runspaces and real child processes (`cmd.exe`, `ping.exe`), with nothing mocked, so it takes about a minute. It prints a `[PASS]`/`[FAIL]` line per check and exits 0 only if everything passed. A few checks send real Ctrl+C, Ctrl+Break and window-close events to child PowerShell processes in hidden consoles of their own, never to yours. Where a machine can't deliver those, they print `[SKIP]` with the reason instead; set `PK_HARNESS_NO_SKIP=1` to count skips as failures. A few `message handler failed` lines in the output are expected: some checks break the message handler on purpose to test the fallback. +It uses real runspaces and real child processes (`cmd.exe`, `ping.exe`), with nothing mocked, so it takes about two minutes. It prints a `[PASS]`/`[FAIL]` line per check and exits 0 only if everything passed. A few checks send real Ctrl+C, Ctrl+Break and window-close events to child PowerShell processes in hidden consoles of their own, never to yours. Where a machine can't deliver those, they print `[SKIP]` with the reason instead; set `PK_HARNESS_NO_SKIP=1` to count skips as failures. A few `message handler failed` lines in the output are expected: some checks break the message handler on purpose to test the fallback. GitHub runs the same harness on Windows PowerShell 5.1 and on PowerShell 7 for every pull request and every change to `main`. +`Testing/` holds the longer stress tests and checks behind some of what this README says. They aren't part of the test suite and aren't needed to use the kit: + +- `Testing/cmdlet-hang/` stresses the kit's worker code, for the [two rules](#the-two-rules-for-worker-code). +- `Testing/third-party/` asks the same question of ThreadJob and PoshRSJob. `results/matrix.log` holds the results behind the FAQ answer, from `Microsoft.PowerShell.ThreadJob` 2.2.0 (the `ThreadJob` package below brings it along) and PoshRSJob 1.7.4.4. To run it yourself, save the modules first, in PowerShell 7.4 or later (which has `Save-PSResource`), then run `matrix.ps1`: + + ```powershell + New-Item -ItemType Directory -Force .\Testing\third-party\modules + Save-PSResource -Name ThreadJob, PoshRSJob -Path .\Testing\third-party\modules + powershell -NoProfile -File .\Testing\third-party\matrix.ps1 + ``` + +- `Testing/hosts/check-host.ps1` checks the kit in a host you choose, such as an editor's terminal or a scheduled task. + ## FAQ **Can I use cmdlets in worker code?** @@ -339,7 +353,16 @@ Yes. What a worker can't do is run a scriptblock that came from another runspace **Do ThreadJob and PoshRSJob have the same scriptblock problem?** -It hasn't been tested with them. The cause is in PowerShell itself, not in this kit: a scriptblock stays tied to the runspace that made it. So any runspace-based tool that lets a scriptblock cross from one runspace to another can hit it. This kit refuses to launch when it sees one. +Yes, both. Neither one rebuilds or refuses a scriptblock you pass to a job (by `-ArgumentList` or `$using:`), so it still runs against your script's session state, just from the job's thread. [The two rules](#the-two-rules-for-worker-code) apply to them as well; they just don't check for you. `ForEach-Object -Parallel` does refuse a `$using:` scriptblock, and in [PowerShell issue #12378](https://github.com/PowerShell/PowerShell/issues/12378) a PowerShell team member wrote of that check: "this should also be disallowed for Start-ThreadJob." + +
+What the tests showed + +In tests with 8 jobs, a scriptblock passed as a job value crashed PowerShell or broke the calling script (its own functions vanished mid-run) in nearly every run where the script was doing anything else at the same time. ThreadJob only got away with it when the script did nothing but sit in `Wait-Job` until the jobs finished. PoshRSJob never did, since `Start-RSJob` is itself PowerShell code that's still running while the first jobs start. A delegate made in your script was handed back to your script's thread to run. With ThreadJob that gave wrong results in some runs while the script was busy, and a deadlock in nearly every run while it waited in `Wait-Job`. With PoshRSJob it failed every time. Passed as text and rebuilt in the job, the same code ran clean every time. + +The test scripts and results are in [`Testing/third-party`](Testing/third-party). + +
**Does it work in PowerShell 7, or on Linux or macOS?** @@ -379,7 +402,7 @@ No. Programs start without a window, in a hidden console of their own, so Ctrl+C **Was this written by AI?** -It was built with Claude Code, as the top of this page says. The worker rules and abort behaviour come from hangs and bugs reproduced in real runs of the original project, and the test harness runs real processes and runspaces, with nothing mocked, on every change. +It was built with Claude Code, as the top of this page says. The worker rules and abort behaviour come from hangs and bugs reproduced in real runs, and the test harness runs real processes and runspaces, with nothing mocked, on every change. ## Bugs and questions diff --git a/Testing/cmdlet-hang/inner.ps1 b/Testing/cmdlet-hang/inner.ps1 new file mode 100644 index 0000000..be34852 --- /dev/null +++ b/Testing/cmdlet-hang/inner.ps1 @@ -0,0 +1,136 @@ +# One stress trial, run in a fresh powershell.exe by run.ps1. +# Runs -Rounds separate Invoke-PkRunspaceJobs calls (a new runspace pool, so +# fresh runspaces, each round). In each round all -Workers workers wait on a +# gate, then are released at the same instant into the -Mode body - the +# "first cmdlet call in two fresh runspaces at once" case this was written to +# test. Progress goes to -ProgressFile so the runner can tell which +# round a hung trial was in. +param( + [Parameter(Mandatory)][string]$KitPath, + [Parameter(Mandatory)][string]$Mode, + [int]$Workers = 8, + [int]$Rounds = 10, + [Parameter(Mandatory)][string]$ProgressFile +) +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +. $KitPath + +function Write-Progress2([string]$line) { [System.IO.File]::AppendAllText($ProgressFile, $line + "`r`n") } + +$tempDir = Join-Path $env:TEMP ('pk-cmdlet-hang-' + [guid]::NewGuid().ToString('N')) +$null = New-Item -ItemType Directory -Path $tempDir + +# Bodies run in the worker right after the gate opens. Each is the first use +# of its cmdlets in that runspace. Control uses no cmdlets at all. +$bodies = @{ + Control = '$n = 0; for ($i = 0; $i -lt 1000; $i++) { $n += $i }; $null = [System.IO.Directory]::Exists($TempDir); $null = [DateTime]::Now' + Core = '$null = 1 | Out-Null; $null = @(1, 2, 3) | ForEach-Object { $_ * 2 } | Where-Object { $_ -gt 2 }; $null = Write-Output 1' + Utility = 'Start-Sleep -Milliseconds 1; $null = Get-Date; $null = 1..5 | Measure-Object -Sum; $null = 1..5 | Select-Object -First 2; $null = 1..5 | Sort-Object -Descending' + Management = '$null = Test-Path -LiteralPath $TempDir; $null = Get-ChildItem -LiteralPath $env:SystemRoot -Filter *.exe; $null = Join-Path $TempDir x; $null = Get-Item -LiteralPath $TempDir' + FileIO = '$p = Join-Path $TempDir ("w" + $Index + ".log"); "line" | Out-File -FilePath $p -Append; Add-Content -Path $p -Value more; $null = Get-Content -Path $p' +} +$bodies['Mixed'] = $bodies['Utility'] + '; ' + $bodies['Management'] + '; ' + $bodies['FileIO'] + '; ' + $bodies['Core'] + +# Positive control: a known hang (README rule 2, a deadlock confirmed +# from a memory dump), to prove this rig can see one. Worker 0 builds a +# Lazy whose factory is a scriptblock delegate that itself hands another +# scriptblock delegate to AddOrUpdate, and publishes it. Then every worker +# forces .Value on 200 of them, in different orders, so a factory often runs +# on a thread that isn't its own runspace's while that runspace waits on the +# same Lazy. +# The Delegate* modes differ only in what the factory does (FACTORY_BODY): +# DelegatePlain - .NET calls only +# DelegateCmdlet - cmdlets that take no scriptblock +# DelegateWhere - cmdlets that take a scriptblock (Where-Object/ForEach-Object) +# DelegateNested - a nested scriptblock delegate (AddOrUpdate), no cmdlets +$factoryBodies = @{ + DelegatePlain = '$n = 0; for ($j = 0; $j -lt 50; $j++) { $n += $j }; $null = [DateTime]::Now' + DelegateCmdlet = '$null = Get-Date; $null = Test-Path -LiteralPath $TempDir; $null = 1 | Out-Null' + DelegateWhere = '$null = @(1, 2, 3) | Where-Object { $_ -gt 1 }; $null = @(1, 2) | ForEach-Object { $_ }' + DelegateNested = '$null = $dict.AddOrUpdate(''k'', ''a'', [Func[string, string, string]] { param($k, $old) $old + ''b'' })' +} +$delegateBody = @' +$count = 200 +if ($Index -eq 0) { + $dict = [System.Collections.Concurrent.ConcurrentDictionary[string, string]]::new() + $factory = [Func[object]] { + FACTORY_BODY + 42 + } + for ($i = 0; $i -lt $count; $i++) { + $Shared[[string]$i] = [System.Lazy[object]]::new($factory, [System.Threading.LazyThreadSafetyMode]::ExecutionAndPublication) + } + $Gate2.Set() +} else { + $Gate2.Wait() +} +for ($n = 0; $n -lt $count; $n++) { + if ($Index % 2 -eq 0) { $i = $n } else { $i = $count - 1 - $n } + $null = $Shared[[string]$i].Value +} +'@ + +# Job-parameter modes: a scriptblock made here on the main thread (so bound +# to the main runspace) is passed as a job value and run with & in every +# worker at once, while the main thread sits in the kit's poll loop. +# JobParamPlain - .NET calls only +# JobParamCmdlet - cmdlets that take no scriptblock +# JobParamWhere - Where-Object/ForEach-Object with an inner scriptblock +# JobParamText - control: the JobParamWhere text, [scriptblock]::Create'd +# in the worker (the proposed workaround) +$jobFns = @{ + JobParamPlain = { param($i) $n = 0; for ($j = 0; $j -lt 50; $j++) { $n += $j }; $n + $i } + JobParamCmdlet = { param($i) $null = Get-Date; $null = Test-Path -LiteralPath $env:SystemRoot; $null = 1 | Out-Null; $i } + JobParamWhere = { param($i) @(1, 2, 3) | Where-Object { $_ -gt 1 } | ForEach-Object { $_ + $i } } +} +$jobFns['JobParamText'] = $jobFns['JobParamWhere'].ToString() +$jobParamBody = 'if ($Fn -is [string]) { $Fn = [scriptblock]::Create($Fn) }; for ($k = 0; $k -lt 500; $k++) { $null = & $Fn $k }' +$fn = $null + +if ($factoryBodies.ContainsKey($Mode)) { $body = $delegateBody.Replace('FACTORY_BODY', $factoryBodies[$Mode]) } +elseif ($bodies.ContainsKey($Mode)) { $body = $bodies[$Mode] } +elseif ($jobFns.ContainsKey($Mode)) { $body = $jobParamBody; $fn = $jobFns[$Mode] } +else { throw "Unknown -Mode '$Mode'. Use one of: $(@($bodies.Keys + $factoryBodies.Keys + $jobFns.Keys) -join ', ')." } + +$workerText = 'param($Ready, $Gate, $Gate2, $Shared, $Index, $TempDir, $Fn)' + "`r`n" + + '$null = $Ready.Signal(); $Gate.Wait()' + "`r`n" + $body +$worker = [scriptblock]::Create($workerText) + +$exitCode = 0 +try { + for ($round = 1; $round -le $Rounds; $round++) { + Write-Progress2 "round $round start" + $ready = [System.Threading.CountdownEvent]::new($Workers) + $gate = [System.Threading.ManualResetEventSlim]::new($false) + $gate2 = [System.Threading.ManualResetEventSlim]::new($false) + $shared = [System.Collections.Concurrent.ConcurrentDictionary[string, object]]::new() + $jobs = [System.Collections.Generic.List[hashtable]]::new() + for ($i = 0; $i -lt $Workers; $i++) { + $jobs.Add(@{ Ready = $ready; Gate = $gate; Gate2 = $gate2; Shared = $shared; Index = $i; TempDir = $tempDir; Fn = $fn }) + } + $state = @{ Started = [System.Diagnostics.Stopwatch]::StartNew(); GateTimedOut = $false } + # Open the gate once every worker is waiting at it. If one never gets + # there in 15s, open it anyway and say so - that's a startup stall, + # not the race this test is after. + $poll = { + if (-not $gate.IsSet) { + if ($ready.CurrentCount -eq 0) { $gate.Set() } + elseif ($state.Started.Elapsed.TotalSeconds -gt 15) { $state.GateTimedOut = $true; $gate.Set() } + } + }.GetNewClosure() + $crashes = [System.Collections.Generic.List[string]]::new() + $out = Invoke-PkRunspaceJobs -MaxParallel $Workers -JobParams $jobs -WorkerScript $worker ` + -PollCallback $poll -JobCrashedHandler { param($err) $crashes.Add([string]$err) }.GetNewClosure() + if ($state.GateTimedOut) { Write-Progress2 "round $round gate-timeout" } + foreach ($c in $crashes) { Write-Progress2 "round $round crash: $c" } + if ($crashes.Count -gt 0) { $exitCode = 3 } + Write-Progress2 ("round $round done {0:N0}ms" -f $state.Started.Elapsed.TotalMilliseconds) + } +} catch { + Write-Progress2 "error: $_" + $exitCode = 4 +} finally { + try { Remove-Item -LiteralPath $tempDir -Recurse -Force } catch { } # best effort; temp dir only +} +exit $exitCode diff --git a/Testing/cmdlet-hang/run.ps1 b/Testing/cmdlet-hang/run.ps1 new file mode 100644 index 0000000..eac77c0 --- /dev/null +++ b/Testing/cmdlet-hang/run.ps1 @@ -0,0 +1,88 @@ +# Stress runner for the cmdlet-hang question. Starts -Trials fresh +# powershell.exe processes running inner.ps1 (-Concurrency at a time). A trial +# still running after -TimeoutSec counts as a hang: the first -DumpFirst hung +# processes get a full memory dump, then it's killed. +# powershell -NoProfile -File .\Testing\cmdlet-hang\run.ps1 -Mode Mixed -Trials 200 +param( + [Parameter(Mandatory)][string]$Mode, + [int]$Trials = 100, + [int]$Workers = 8, + [int]$Rounds = 10, + [int]$Concurrency = 3, + [int]$TimeoutSec = 60, + [int]$DumpFirst = 2, + [string]$OutDir = '' +) +$ErrorActionPreference = 'Stop' +if ($OutDir -eq '') { $OutDir = Join-Path $PSScriptRoot 'results' } + +if (-not ('PkStressDump' -as [type])) { + Add-Type @' +using System; +using System.IO; +using System.Runtime.InteropServices; +public static class PkStressDump { + [DllImport("dbghelp.dll", SetLastError = true)] + static extern bool MiniDumpWriteDump(IntPtr hProcess, uint processId, Microsoft.Win32.SafeHandles.SafeFileHandle hFile, + uint dumpType, IntPtr exceptionParam, IntPtr userStreamParam, IntPtr callbackParam); + public static bool Write(System.Diagnostics.Process p, string path) { + using (FileStream fs = new FileStream(path, FileMode.Create, FileAccess.ReadWrite, FileShare.None)) { + // MiniDumpWithFullMemory | MiniDumpWithHandleData | MiniDumpWithThreadInfo + return MiniDumpWriteDump(p.Handle, (uint)p.Id, fs.SafeFileHandle, 0x2 | 0x4 | 0x1000, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero); + } + } +} +'@ +} + +$kit = (Resolve-Path (Join-Path $PSScriptRoot '..\..\ParallelKit.ps1')).Path +$inner = Join-Path $PSScriptRoot 'inner.ps1' +$stamp = Get-Date -Format 'yyyyMMdd-HHmmss' +$runDir = Join-Path $OutDir ("$Mode-$stamp") +$null = New-Item -ItemType Directory -Path $runDir -Force + +$results = [System.Collections.Generic.List[object]]::new() +$running = [System.Collections.Generic.List[hashtable]]::new() +$next = 1; $dumps = 0 +$clock = [System.Diagnostics.Stopwatch]::StartNew() + +while ($next -le $Trials -or $running.Count -gt 0) { + while ($next -le $Trials -and $running.Count -lt $Concurrency) { + $progress = Join-Path $runDir ("trial-$next.txt") + $argLine = ('-NoProfile -File "{0}" -KitPath "{1}" -Mode {2} -Workers {3} -Rounds {4} -ProgressFile "{5}"' -f + $inner, $kit, $Mode, $Workers, $Rounds, $progress) + $p = Start-Process -FilePath powershell.exe -ArgumentList $argLine -WindowStyle Hidden -PassThru + $running.Add(@{ Trial = $next; Proc = $p; Watch = [System.Diagnostics.Stopwatch]::StartNew(); Progress = $progress }) + $next++ + } + Start-Sleep -Milliseconds 200 + foreach ($t in @($running)) { + $outcome = $null + if ($t.Proc.HasExited) { + $outcome = switch ($t.Proc.ExitCode) { 0 { 'ok' } 3 { 'crash' } default { 'error' } } + } elseif ($t.Watch.Elapsed.TotalSeconds -gt $TimeoutSec) { + $outcome = 'hang' + if ($dumps -lt $DumpFirst) { + $dumpPath = Join-Path $runDir ("trial-$($t.Trial)-pid$($t.Proc.Id).dmp") + try { if ([PkStressDump]::Write($t.Proc, $dumpPath)) { $dumps++ } } catch { Write-Warning "dump failed: $_" } + } + try { $t.Proc.Kill() } catch { } # it may have exited between the check and the kill + } + if ($null -eq $outcome) { continue } + $lines = @(); if (Test-Path $t.Progress) { $lines = @(Get-Content $t.Progress) } + $lastRound = ($lines | Where-Object { $_ -match '^round (\d+) start' } | Select-Object -Last 1) -replace '^round (\d+).*', '$1' + $notes = ($lines | Where-Object { $_ -match 'gate-timeout|crash|error' }) -join ' | ' + if ($outcome -eq 'error') { $notes = ('exit 0x{0:X8} ' -f $t.Proc.ExitCode) + $notes } + $results.Add([pscustomobject]@{ Trial = $t.Trial; Outcome = $outcome; Round = $lastRound; Seconds = [math]::Round($t.Watch.Elapsed.TotalSeconds, 1); Notes = $notes }) + if ($outcome -ne 'ok') { Write-Host ("trial {0}: {1} in round {2} {3}" -f $t.Trial, $outcome, $lastRound, $notes) -ForegroundColor Yellow } + $null = $running.Remove($t) + } +} + +$results | Sort-Object Trial | Export-Csv -NoTypeInformation -Path (Join-Path $runDir 'results.csv') +$hangs = @($results | Where-Object Outcome -eq 'hang').Count +$other = @($results | Where-Object { $_.Outcome -ne 'ok' -and $_.Outcome -ne 'hang' }).Count +$gateTimeouts = @($results | Where-Object { $_.Notes -match 'gate-timeout' }).Count +$rounds = $Trials * $Rounds +"{0}: {1} hangs, {2} crash/error, {3} gate timeouts in {4} trials ({5} rounds of {6} workers) - {7:N0}s. Results: {8}" -f + $Mode, $hangs, $other, $gateTimeouts, $Trials, $rounds, $Workers, $clock.Elapsed.TotalSeconds, $runDir diff --git a/Testing/hosts/check-host.ps1 b/Testing/hosts/check-host.ps1 new file mode 100644 index 0000000..d8034cd --- /dev/null +++ b/Testing/hosts/check-host.ps1 @@ -0,0 +1,43 @@ +# Host check for ParallelKit: run this in the host you want to check (the +# ISE, the VS Code PowerShell terminal, a scheduled task) and look at the +# log it writes next to itself, check-host.log. +# +# Interactive hosts: when it says "press Ctrl+C now", press it once. A +# graceful abort shows "[ABORT] Ctrl+C detected" and the log ends with +# aborted=True and a time well under 30s. If Ctrl+C kills the script +# outright instead, the log stops at "batch started". +# +# Scheduled task: run it as +# powershell.exe -NoProfile -ExecutionPolicy Bypass -File "\check-host.ps1" -NoCtrlC +# and check that the log reaches "done". +param([switch]$NoCtrlC) + +$log = Join-Path $PSScriptRoot 'check-host.log' +function Write-Log([string]$line) { Add-Content -LiteralPath $log -Value ((Get-Date -Format 'HH:mm:ss') + ' ' + $line); Write-Host $line } +Remove-Item -LiteralPath $log -ErrorAction SilentlyContinue + +try { + Write-Log ("host: " + $Host.Name + " | PowerShell " + $PSVersionTable.PSVersion + " | interactive: " + [Environment]::UserInteractive) + . (Join-Path $PSScriptRoot '..\..\ParallelKit.ps1') + Write-Log "kit loaded" + + $init = Initialize-ParallelKit + Write-Log ("Initialize-ParallelKit: AbortHandler=" + $init.AbortHandler + " ChildProcessJob=" + $init.ChildProcessJob + " ThreadPool=" + $init.ThreadPool) + + $ping = Join-Path $env:SystemRoot 'System32\ping.exe' + $seconds = 30 + if ($NoCtrlC) { $seconds = 3 } + $commands = @(1..2 | ForEach-Object { @{ FilePath = $ping; Arguments = ('-n ' + $seconds + ' 127.0.0.1') } }) + if (-not $NoCtrlC) { Write-Log ("press Ctrl+C now (the programs run for " + $seconds + "s)") } + Write-Log "batch started" + $sw = [System.Diagnostics.Stopwatch]::StartNew() + $batch = Invoke-PkRun { Invoke-PkProcessBatch -Commands $commands -MaxConcurrent 2 -AbortTimeoutSeconds 5 } + Write-Log ("batch returned after " + [int]$sw.Elapsed.TotalSeconds + "s: aborted=" + $batch.Aborted + " results=" + $batch.Results.Count + " exit codes=" + (@($batch.Results | ForEach-Object { $_.ExitCode }) -join ',')) + + Reset-PkAbortState + $jobs = Invoke-PkRunspaceJobs -MaxParallel 2 -JobParams @(@{ Id = 1 }, @{ Id = 2 }) -WorkerScript { param($Id) Write-PkMessage ("worker " + $Id); $Id * 10 } -ResultHandler { param($r) Write-Log ("worker result: " + $r) } + Write-Log ("runspace jobs: crashed=" + $jobs.CrashedCount + " aborted=" + $jobs.Aborted) + Write-Log "done" +} catch { + Write-Log ("ERROR: " + $_) +} diff --git a/Testing/third-party/inner.ps1 b/Testing/third-party/inner.ps1 new file mode 100644 index 0000000..d2fd3c6 --- /dev/null +++ b/Testing/third-party/inner.ps1 @@ -0,0 +1,176 @@ +# One trial of the ThreadJob / PoshRSJob scriptblock question, run in a fresh +# powershell.exe or pwsh.exe by run.ps1. Starts -Workers jobs with the given +# -Tool. Each job gets a payload made in this (the main) runspace and uses it +# -Iterations times: +# JobValue a scriptblock made here, passed in and run with & $sb +# Delegate a [Func[object]] made here from a scriptblock that runs a +# nested scriptblock (Where-Object { }), called with .Invoke() +# TextJobValue control: the JobValue block's text, rebuilt in the job +# TextDelegate control: the Delegate block's text, rebuilt and cast in the job +# ProbeJobValue / ProbeDelegate one call, reporting which session state and +# thread the payload ran on +# -Via says how the payload crosses: Args (-ArgumentList) or Using ($using:). +# -Main says what the main thread does meanwhile: Busy (loops over its own +# functions, like a dispatch loop) or Wait (blocked in Wait-Job / Wait-RSJob). +# Exit 0 = every iteration gave the right answer and the main runspace is +# intact; 2 = wrong answers, job errors or main-runspace damage; anything else +# means the process died. Progress goes to -ProgressFile. +param( + [Parameter(Mandatory)][ValidateSet('ThreadJob', 'RSJob')][string]$Tool, + [Parameter(Mandatory)][ValidateSet('JobValue', 'Delegate', 'TextJobValue', 'TextDelegate', 'ProbeJobValue', 'ProbeDelegate')][string]$Case, + [ValidateSet('Args', 'Using')][string]$Via = 'Args', + [ValidateSet('Busy', 'Wait')][string]$Main = 'Busy', + [int]$Workers = 8, + [int]$Iterations = 2000, + [int]$WaitTimeoutSec = 40, + [Parameter(Mandatory)][string]$ModuleDir, + [Parameter(Mandatory)][string]$ProgressFile +) +$ErrorActionPreference = 'Stop' + +[AppDomain]::CurrentDomain.SetData('pkProgressFile', $ProgressFile) +function Write-TrialLine([string]$line) { [System.IO.File]::AppendAllText($ProgressFile, $line + "`r`n") } +function Test-MainAlive { return 42 } + +$global:pkMainOnly = 'set only in the main runspace' +$mainThread = [System.Threading.Thread]::CurrentThread.ManagedThreadId + +try { + if ($Tool -eq 'ThreadJob') { + Import-Module (Join-Path $ModuleDir 'Microsoft.PowerShell.ThreadJob\2.2.0\Microsoft.PowerShell.ThreadJob.psd1') -Force + } else { + Import-Module (Join-Path $ModuleDir 'PoshRSJob\1.7.4.4\PoshRSJob.psd1') -Force + } + + # Literal blocks, so they're bound to this runspace's session state (a + # [scriptblock]::Create block is unbound, which is the safe text path). + $jobValueBlock = { $x = [Math]::Sqrt(16); $null = [string]::Concat("a", "b"); $x } + $delegateBlock = { @(1, 2, 3) | Where-Object { $_ -gt 1 } | Measure-Object | ForEach-Object Count } + $probeBlock = { [pscustomobject]@{ SeesMainVar = (Test-Path variable:global:pkMainOnly); Thread = [System.Threading.Thread]::CurrentThread.ManagedThreadId } } + + $payload = switch ($Case) { + 'JobValue' { $jobValueBlock } + 'Delegate' { [Func[object]]$delegateBlock } + 'TextJobValue' { $jobValueBlock.ToString() } + 'TextDelegate' { $delegateBlock.ToString() } + 'ProbeJobValue' { $probeBlock } + 'ProbeDelegate' { [Func[object]]$probeBlock } + } + + $body = @' +$ok = 0; $errCount = 0; $errs = [System.Collections.Generic.List[string]]::new() +$f = switch ($Case) { + 'TextJobValue' { [scriptblock]::Create($Payload) } + 'TextDelegate' { [Func[object]][scriptblock]::Create($Payload) } + default { $Payload } +} +$isDelegate = $Case -in 'Delegate', 'TextDelegate', 'ProbeDelegate' +$expected = if ($Case -like '*JobValue') { 4 } else { 2 } +if ($Case -like 'Probe*') { + $r = if ($isDelegate) { $f.Invoke() } else { & $f } + $r = @($r)[0] + return [pscustomobject]@{ Ok = 1; ErrorCount = 0; Errors = ''; PayloadType = $Payload.GetType().FullName + SeesMainVar = $r.SeesMainVar; PayloadThread = $r.Thread; JobThread = [System.Threading.Thread]::CurrentThread.ManagedThreadId + JobSeesMainVar = (Test-Path variable:global:pkMainOnly) } +} +for ($i = 0; $i -lt $Iterations; $i++) { + try { + $r = if ($isDelegate) { $f.Invoke() } else { & $f } + if ("$r" -eq "$expected") { $ok++ } else { $errCount++; if ($errs.Count -lt 3) { $errs.Add("wrong result '$r'") } } + } catch { + $errCount++; if ($errs.Count -lt 3) { $errs.Add($_.Exception.GetType().Name + ': ' + $_.Exception.Message) } + } +} +[pscustomobject]@{ Ok = $ok; ErrorCount = $errCount; Errors = ($errs -join ' | '); PayloadType = $Payload.GetType().FullName } +'@ + if ($Via -eq 'Args') { + $worker = [scriptblock]::Create("param(`$Payload, `$Case, `$Iterations)`n" + $body) + } else { + $worker = [scriptblock]::Create("`$Payload = `$using:payload; `$Case = `$using:Case; `$Iterations = `$using:Iterations`n" + $body) + } + + $n = if ($Case -like 'Probe*') { 1 } else { $Workers } + Write-TrialLine "start tool=$Tool case=$Case via=$Via main=$Main workers=$n ps=$($PSVersionTable.PSVersion) mainThread=$mainThread" + $jobs = @(for ($w = 0; $w -lt $n; $w++) { + if ($Tool -eq 'ThreadJob') { + if ($Via -eq 'Args') { Start-ThreadJob -ScriptBlock $worker -ArgumentList $payload, $Case, $Iterations -ThrottleLimit $n } + else { Start-ThreadJob -ScriptBlock $worker -ThrottleLimit $n } + } else { + if ($Via -eq 'Args') { Start-RSJob -ScriptBlock $worker -ArgumentList $payload, $Case, $Iterations -Throttle $n } + else { Start-RSJob -ScriptBlock $worker -Throttle $n } + } + }) + Write-TrialLine "started $($jobs.Count) jobs" + + $damage = [System.Collections.Generic.List[string]]::new() + $clock = [System.Diagnostics.Stopwatch]::StartNew() + $timedOut = $false + if ($Main -eq 'Wait') { + if ($Tool -eq 'ThreadJob') { $null = Wait-Job -Job $jobs -Timeout $WaitTimeoutSec } + else { $null = $jobs | Wait-RSJob -Timeout $WaitTimeoutSec } + } else { + $loops = 0 + while ($true) { + $loops++ + if ((Test-MainAlive) -ne 42) { $damage.Add('Test-MainAlive returned the wrong value') } + if (-not (Get-Command Test-MainAlive -ErrorAction SilentlyContinue)) { $damage.Add("Test-MainAlive vanished at loop $loops"); break } + if ($global:pkMainOnly -ne 'set only in the main runspace') { $damage.Add("main variable changed at loop $loops"); break } + $null = @(1..20 | Where-Object { $_ % 2 }) + $pending = @($jobs | Where-Object { $_.State -in 'NotStarted', 'Running' }) + if ($pending.Count -eq 0) { break } + if ($clock.Elapsed.TotalSeconds -gt $WaitTimeoutSec) { break } + } + } + $pending = @($jobs | Where-Object { $_.State -in 'NotStarted', 'Running' }) + if ($pending.Count -gt 0) { + $timedOut = $true + Write-TrialLine "timeout: $($pending.Count) jobs still running after $([int]$clock.Elapsed.TotalSeconds)s" + } + Write-TrialLine ("states: " + (($jobs | ForEach-Object { "$($_.State)" }) -join ',')) + + if (-not $timedOut) { + # PoshRSJob can flip State to Completed before the output is collected; Wait-RSJob settles it. + if ($Tool -eq 'RSJob') { $null = $jobs | Wait-RSJob -Timeout 10 } + $results =@(if ($Tool -eq 'ThreadJob') { Receive-Job -Job $jobs -ErrorAction Continue 2>&1 } else { $jobs | Receive-RSJob -ErrorAction Continue 2>&1 }) + $bad = 0; $okTotal = 0 + Write-TrialLine "received $($results.Count) outputs" + foreach ($r in $results) { + if ($r -is [System.Management.Automation.ErrorRecord]) { $bad++; Write-TrialLine "job error: $r"; continue } + if ($null -eq $r -or -not $r.PSObject.Properties['Ok']) { $bad++; Write-TrialLine "odd output: $r"; continue } + $okTotal += $r.Ok + if ($r.ErrorCount -gt 0) { $bad++; Write-TrialLine "iteration errors=$($r.ErrorCount): $($r.Errors)" } + if ($Case -like 'Probe*') { + Write-TrialLine ("probe: payloadType={0} payloadSeesMainVar={1} jobSeesMainVar={2} payloadThread={3} jobThread={4} mainThread={5}" -f + $r.PayloadType, $r.SeesMainVar, $r.JobSeesMainVar, $r.PayloadThread, $r.JobThread, $mainThread) + } + } + foreach ($j in $jobs) { + if ($j.State -eq 'Failed') { $bad++; Write-TrialLine "job failed: $(if ($j.PSObject.Properties['Error']) { $j.Error } else { $j.JobStateInfo.Reason })" } + if ($Tool -eq 'RSJob' -and $j.HasErrors) { $bad++; Write-TrialLine ("rsjob error: " + (($j.Error | ForEach-Object { "$_" }) -join ' | ')) } + } + $expectedOk = if ($Case -like 'Probe*') { 1 } else { $n * $Iterations } + if ($okTotal -ne $expectedOk) { $bad++; Write-TrialLine "ok iterations $okTotal of $expectedOk" } + } else { $bad = 1 } + + # The main runspace must still work after the jobs ran. + if (-not (Get-Command Test-MainAlive -ErrorAction SilentlyContinue)) { $damage.Add('Test-MainAlive missing at end') } + elseif ((Test-MainAlive) -ne 42) { $damage.Add('Test-MainAlive wrong at end') } + foreach ($d in $damage) { Write-TrialLine "main damage: $d" } + + if (-not $timedOut) { + if ($Tool -eq 'ThreadJob') { Remove-Job -Job $jobs -Force } else { $jobs | Remove-RSJob -Force } + } + if ($timedOut) { Write-TrialLine 'result: timeout'; [Environment]::Exit(4) } + if ($bad -gt 0 -or $damage.Count -gt 0) { Write-TrialLine "result: wrong (bad=$bad damage=$($damage.Count))"; [Environment]::Exit(2) } + Write-TrialLine 'result: ok' + [Environment]::Exit(0) +} catch { + $err = $_ + try { Write-TrialLine ("exception: " + $err.Exception.GetType().Name + ': ' + $err.Exception.Message + ' @ ' + $err.InvocationInfo.PositionMessage) } + catch { + # Write-TrialLine itself may be gone; log with .NET calls only, through a path kept outside session state. + $why = 'logger failed: ' + $_.Exception.GetType().Name + ': ' + $_.Exception.Message + ' || original: ' + $err.Exception.GetType().Name + ': ' + $err.Exception.Message + try { [System.IO.File]::AppendAllText([AppDomain]::CurrentDomain.GetData('pkProgressFile'), $why + "`r`n") } catch { } # nothing left to log with + } + [Environment]::Exit(2) +} diff --git a/Testing/third-party/matrix.ps1 b/Testing/third-party/matrix.ps1 new file mode 100644 index 0000000..a170760 --- /dev/null +++ b/Testing/third-party/matrix.ps1 @@ -0,0 +1,18 @@ +# Runs the whole stress matrix through run.ps1 and appends each summary line to -Log. +# powershell -NoProfile -File .\Testing\third-party\matrix.ps1 -Trials 20 +param([int]$Trials = 20, [string]$Log = (Join-Path $PSScriptRoot 'results\matrix.log')) +$runner = Join-Path $PSScriptRoot 'run.ps1' +foreach ($exe in 'powershell', 'pwsh') { + foreach ($tool in 'ThreadJob', 'RSJob') { + foreach ($main in 'Busy', 'Wait') { + $combos = @( + @('JobValue', 'Args'), @('JobValue', 'Using'), @('Delegate', 'Args'), @('Delegate', 'Using'), + @('TextJobValue', 'Args'), @('TextDelegate', 'Args')) + foreach ($c in $combos) { + $line = & powershell -NoProfile -File $runner -Exe $exe -Tool $tool -Case $c[0] -Via $c[1] -Main $main -Trials $Trials + Add-Content -Path $Log -Value $line + } + } + } +} +Add-Content -Path $Log -Value 'MATRIX DONE' diff --git a/Testing/third-party/results/matrix.log b/Testing/third-party/results/matrix.log new file mode 100644 index 0000000..c9007d0 --- /dev/null +++ b/Testing/third-party/results/matrix.log @@ -0,0 +1,49 @@ +powershell-ThreadJob-JobValue-Args-Busy: crash=15 ok=1 wrong=4 of 20 trials (7s). exit 0x00000001 +powershell-ThreadJob-JobValue-Using-Busy: crash=19 ok=1 of 20 trials (4s). exit 0x00000001 +powershell-ThreadJob-Delegate-Args-Busy: ok=18 wrong=2 of 20 trials (49s). iteration errors=1: wrong result '' || ok iterations 15999 of 16000 +powershell-ThreadJob-Delegate-Using-Busy: ok=16 wrong=4 of 20 trials (51s). iteration errors=1: wrong result '' || ok iterations 15999 of 16000 +powershell-ThreadJob-TextJobValue-Args-Busy: ok=20 of 20 trials (23s). +powershell-ThreadJob-TextDelegate-Args-Busy: ok=20 of 20 trials (30s). +powershell-ThreadJob-JobValue-Args-Wait: ok=20 of 20 trials (23s). +powershell-ThreadJob-JobValue-Using-Wait: ok=20 of 20 trials (22s). +powershell-ThreadJob-Delegate-Args-Wait: crash=3 timeout=17 of 20 trials (331s). timeout: 8 jobs still running after 45s +powershell-ThreadJob-Delegate-Using-Wait: crash=2 timeout=18 of 20 trials (332s). timeout: 8 jobs still running after 45s +powershell-ThreadJob-TextJobValue-Args-Wait: ok=20 of 20 trials (22s). +powershell-ThreadJob-TextDelegate-Args-Wait: ok=20 of 20 trials (28s). +powershell-RSJob-JobValue-Args-Busy: crash=17 wrong=3 of 20 trials (37s). exit 0x00000001 +powershell-RSJob-JobValue-Using-Busy: crash=14 wrong=6 of 20 trials (33s). exit 0x00000001 +powershell-RSJob-Delegate-Args-Busy: wrong=20 of 20 trials (26s). exception: CommandNotFoundException: The term 'Increment' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. @ At H:\_Projects\ParallelKit\Testing\third-party\inner.ps1:99 char:36 +powershell-RSJob-Delegate-Using-Busy: wrong=20 of 20 trials (27s). exception: CommandNotFoundException: The term 'Increment' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. @ At H:\_Projects\ParallelKit\Testing\third-party\inner.ps1:100 char:20 +powershell-RSJob-TextJobValue-Args-Busy: ok=20 of 20 trials (26s). +powershell-RSJob-TextDelegate-Args-Busy: ok=20 of 20 trials (31s). +powershell-RSJob-JobValue-Args-Wait: crash=19 wrong=1 of 20 trials (39s). exit 0x00000001 +powershell-RSJob-JobValue-Using-Wait: crash=17 wrong=3 of 20 trials (37s). +powershell-RSJob-Delegate-Args-Wait: wrong=20 of 20 trials (26s). exception: CommandNotFoundException: The term 'Increment' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. @ At H:\_Projects\ParallelKit\Testing\third-party\inner.ps1:99 char:36 +powershell-RSJob-Delegate-Using-Wait: wrong=20 of 20 trials (27s). exception: CommandNotFoundException: The term 'Increment' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. @ At H:\_Projects\ParallelKit\Testing\third-party\inner.ps1:100 char:20 +powershell-RSJob-TextJobValue-Args-Wait: ok=20 of 20 trials (26s). +powershell-RSJob-TextDelegate-Args-Wait: ok=20 of 20 trials (32s). +pwsh-ThreadJob-JobValue-Args-Busy: crash=20 of 20 trials (6s). exit 0xE0434352 +pwsh-ThreadJob-JobValue-Using-Busy: crash=20 of 20 trials (6s). exit 0xE0434352 +pwsh-ThreadJob-Delegate-Args-Busy: ok=14 wrong=6 of 20 trials (166s). iteration errors=1: wrong result '' || ok iterations 15999 of 16000 +pwsh-ThreadJob-Delegate-Using-Busy: ok=15 wrong=5 of 20 trials (164s). iteration errors=1: wrong result '' || ok iterations 15999 of 16000 +pwsh-ThreadJob-TextJobValue-Args-Busy: ok=20 of 20 trials (5s). +pwsh-ThreadJob-TextDelegate-Args-Busy: ok=20 of 20 trials (10s). +pwsh-ThreadJob-JobValue-Args-Wait: ok=20 of 20 trials (5s). +pwsh-ThreadJob-JobValue-Using-Wait: ok=20 of 20 trials (5s). +pwsh-ThreadJob-Delegate-Args-Wait: crash=1 hang=1 timeout=18 of 20 trials (318s). timeout: 8 jobs still running after 45s +pwsh-ThreadJob-Delegate-Using-Wait: hang=1 timeout=19 of 20 trials (318s). timeout: 8 jobs still running after 45s +pwsh-ThreadJob-TextJobValue-Args-Wait: ok=20 of 20 trials (5s). +pwsh-ThreadJob-TextDelegate-Args-Wait: ok=20 of 20 trials (10s). +pwsh-RSJob-JobValue-Args-Busy: crash=20 of 20 trials (9s). exit 0xE0434352 +pwsh-RSJob-JobValue-Using-Busy: crash=18 wrong=2 of 20 trials (12s). exception: RuntimeException: @ +pwsh-RSJob-Delegate-Args-Busy: wrong=20 of 20 trials (9s). exception: PSArgumentException: Cannot find type [RSRunspacePool]: verify that the assembly containing this type is loaded. @ At H:\_Projects\ParallelKit\Testing\third-party\modules\PoshRSJob\1.7.4.4\Public\Start-RSJob.ps1:525 char:30 +pwsh-RSJob-Delegate-Using-Busy: wrong=20 of 20 trials (10s). exception: PSArgumentException: Cannot find type [RSRunspacePool]: verify that the assembly containing this type is loaded. @ At H:\_Projects\ParallelKit\Testing\third-party\modules\PoshRSJob\1.7.4.4\Public\Start-RSJob.ps1:525 char:30 +pwsh-RSJob-TextJobValue-Args-Busy: ok=20 of 20 trials (7s). +pwsh-RSJob-TextDelegate-Args-Busy: ok=20 of 20 trials (13s). +pwsh-RSJob-JobValue-Args-Wait: crash=20 of 20 trials (9s). exit 0xE0434352 +pwsh-RSJob-JobValue-Using-Wait: crash=15 wrong=5 of 20 trials (11s). exit 0x00000001 +pwsh-RSJob-Delegate-Args-Wait: wrong=20 of 20 trials (9s). exception: PSArgumentException: Cannot find type [RSRunspacePool]: verify that the assembly containing this type is loaded. @ At H:\_Projects\ParallelKit\Testing\third-party\modules\PoshRSJob\1.7.4.4\Public\Start-RSJob.ps1:525 char:30 +pwsh-RSJob-Delegate-Using-Wait: wrong=20 of 20 trials (10s). exception: PSArgumentException: Cannot find type [RSRunspacePool]: verify that the assembly containing this type is loaded. @ At H:\_Projects\ParallelKit\Testing\third-party\modules\PoshRSJob\1.7.4.4\Public\Start-RSJob.ps1:525 char:30 +pwsh-RSJob-TextJobValue-Args-Wait: ok=20 of 20 trials (8s). +pwsh-RSJob-TextDelegate-Args-Wait: ok=20 of 20 trials (12s). +MATRIX DONE diff --git a/Testing/third-party/run.ps1 b/Testing/third-party/run.ps1 new file mode 100644 index 0000000..baab6bf --- /dev/null +++ b/Testing/third-party/run.ps1 @@ -0,0 +1,64 @@ +# Stress runner for the ThreadJob / PoshRSJob scriptblock question. Starts +# -Trials fresh processes (-Exe powershell or pwsh) running inner.ps1, +# -Concurrency at a time. A trial still running after -TimeoutSec is a hang +# and is killed. Modules load from .\modules (saved with Save-PSResource, not +# installed). +# powershell -NoProfile -File .\Testing\third-party\run.ps1 -Exe pwsh -Tool RSJob -Case Delegate -Main Wait -Trials 20 +param( + [Parameter(Mandatory)][ValidateSet('powershell', 'pwsh')][string]$Exe, + [Parameter(Mandatory)][string]$Tool, + [Parameter(Mandatory)][string]$Case, + [string]$Via = 'Args', + [string]$Main = 'Busy', + [int]$Trials = 20, + [int]$Workers = 8, + [int]$Iterations = 2000, + [int]$Concurrency = 3, + [int]$TimeoutSec = 60, + [string]$OutDir = '' +) +$ErrorActionPreference = 'Stop' +if ($OutDir -eq '') { $OutDir = Join-Path $PSScriptRoot 'results' } + +$inner = Join-Path $PSScriptRoot 'inner.ps1' +$modules = Join-Path $PSScriptRoot 'modules' +$label = "$Exe-$Tool-$Case-$Via-$Main" +$runDir = Join-Path $OutDir ($label + '-' + (Get-Date -Format 'yyyyMMdd-HHmmss')) +$null = New-Item -ItemType Directory -Path $runDir -Force + +$results = [System.Collections.Generic.List[object]]::new() +$running = [System.Collections.Generic.List[hashtable]]::new() +$next = 1 +$clock = [System.Diagnostics.Stopwatch]::StartNew() + +while ($next -le $Trials -or $running.Count -gt 0) { + while ($next -le $Trials -and $running.Count -lt $Concurrency) { + $progress = Join-Path $runDir ("trial-$next.txt") + $argLine = ('-NoProfile -File "{0}" -Tool {1} -Case {2} -Via {3} -Main {4} -Workers {5} -Iterations {6} -WaitTimeoutSec {7} -ModuleDir "{8}" -ProgressFile "{9}"' -f + $inner, $Tool, $Case, $Via, $Main, $Workers, $Iterations, ($TimeoutSec - 15), $modules, $progress) + $p = Start-Process -FilePath "$Exe.exe" -ArgumentList $argLine -WindowStyle Hidden -PassThru + $running.Add(@{ Trial = $next; Proc = $p; Watch = [System.Diagnostics.Stopwatch]::StartNew(); Progress = $progress }) + $next++ + } + Start-Sleep -Milliseconds 200 + foreach ($t in @($running)) { + $outcome = $null + if ($t.Proc.HasExited) { + $outcome = switch ($t.Proc.ExitCode) { 0 { 'ok' } 2 { 'wrong' } 4 { 'timeout' } default { 'crash' } } + } elseif ($t.Watch.Elapsed.TotalSeconds -gt $TimeoutSec) { + $outcome = 'hang' + try { $t.Proc.Kill() } catch { } # it may have exited between the check and the kill + } + if ($null -eq $outcome) { continue } + $lines = @(); if (Test-Path $t.Progress) { $lines = @(Get-Content $t.Progress) } + $notes = ($lines | Where-Object { $_ -match '^(probe|timeout|job |iteration|odd|ok iter|main damage|exception|rsjob)' } | Select-Object -First 4) -join ' || ' + if ($outcome -eq 'crash') { $notes = ('exit 0x{0:X8} ' -f $t.Proc.ExitCode) + $notes } + $results.Add([pscustomobject]@{ Trial = $t.Trial; Outcome = $outcome; Seconds = [math]::Round($t.Watch.Elapsed.TotalSeconds, 1); Notes = $notes }) + $null = $running.Remove($t) + } +} + +$results | Sort-Object Trial | Export-Csv -NoTypeInformation -Path (Join-Path $runDir 'results.csv') +$counts = ($results | Group-Object Outcome | Sort-Object Name | ForEach-Object { "$($_.Name)=$($_.Count)" }) -join ' ' +$firstNote = ($results | Where-Object { $_.Outcome -ne 'ok' -or $_.Notes -like 'probe*' } | Sort-Object Trial | Select-Object -First 1).Notes +"{0}: {1} of {2} trials ({3:N0}s). {4}" -f $label, $counts, $Trials, $clock.Elapsed.TotalSeconds, $firstNote