Run ten functional tests concurrently and balance cluster jobs

Author: KSemenenko

docs/ADR/ADR-148-parallel-ci-shards.md

@@ -15,10 +15,10 @@ Build and Tests ran every suite one after another in one job on a 4-vCPU ubuntu-
 Build and Tests is a set of parallel jobs driven by one checked-in manifest, `scripts/Features/TestInfrastructure/ci-test-shards.json`, and its runner `ci-test-shards.mjs`:
 
 - **Build:** governance and website inventory checks, the static shard partition check and `dotnet build KeyLoad.slnx --configuration Release -warnaserror`.
-- **Units:** four shards. Each runs one process per namespace with the assembly-pinned filter `/KeyLoad.FunctionalTests/<namespace>/*/*` at processor-count parallelism.
+- **Functional operations:** four descriptive shards. Each runs one process per namespace with the assembly-pinned filter `/KeyLoad.FunctionalTests/<namespace>/*/*` and ten concurrent tests, both locally and in CI.
 - **Analyzers and process recovery:** one job with filters `/KeyLoad.AnalyzerTests/*/*/*` and `/KeyLoad.ProcessRecoveryTests/*/*/*`.
-- **Aspire topology:** twelve shards of two lanes each.
-  - A lane is its own `dotnet test` process with `--maximum-parallel-tests 1` and a 45-minute deadline, so a runner holds at most two clusters. Measured lanes take 20–25 minutes, so the deadline sits well above a complete lane.
+- **Aspire topology:** twenty descriptive shards containing thirty-two lanes, with one or two lanes per runner.
+  - A lane is its own `dotnet test` process with `--maximum-parallel-tests 1` and a 45-minute deadline, so a runner holds at most two clusters.
   - Lanes are typed: one keyed shared cluster, own clusters, other keyed clusters, or one long functional gate method.
   - Every lane process owns its own keyed shared cluster, so a test that kills, restarts or cold-reopens nodes cannot affect another lane.
 - **Verify test partition:** reads every shard's TRX artifact and proves that each test ran exactly once, in its assigned shard and lane.
@@ -37,8 +37,8 @@ Partition correctness is checked twice:
 
 ## Consequences
 
-- Wall time is bounded by the slowest shard instead of the sum of all suites. The estimate is about 25–28 minutes, assuming two lanes per runner run about 1.3 times slower than alone. Lane estimates come from run 40091311302 and need rebalancing from the first sharded reports.
-- The workflow uses 19 test-related jobs at once (Build, the planning job, four unit shards, one analyzers-and-recovery job, twelve topology shards) plus Verify test partition at the end.
+- Original Linux reports from run 40510797452 identify serial lane workloads of up to 41 minutes, including approximately 27 minutes in the four composite-prefix cases. Dedicated jobs separate composite-prefix, subscription cold-cut, revision-plan, graph-frontier, SQL/JSON/CSharp equivalence, ordered-queue and file-ownership operations. Those original case timings inform assignment; they exclude build, queue, shared fixture startup and teardown. Speedup remains unqualified until the changed partition completes on Linux.
+- The matrix permits twenty cluster jobs with thirty-two concurrent serial lanes, plus four functional jobs with ten concurrent tests each and one analyzers-and-process-recovery job. Build and planning are separate jobs; the final partition check waits for test results. Actual concurrency also depends on available GitHub runner capacity.
 - Adding a test class is a one-line manifest change. `inventory <suite>` shows the cluster each class uses and whether it disrupts nodes. A class missing from the manifest fails Build.
 - Product behaviour and test assertions are unchanged. No product request or admission limit is added; the bound is on concurrent test processes per runner.
 - Static source scanning is not runtime proof. Discovery, the TRX identities and the topology lanes are qualified only by actual Linux GitHub runs.

docs/Features/TestInfrastructure.md

@@ -152,12 +152,14 @@ Whole regression: existing ImagesProduceClosedManifestAndRejectTamperingWithoutR
 | Job | Contents | Filter and concurrency |
 |---|---|---|
 | Build | governance, website source inventory, static shard partition check, `dotnet build KeyLoad.slnx -warnaserror` | none |
-| Named functional operation groups | the namespaces listed for the shard | one process per namespace selected by its descriptive group ID, `/KeyLoad.FunctionalTests/<namespace>/*/*`, `--maximum-parallel-tests 4` on local and CI runners |
-| Analyzers and process recovery | shards AN then RC | `/KeyLoad.AnalyzerTests/*/*/*` and `/KeyLoad.ProcessRecoveryTests/*/*/*` |
-| Named cluster capability groups | two lanes per runner | each lane is its own `dotnet test` process, `--maximum-parallel-tests 1 --timeout 45m`, class or method alternation under `/KeyLoad.ClusterTests/*/` |
+| Named functional operation groups | the namespaces listed for the shard | one process per namespace selected by its descriptive group ID, `/KeyLoad.FunctionalTests/<namespace>/*/*`, `--maximum-parallel-tests 10` on local and CI runners |
+| Analyzers and process recovery | compiler analyzers, then process crash recovery | `/KeyLoad.AnalyzerTests/*/*/*` and `/KeyLoad.ProcessRecoveryTests/*/*/*` |
+| Named cluster capability groups | one or two lanes per runner | each lane is its own `dotnet test` process, `--maximum-parallel-tests 1 --timeout 45m`, class or method alternation under `/KeyLoad.ClusterTests/*/` |
 | Verify test partition | every shard's TRX artifact | `ci-test-shards.mjs verify TestResults` |
 
-Every test job runs `dotnet build KeyLoad.slnx -warnaserror` in the default Debug configuration, because tests read compiled outputs of projects their test project does not reference: ProductionSourceManifestOperationTests reads the image of every solution project, and ColdBootstrapSdkTests launches `src/KeyLoad.Cli/bin/Debug/net10.0/KeyLoad.Cli.dll`. Each job then writes TRX reports to `TestResults/<suite>/<shard>/<lane or namespace>` and uploads them as `test-results-<suite>-<shard>` even when tests fail. The filters pin the assembly segment, so the process recovery tests that the unit host also registers run once, in RC.
+Every test job runs `dotnet build KeyLoad.slnx --configuration Release -warnaserror`, because tests read compiled outputs of projects their test project does not reference: ProductionSourceManifestOperationTests reads the image of every solution project, and ColdBootstrapSdkTests launches the CLI from the selected configuration. Local development uses the SDK default Debug configuration. Each job writes original TRX reports below its test assembly's `bin/Release/net10.0/ci-results/<suite>/<shard>/<lane or namespace>` and uploads them even when tests fail. The filters pin the assembly segment, so process recovery tests registered by the functional host run only in the process crash recovery shard.
+
+The manifest separates the long composite-prefix, subscription cold-cut, revision-plan, graph-frontier, SQL/JSON/CSharp equivalence, ordered-queue and file-ownership operations into dedicated jobs. Twenty cluster jobs contain thirty-two lanes in total. Original Linux reports identified these long serial sets; the new wall time requires an actual run of this partition. Additional parallelism comes from separate runners while each runner retains its two-cluster bound.
 
 A lane runs one test at a time, so it holds at most one 3-node cluster at a time and a 4-vCPU runner holds two. Lane kinds keep that bound: `shared-readers` and `shared-exclusive` lanes use one keyed shared cluster and create no clusters of their own; `own-clusters` and `functional-gate` lanes use no shared key; `isolated-clusters` lanes hold the other keyed clusters and tests without a cluster. Each lane process has its own keyed shared cluster, so a test that kills or restarts nodes affects only its own lane. Playwright is installed only on the shards that the manifest marks `playwright`.
 

scripts/Features/TestInfrastructure/ci-test-shards.json

@@ -19,7 +19,7 @@
       "sources": "tests/KeyLoad.FunctionalTests",
       "assembly": "KeyLoad.FunctionalTests",
       "partition": "namespace",
-      "parallelism": 4,
+      "parallelism": 10,
       "timeoutMinutes": 20,
       "shards": [
         {
@@ -87,6 +87,98 @@
         "HeavyLoad"
       ],
       "shards": [
+        {
+          "id": "cluster-composite-prefix-cold-recovery",
+          "name": "Composite prefixes and cold recovery",
+          "lanes": [
+            {
+              "id": "composite-prefix-public-adapters",
+              "kind": "shared-exclusive",
+              "classes": [
+                "SparseCompositePrefixClusterTests"
+              ]
+            }
+          ]
+        },
+        {
+          "id": "cluster-subscription-policy-and-cold-cuts",
+          "name": "Subscription policy and cold cuts",
+          "lanes": [
+            {
+              "id": "subscription-policy-and-cold-cuts",
+              "kind": "own-clusters",
+              "classes": [
+                "SubscriptionPhasePublicColdTests"
+              ]
+            }
+          ]
+        },
+        {
+          "id": "cluster-query-revision-and-index-plans",
+          "name": "Query revisions and index plans",
+          "lanes": [
+            {
+              "id": "metadata-revision-and-index-plans",
+              "kind": "own-clusters",
+              "classes": [
+                "QueryRuleOptimizationClusterTests"
+              ]
+            }
+          ]
+        },
+        {
+          "id": "cluster-graph-frontiers-and-vertex-policy",
+          "name": "Graph frontiers and vertex policy",
+          "lanes": [
+            {
+              "id": "graph-frontiers-and-vertex-policy",
+              "kind": "shared-exclusive",
+              "classes": [
+                "GraphLabelFrontierPublicTests",
+                "GraphVertexPublicWholeFlowTests"
+              ]
+            }
+          ]
+        },
+        {
+          "id": "cluster-sql-json-and-csharp-equivalence",
+          "name": "SQL JSON and CSharp equivalence",
+          "lanes": [
+            {
+              "id": "sql-json-and-csharp-equivalence",
+              "kind": "own-clusters",
+              "classes": [
+                "MetamorphicPublicClusterTests"
+              ]
+            }
+          ]
+        },
+        {
+          "id": "cluster-ordered-queue-retry-and-cold-cuts",
+          "name": "Ordered queue retry and cold cuts",
+          "lanes": [
+            {
+              "id": "ordered-queue-retry-and-cold-cuts",
+              "kind": "shared-exclusive",
+              "classes": [
+                "QueueOrderedRetryPublicClusterTests"
+              ]
+            }
+          ]
+        },
+        {
+          "id": "cluster-file-ownership-and-cold-recovery",
+          "name": "File ownership and cold recovery",
+          "lanes": [
+            {
+              "id": "file-ownership-and-cold-recovery",
+              "kind": "own-clusters",
+              "classes": [
+                "FunctionalFileOwnershipClusterTests"
+              ]
+            }
+          ]
+        },
         {
           "id": "cluster-public-apis-and-administration",
           "playwright": true,
@@ -227,7 +319,6 @@
               "classes": [
                 "AtomicTriadStagedClusterTests",
                 "AuthenticationExecutionTests",
-                "QueryRuleOptimizationClusterTests",
                 "QueueDeadlineSiloClusterTests",
                 "StrongReadSessionAbsenceClusterTests"
               ]
@@ -286,7 +377,6 @@
               "kind": "own-clusters",
               "classes": [
                 "GraphLabelUseClusterTests",
-                "MetamorphicPublicClusterTests",
                 "QueueWorkerProcessClusterTests",
                 "RequiredProcessingPolicyClusterTests",
                 "SqlParserAdmissionClusterWholeFlowTests"
@@ -300,8 +390,6 @@
                 "AggregateReplayClusterSiloRestartTests",
                 "EventCurrentReadAuthorityClusterTests",
                 "EventProjectionClusterTests",
-                "GraphLabelFrontierPublicTests",
-                "GraphVertexPublicWholeFlowTests",
                 "GuideLifecyclePublicTests",
                 "PartitionQueryPublicSiloRestartTests",
                 "RecurringDueClusterTests",
@@ -328,8 +416,7 @@
               "classes": [
                 "ModelHeadAuthorityClusterTests",
                 "RemoteTransferRepairClusterTests",
-                "RequestBodyLimitClusterTests",
-                "SubscriptionPhasePublicColdTests"
+                "RequestBodyLimitClusterTests"
               ]
             },
             {
@@ -345,7 +432,6 @@
                 "LeftFilteredJoinClusterTests",
                 "QueueDeadLetterPublicClusterTests",
                 "RecurringScheduleClusterTests",
-                "SparseCompositePrefixClusterTests",
                 "SqlModelViewClusterParityTests",
                 "TimeSeriesCurrentCapabilityClusterTests"
               ]
@@ -369,7 +455,6 @@
               "kind": "own-clusters",
               "classes": [
                 "CompetingInboxPublicTests",
-                "FunctionalFileOwnershipClusterTests",
                 "RemoteTransferAttemptClusterTests",
                 "ResourcePolicyUpdateClusterTests"
               ]
@@ -431,7 +516,6 @@
                 "DeclarativeCompositionClusterTests",
                 "DocumentSessionReadClusterTests",
                 "GraphPathClusterSiloRestartTests",
-                "QueueOrderedRetryPublicClusterTests",
                 "SqlGraphSearchClusterTests",
                 "StreamCatchupColdClusterTests",
                 "TimeSeriesAggregateScanBoundClusterTests"