Skip to content

Commit c08507f

Browse files
committed
Admit required telemetry controls in the execution inventory
Register both capture controls as mandatory non-contributors and narrow the existing coverage selector without changing its cases. Rebind the saga source identity to the actual compiled census. Retain original saga and queue deadline assertions and include phase/shard/public problem details on failure. Update telemetry ownership and runner documentation. Verified: strict Debug diagnostic builds, canonical discovery for 3294 cases plus all five unchanged coverage groups, scoped saga format check, two restart/follow-up controls, one real three-silo queue deadline kill/rejoin/cold case, and repository governance. Earlier whole runtime shard retained one saga failure; final-source full Debug and Release qualification remain pending.
1 parent 00477d4 commit c08507f

5 files changed

Lines changed: 75 additions & 9 deletions

File tree

‎docs/Features/OrleansExecution/RegressionScheduling.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,15 @@ flowchart LR
2424
| REQ-EXECUTION-SCHEDULE-001 | Preserve ordinary maximum 50 and existing heavy and replicated-cluster test ownership. Use keyed `NotInParallel` for the default `RequestCqrsDataSource`'s mutable `PerTestSession` fixture. | AC-EXECUTION-SCHEDULE-001: `RequestCqrsFixtureScheduling.SharedSession` identifies the same collision domain in `RequestCqrsBoundaryTests`, `RequestCqrsFatalSettlementTests`, `RequestCqrsRoutingTests`, `MultiLaneReceiveWholeFlowTests` and `ConnectionProducerCleanupTests`. Existing methods and arguments remain unchanged; execute the original ordinary cohort. |
2525
| REQ-EXECUTION-SCHEDULE-002 | Permit independent ANN, text and connection fixtures to overlap unrelated work while retaining original cancellation and joined shutdown. Shared serialization uses its existing concurrent provider registry and per-call serializer sessions. | AC-EXECUTION-SCHEDULE-002: `RequestCqrsAnnMaintenanceTests`, `RequestCqrsTextMaintenanceTests`, `ConnectionAwaitedSettlementTests` and `RequestCqrsAnnEnrollmentAckTests` have no blanket exclusion. Their actual owned factory or explicit constructor creates their database/runtime fixture. Run these original operation cases alongside the ordinary cohort. |
2626
| REQ-EXECUTION-SCHEDULE-003 | Prove independent fixture overlap with observable original operation barriers, distinct ownership and substantive stored results. | AC-EXECUTION-SCHEDULE-003: `ConnectionFixtureConcurrencyTests.IndependentlyOwnedConnectionsReachBothOriginalBarriersAndSettle` starts two original `ConnectionScenario` fixtures. Both actual signed commands reach `RequestStarted` before either releases. Distinct directories, connection IDs, work owners and grain identities, persisted principal binding, producer disposal, both committed records and signed close are asserted. |
27-
| REQ-EXECUTION-SCHEDULE-004 | Preserve actual global-resource exclusion until that resource's owner supplies a qualified narrower contract. Do not infer isolation from fixture names. | AC-EXECUTION-SCHEDULE-004: `OrleansRuntimeTelemetryTests` retains its existing blanket exclusion because process-wide OpenTelemetry capture observes activity and meters through bounded exporters. Scheduling does not modify telemetry capture, runner settings or any heavy-test policy. |
27+
| REQ-EXECUTION-SCHEDULE-004 | Preserve actual global-resource exclusion until that resource's owner supplies a qualified narrower contract. Do not infer isolation from fixture names. | AC-EXECUTION-SCHEDULE-004: `OrleansRuntimeTelemetryTests` retains blanket exclusion for process-wide metric capture. Its bounded activity exporter selects traces registered before each owned operation starts, and privacy sentinels select the original caller parent. Both capture isolation and exhaustion controls remain mandatory ordinary tests. |
2828

2929
## Source map and evidence
3030

3131
The implementation paths are `tests/KeyLoad.FunctionalTests/OrleansExecution/{Cases,Fixtures,Helpers}/` and the existing `tests/KeyLoad.FunctionalTests/Messaging/Cases/MultiLane/MultiLaneReceiveWholeFlowTests.cs`. Changes comprise the shared-session key, guarded attribute changes and the genuine two-fixture whole operation case. Frontend, public SDK/MCP operations and server database execution are N/A: this work changes test scheduling and adds original-fixture evidence.
3232

33+
The named functional shard runner uses four concurrent test slots on local and CI
34+
runners. This resource bound is declared in `ci-test-shards.json`; the Aspire-owned
35+
TUnit entry retains its separately validated ordinary maximum of 50. Host startup
36+
and operation phases retain their own deadlines, including flows owning two hosts.
37+
3338
Qualification: the focused cases and the ordinary cohort must run with maximum 50 through the TUnit entry; the solution build, process recovery and Aspire replicated-cluster .NET SDK/official MCP gates remain mandatory and are not proven by this document.

‎docs/Features/OrleansExecution/RuntimeAdoption.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,11 +102,18 @@ principal, raw routing ID, private exception text or exemplar attribute escapes.
102102
Concurrent principals keep distinct request contexts. Immutable privacy sentinels
103103
in an actual operation span prove fail-closed suppression while the operation
104104
keeps its correct outcome. Capture is bounded and every provider is flushed and
105-
joined. `OrleansRuntimeTelemetryTests` on `RequestCqrsClusterFixture` own the
105+
joined. `OrleansRuntimeTelemetryTests` on `RequestCqrsHostedRuntimeFixture` own the
106106
cases; no synthetic exported span, raw-tag export, relaxed assertion, new
107107
provider or higher limit is permitted. SDK/MCP cluster qualification remains
108108
mandatory.
109109

110+
The activity capture registers each original operation trace before starting its
111+
parent and retains every exported span in that trace within the existing bound.
112+
The sentinel targets that caller's direct application span. Process-wide metrics
113+
retain the whole-operation test's `NotInParallel` ownership. Separate required
114+
capture controls prove unrelated trace isolation and visible capacity exhaustion;
115+
they are not whole-operation coverage contributors.
116+
110117
## Verification and rollout
111118

112119
After code changes: `dotnet build KeyLoad.slnx --no-restore -warnaserror`,

‎scripts/Features/CodeQuality/functional-coverage.unit-test-inventory.json‎

Lines changed: 43 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24886,9 +24886,9 @@
2488624886
"methodName": "ScheduleDoesNotPersistAmbientCallerTraceState",
2488724887
"instanceName": "ScheduleDoesNotPersistAmbientCallerTraceState",
2488824888
"sourcePath": "tests/KeyLoad.FunctionalTests/Messaging/Cases/RuntimeSaga/SagaTimeoutTraceIsolationTests.cs",
24889-
"sourceSha256": "3ec76f59f1e53428e0324fcd43515e7290c0fa6ff82d1a22b3b692f05efb92d7",
24889+
"sourceSha256": "b0b76b525c2c79d26d9a83845fc2fd61e08f5f814d1aacc1acec9ba28d09857a",
2489024890
"lineNumber": 16,
24891-
"endLineNumber": 33,
24891+
"endLineNumber": 46,
2489224892
"classification": "functional",
2489324893
"exclusionReason": null,
2489424894
"requirements": [
@@ -66259,12 +66259,52 @@
6625966259
],
6626066260
"nonContributorReason": "required-control",
6626166261
"reviewBasis": "Executes the actual governance CLI against owned positive and negative source cases; it controls test path ownership and does not execute a database operation."
66262+
},
66263+
{
66264+
"className": "KeyLoad.FunctionalTests.OrleansExecution.OrleansTelemetryCaptureIsolationTests",
66265+
"reportClassName": "KeyLoad.FunctionalTests.OrleansExecution.OrleansTelemetryCaptureIsolationTests",
66266+
"methodName": "AnOwnedTraceStillReportsExhaustedCaptureCapacity",
66267+
"instanceName": "AnOwnedTraceStillReportsExhaustedCaptureCapacity",
66268+
"sourcePath": "tests/KeyLoad.FunctionalTests/OrleansExecution/Cases/Orleans/OrleansTelemetryCaptureIsolationTests.cs",
66269+
"sourceSha256": "77e4e35c0925c875b094b32a10025eafc0edf12582c8bb6c6e0fd8a4d24bf85e",
66270+
"lineNumber": 30,
66271+
"endLineNumber": 42,
66272+
"classification": "required-non-contributor",
66273+
"exclusionReason": null,
66274+
"requirements": [
66275+
"REQ-ORL-012"
66276+
],
66277+
"acceptance": [
66278+
"AC-ORL-012"
66279+
],
66280+
"nonContributorReason": "required-control",
66281+
"reviewBasis": "Real OpenTelemetry provider/exporter capture isolation and exhaustion control. Required ordinary test; no database operation or distributed telemetry acceptance is claimed."
66282+
},
66283+
{
66284+
"className": "KeyLoad.FunctionalTests.OrleansExecution.OrleansTelemetryCaptureIsolationTests",
66285+
"reportClassName": "KeyLoad.FunctionalTests.OrleansExecution.OrleansTelemetryCaptureIsolationTests",
66286+
"methodName": "UnrelatedTracesLeaveCapacityForTheCompleteOwnedTrace",
66287+
"instanceName": "UnrelatedTracesLeaveCapacityForTheCompleteOwnedTrace",
66288+
"sourcePath": "tests/KeyLoad.FunctionalTests/OrleansExecution/Cases/Orleans/OrleansTelemetryCaptureIsolationTests.cs",
66289+
"sourceSha256": "77e4e35c0925c875b094b32a10025eafc0edf12582c8bb6c6e0fd8a4d24bf85e",
66290+
"lineNumber": 11,
66291+
"endLineNumber": 28,
66292+
"classification": "required-non-contributor",
66293+
"exclusionReason": null,
66294+
"requirements": [
66295+
"REQ-ORL-012"
66296+
],
66297+
"acceptance": [
66298+
"AC-ORL-012"
66299+
],
66300+
"nonContributorReason": "required-control",
66301+
"reviewBasis": "Real OpenTelemetry provider/exporter capture isolation and exhaustion control. Required ordinary test; no database operation or distributed telemetry acceptance is claimed."
6626266302
}
6626366303
],
6626466304
"coverageGroups": [
6626566305
{
6626666306
"groupId": "unit-functional-01",
66267-
"selector": "/KeyLoad.FunctionalTests/*/(AggregateReplayS*)|(AnalyticsComp*)|(AnnLifetimeD*)|(AnnPublicR*)|(AnnSeedC*)|(AnnSeedF*)|(AnnWaitA*)|(AtomicPartitionRosterB*)|(Bi*)|(Bm25OriginalP*)|(Bm25R*)|(CacheReadPermitC*)|(ChunkC*)|(ChunkManifestT*)|(ChunkW*)|(CliBackupJ*)|(CliBackupRestoreL*)|(Compe*)|(CompositionL*)|(CompositionP*)|(Conf*)|(ConnectionFi*)|(Cont*)|(CqrsT*)|(CredentialF*)|(CurrentA*)|(CurrentJ*)|(CurrentW*)|(DatabaseCompositionF*)|(Dec*)|(Disp*)|(DistributedSearchD*)|(DocumentE*)|(DocumentMutationImageT*)|(DropL*)|(DueF*)|(EventAppendC*)|(EventFeedPubSubA*)|(EventProjectionI*)|(EventQ*)|(FeedV*)|(Fie*)|(GlobalBranchWindowMergeB*)|(GraphCrossPartitionC*)|(GraphCrossPartitionRec*)|(GraphCrossPartitionS*)|(GraphIncomingE*)|(GraphIncomingV*)|(GraphSearchI*)|(GraphShortestPathC*)|(Ho*)|(HybridE*)|(Ini*)|(KeyCodecI*)|(KeyCodecM*)|(KeyLoadClientT*)|(KeyLoadQ*)|(LiveQ*)|(Man*)|(McpAuthenticationE*)|(McpB*)|(McpE*)|(McpF*)|(McpGatewayP*)|(McpP*)|(McpRes*)|(MetadataB*)|(Metam*)|(MultiLaneC*)|(OperationH*)|(OrleansT*)|(PackedAnnConc*)|(PackedAnnF*)|(PackedAnnOwner*)|(Partia*)|(PartitionQueryC*)|(PartitionQueryPublicC*)|(PartitionRecordC*)|(PartitionStoragePu*)|(ProjectionPr*)|(ProjectionR*)|(PublicN*)|(QueueA*)|(QueueEn*)|(QueueLifecycleC*)|(QueueLifecycleN*)|(QueueLifecycleP*)|(QueueO*)|(ReadB*)|(ReadCutCa*)|(ReadD*)|(ReadExecutionBudgetGrantC*)|(ReadOnlyCollectionV*)|(RelationalL*)|(RelationalQ*)|(RemoteTransferL*)|(RemoteTransferRec*)|(RemoteTransferRet*)|(RequestCqrsF*)|(ResourcePa*)|(ResourcePolicyUpdateV*)|(RowRevisionI*)|(RuntimeJournalL*)|(SagaS*)|(SagaTimeoutR*)|(SagaTimeoutTe*)|(SampleAggregateT*)|(SampleChunkCod*)|(SampleChunkPar*)|(SampleChunkStoreF*)|(SampleO*)|(SampleRetentionR*)|(SampleRollupO*)|(SampleRollupW*)|(SdkPr*)|(Secu*)|(SqlBetweenB*)|(SqlBo*)|(SqlCr*)|(SqlInnerJoinD*)|(SqlInnerJoinS*)|(SqlModelViewB*)|(StorageR*)|(StoreO*)|(StreamTe*)|(SubscriptionLeaseV*)|(TargetInboxA*)|(TargetInboxCap*)|(TextBi*)|(TextBm25E*)|(TextOnlineW*)|(TextProjectionA*)|(TextProjectionO*)|(TextProjectionRem*)|(TextProv*)|(TextWaitF*)|(TextWaitObserve*)|(TimeP*)|(TimeSeriesR*)|(TopicSqlR*)|(TransactionP*)|(Un*)|(ValueO*)|(VectorR*)|(ZoneTreeCoordinatedPointCacheE*)|(ZoneTreePointCacheP*)/(*)&(!AcChunk002UnpairedCodeUnitsUseExactUtf16Encoding*)&(!AcCq001_TypedStorageHelpersRejectNullArgumentsBeforeWork*)&(!EmptyAndUnknownVersionsRemainFormatUnsupported*)&(!LiteralMalformedTagScalarEscapeTextAndTimestampVectorsFailClosed*)&(!LiteralNoncanonicalAndUnrepresentableDecimalVectorsFailClosed*)&(!MultiLanePublicJsonDescriptorRetainsLiteralParentAndLeafPayloads*)&(!ValidUnicodeEscapesAndNumericExtremaRemainReadable*)",
66307+
"selector": "/KeyLoad.FunctionalTests/*/(AggregateReplayS*)|(AnalyticsComp*)|(AnnLifetimeD*)|(AnnPublicR*)|(AnnSeedC*)|(AnnSeedF*)|(AnnWaitA*)|(AtomicPartitionRosterB*)|(Bi*)|(Bm25OriginalP*)|(Bm25R*)|(CacheReadPermitC*)|(ChunkC*)|(ChunkManifestT*)|(ChunkW*)|(CliBackupJ*)|(CliBackupRestoreL*)|(Compe*)|(CompositionL*)|(CompositionP*)|(Conf*)|(ConnectionFi*)|(Cont*)|(CqrsT*)|(CredentialF*)|(CurrentA*)|(CurrentJ*)|(CurrentW*)|(DatabaseCompositionF*)|(Dec*)|(Disp*)|(DistributedSearchD*)|(DocumentE*)|(DocumentMutationImageT*)|(DropL*)|(DueF*)|(EventAppendC*)|(EventFeedPubSubA*)|(EventProjectionI*)|(EventQ*)|(FeedV*)|(Fie*)|(GlobalBranchWindowMergeB*)|(GraphCrossPartitionC*)|(GraphCrossPartitionRec*)|(GraphCrossPartitionS*)|(GraphIncomingE*)|(GraphIncomingV*)|(GraphSearchI*)|(GraphShortestPathC*)|(Ho*)|(HybridE*)|(Ini*)|(KeyCodecI*)|(KeyCodecM*)|(KeyLoadClientT*)|(KeyLoadQ*)|(LiveQ*)|(Man*)|(McpAuthenticationE*)|(McpB*)|(McpE*)|(McpF*)|(McpGatewayP*)|(McpP*)|(McpRes*)|(MetadataB*)|(Metam*)|(MultiLaneC*)|(OperationH*)|(OrleansTelemetryA*)|(PackedAnnConc*)|(PackedAnnF*)|(PackedAnnOwner*)|(Partia*)|(PartitionQueryC*)|(PartitionQueryPublicC*)|(PartitionRecordC*)|(PartitionStoragePu*)|(ProjectionPr*)|(ProjectionR*)|(PublicN*)|(QueueA*)|(QueueEn*)|(QueueLifecycleC*)|(QueueLifecycleN*)|(QueueLifecycleP*)|(QueueO*)|(ReadB*)|(ReadCutCa*)|(ReadD*)|(ReadExecutionBudgetGrantC*)|(ReadOnlyCollectionV*)|(RelationalL*)|(RelationalQ*)|(RemoteTransferL*)|(RemoteTransferRec*)|(RemoteTransferRet*)|(RequestCqrsF*)|(ResourcePa*)|(ResourcePolicyUpdateV*)|(RowRevisionI*)|(RuntimeJournalL*)|(SagaS*)|(SagaTimeoutR*)|(SagaTimeoutTe*)|(SampleAggregateT*)|(SampleChunkCod*)|(SampleChunkPar*)|(SampleChunkStoreF*)|(SampleO*)|(SampleRetentionR*)|(SampleRollupO*)|(SampleRollupW*)|(SdkPr*)|(Secu*)|(SqlBetweenB*)|(SqlBo*)|(SqlCr*)|(SqlInnerJoinD*)|(SqlInnerJoinS*)|(SqlModelViewB*)|(StorageR*)|(StoreO*)|(StreamTe*)|(SubscriptionLeaseV*)|(TargetInboxA*)|(TargetInboxCap*)|(TextBi*)|(TextBm25E*)|(TextOnlineW*)|(TextProjectionA*)|(TextProjectionO*)|(TextProjectionRem*)|(TextProv*)|(TextWaitF*)|(TextWaitObserve*)|(TimeP*)|(TimeSeriesR*)|(TopicSqlR*)|(TransactionP*)|(Un*)|(ValueO*)|(VectorR*)|(ZoneTreeCoordinatedPointCacheE*)|(ZoneTreePointCacheP*)/(*)&(!AcChunk002UnpairedCodeUnitsUseExactUtf16Encoding*)&(!AcCq001_TypedStorageHelpersRejectNullArgumentsBeforeWork*)&(!EmptyAndUnknownVersionsRemainFormatUnsupported*)&(!LiteralMalformedTagScalarEscapeTextAndTimestampVectorsFailClosed*)&(!LiteralNoncanonicalAndUnrepresentableDecimalVectorsFailClosed*)&(!MultiLanePublicJsonDescriptorRetainsLiteralParentAndLeafPayloads*)&(!ValidUnicodeEscapesAndNumericExtremaRemainReadable*)",
6626866308
"classNames": [
6626966309
"KeyLoad.FunctionalTests.AnalyticsStore.AnalyticsCompositionTests",
6627066310
"KeyLoad.FunctionalTests.Authorization.CredentialFrameBudgetTests",

‎tests/KeyLoad.ClusterTests/Messaging/Scenarios/QueueScheduling/Deadlines/QueueDeadline/QueueDeadlineSiloClusterActions.cs‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
using KeyLoad.ClusterTests.ClientApi;
22
using KeyLoad.ClusterTests.OrleansExecution;
3-
using KeyLoad.ClusterTests.QueryExecution;
43

54
namespace KeyLoad.ClusterTests.Messaging;
65

@@ -41,8 +40,10 @@ [new AdvanceQueueDeadline(state.Lane.Queue, QueueDeadlineSiloClusterProtocol.Mes
4140
var result = await admin.Sdk.CommitAsync(command, token);
4241
await Assert.That(result.IsSuccess).IsFalse();
4342
await Assert.That(result.Value).IsNull();
44-
await SqlClusterProtocol.EqualAsync(Errors.Problem(ErrorCode.RevisionConflict,
45-
QueueDeadlineSiloClusterProtocol.StaleDetail), result.Problem);
43+
var expected = Errors.Problem(ErrorCode.RevisionConflict, QueueDeadlineSiloClusterProtocol.StaleDetail);
44+
var actual = JsonDefaults.Serialize(result.Problem);
45+
await Assert.That(actual.AsSpan().SequenceEqual(JsonDefaults.Serialize(expected))).IsTrue()
46+
.Because(System.Text.Encoding.UTF8.GetString(actual));
4647
await QueueDeadlineSiloClusterCurrent.ObserveAsync(admin, state, token);
4748
var reply = await admin.Mcp.CallAsync(McpCallerTools.DocumentsCommit, command, token);
4849
_ = (await McpCallerAssertions.ErrorAsync(reply, ErrorCode.RevisionConflict, dispatched: true)).Value;

‎tests/KeyLoad.FunctionalTests/Messaging/Cases/RuntimeSaga/SagaTimeoutTraceIsolationTests.cs‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,20 @@ public async Task ScheduleDoesNotPersistAmbientCallerTraceState()
2727

2828
await AssertIsolatedTraceAsync(scheduled, caller.TraceId.ToString(), saga.Hint);
2929
var shards = await fixture.JobHarness.CaptureOwnedShardsAsync(dueTime, CancellationToken.None);
30-
await fixture.JobHarness.WaitUntilSettledAsync(shards, CancellationToken.None);
30+
try
31+
{
32+
await fixture.JobHarness.WaitUntilSettledAsync(shards, CancellationToken.None);
33+
}
34+
catch (TimeoutException failure)
35+
{
36+
var current = fixture.Database.Database.InspectSaga(SagaPrincipalId, saga.Lane, saga.Id);
37+
var counts = new List<string>();
38+
foreach (var shard in shards)
39+
{
40+
counts.Add($"{shard.Id}: {await shard.GetJobCountAsync()} jobs, scheduled={shard.Id == scheduled.ShardId}");
41+
}
42+
throw new TimeoutException($"{failure.Message} Saga phase: {current?.Phase}; {string.Join("; ", counts)}", failure);
43+
}
3144
var actual = fixture.Database.Database.InspectSaga(SagaPrincipalId, saga.Lane, saga.Id);
3245
await Assert.That(actual!.Phase).IsEqualTo(SagaPhase.TimedOut);
3346
}

0 commit comments

Comments
 (0)