Skip to content

Commit 85932f8

Browse files
committed
Collect hosted benchmark recordings with explicit resource policy
Hosted jobs can use different CPU models even when every database session passes. Explicitly use the existing recorded resource policy for collection while preserving the native CLI comparison default and all original execution, accounting, hash and provenance checks. Require all 32 workers and include resource-policy.json in the combined artifact; do not claim matched-hardware comparative qualification. Verified: 10 production Node contract probes with 161 rejected corrupt inputs, seven invalid CLI selections, actionlint, Node syntax, governance, source closure, shard partition, restore, format and format verification. Full Debug solution build passed with warnings as errors and one build worker (0 warnings/errors); the preceding parallel MSBuild lost child nodes. Local Website TUnit required the authenticated source/coverage packet at setup; full GitHub qualification remains pending for this checkpoint.
1 parent ea36682 commit 85932f8

3 files changed

Lines changed: 21 additions & 8 deletions

File tree

‎.github/workflows/benchmarks.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -370,7 +370,9 @@ jobs:
370370
mkdir -p artifacts/comparisons/sessions
371371
node scripts/Features/BenchmarkComparisons/session-github-collect.mjs --input="$PWD/artifacts/comparisons/sessions/provider"
372372
- name: Check complete session accounting
373-
run: node scripts/Features/BenchmarkComparisons/session-aggregate-cli.mjs --input="$PWD/artifacts/comparisons/sessions/provider" --output="$PWD/artifacts/comparisons/sessions/aggregate"
373+
run: |
374+
node scripts/Features/BenchmarkComparisons/session-aggregate-cli.mjs --input="$PWD/artifacts/comparisons/sessions/provider" --output="$PWD/artifacts/comparisons/sessions/aggregate" --resource-policy=recorded
375+
printf '%s\n' 'All operation, correctness and provenance checks passed. These recorded timings do not qualify matched-hardware performance comparisons across GitHub-hosted runners. The native CLI comparison policy remains strict.' >> "$GITHUB_STEP_SUMMARY"
374376
- name: Save combined benchmark results
375377
uses: actions/upload-artifact@v7
376378
with:

‎docs/Features/BenchmarkComparisons.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -420,7 +420,7 @@ complete original TRX, qualification, runtime/resource identities, hashes and
420420
correctness evidence. Incomplete server resource telemetry does not hide recorded
421421
operation measurements. Preserve resourceQualified=false and label these rows
422422
Measured · incomplete telemetry. Missing resource observations remain an em dash;
423-
server resource comparability remains unverified. Website records original operation timings independently of hosted-runner hardware equality; the native aggregation CLI retains its strict comparison policy. Selected corrupt, expired, foreign or mixed evidence is
423+
server resource comparability remains unverified. Website records original operation timings independently of hosted-runner hardware equality. The hosted Benchmarks workflow explicitly collects with `--resource-policy=recorded`, retaining every execution, correctness, accounting and provenance check without claiming matched hardware. The native aggregation CLI defaults to `comparison`; that policy still rejects unequal observed hardware and resource profiles. Its aggregate artifact includes `resource-policy.json` so the selected policy stays explicit outside the workflow log. A green collection run does not qualify cross-hardware performance comparisons. Selected corrupt, expired, foreign or mixed evidence is
424424
rejected. Workflows and triggers are defined in ADR-062, ADR-064 and ADR-112:
425425
Website intake reads original successful worker/runtime artifacts directly, even
426426
when the overall run or combined-results job failed. With no eligible sessions it

‎scripts/Features/BenchmarkComparisons/session-aggregate-cli.mjs‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,27 @@
11
import path from 'node:path';
22
import { aggregateSessions } from './session-aggregate.mjs';
33
import { SESSION, requireSession } from './session-contract.mjs';
4+
import { writeJson } from './isolated-github-files.mjs';
45
try {
56
const args = new Map();
67
for (const argument of process.argv.slice(2)) {
7-
const match = /^--(input|output)=(.+)$/.exec(argument);
8-
requireSession(match !== null && !args.has(match[1]) && path.isAbsolute(match[2]));
9-
args.set(match[1], path.resolve(match[2]));
8+
const match = /^--(input|output|resource-policy)=(.+)$/.exec(argument);
9+
requireSession(match !== null && !args.has(match[1]));
10+
if (match[1] === 'resource-policy') {
11+
requireSession(['comparison', 'recorded'].includes(match[2]));
12+
args.set(match[1], match[2]);
13+
} else {
14+
requireSession(path.isAbsolute(match[2]));
15+
args.set(match[1], path.resolve(match[2]));
16+
}
1017
}
11-
requireSession(args.size === 2);
12-
const result = await aggregateSessions({ ...Object.fromEntries(args), resourcePolicy: 'comparison' });
13-
process.stdout.write(JSON.stringify({ schemaVersion: result.schemaVersion, workers: result.workers.length }) + '\n');
18+
requireSession(args.has('input') && args.has('output'));
19+
const resourcePolicy = args.get('resource-policy') ?? 'comparison';
20+
const result = await aggregateSessions({ input: args.get('input'), output: args.get('output'), resourcePolicy });
21+
requireSession(result.workers.length === SESSION.workers);
22+
await writeJson(path.join(args.get('output'), 'resource-policy.json'), { resourcePolicy });
23+
process.stdout.write(JSON.stringify({ schemaVersion: result.schemaVersion, workers: result.workers.length,
24+
resourcePolicy }) + '\n');
1425
} catch {
1526
process.stderr.write(SESSION.failure + '\n');
1627
process.exitCode = 1;

0 commit comments

Comments
 (0)