From 9cc7ff5104c8e85e5eb9605df5e8fd27de2571db Mon Sep 17 00:00:00 2001 From: "liqiankun.1111" Date: Fri, 18 Sep 2026 10:40:23 +0800 Subject: [PATCH] docs(quality): unify skill concepts and performance interpretation Quality skills used conflicting Environment, Runner, and Workload meanings and lacked the performance evidence rules needed to distinguish configured rate from sustained capacity. Define shared identity, execution, evidence, and judgment semantics in CONCEPTS.md, then have E2E, perf, and trajectory explicitly consume them while retaining domain workflows. Separate environment identity from execution location and connection conditions. Align perf guidance with request-rate/inflight limits, realized phase windows, independent request judgments, and comparable capacity curves; align trajectory guidance with Detector/Verifier roles. Bump both quality plugin manifests to 0.0.10. Validation: all three skills pass skill_lint; 20 local Markdown links and reference reachability pass; Claude/Codex manifest versions agree; git diff --check passes. Documentation and manifest changes only; no live load test was run. --- quality/.claude-plugin/plugin.json | 2 +- quality/.codex-plugin/plugin.json | 2 +- quality/CONCEPTS.md | 121 ++++++++++ quality/README.md | 65 ++---- quality/references/environment.md | 178 ++++++-------- quality/skills/e2e/SKILL.md | 63 ++--- .../e2e/references/scenario-environment.md | 4 +- quality/skills/perf/SKILL.md | 220 ++++++++---------- .../perf/references/load-and-evidence.md | 114 +++++++++ .../perf/references/load-environment.md | 72 +++--- quality/skills/trajectory/SKILL.md | 62 +++-- .../references/problem-discovery.md | 6 +- 12 files changed, 519 insertions(+), 390 deletions(-) create mode 100644 quality/CONCEPTS.md create mode 100644 quality/skills/perf/references/load-and-evidence.md diff --git a/quality/.claude-plugin/plugin.json b/quality/.claude-plugin/plugin.json index d58c75d..2f399c3 100644 --- a/quality/.claude-plugin/plugin.json +++ b/quality/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "quality", - "version": "0.0.9", + "version": "0.0.10", "description": "Evidence-driven skills that operate project-owned E2E, performance, and agent trajectory quality capabilities.", "author": { "name": "compforge", diff --git a/quality/.codex-plugin/plugin.json b/quality/.codex-plugin/plugin.json index 27a37d0..55587a4 100644 --- a/quality/.codex-plugin/plugin.json +++ b/quality/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "quality", - "version": "0.0.9", + "version": "0.0.10", "description": "Evidence-driven skills that operate project-owned E2E, performance, and agent trajectory quality capabilities.", "author": { "name": "compforge", diff --git a/quality/CONCEPTS.md b/quality/CONCEPTS.md new file mode 100644 index 0000000..644e825 --- /dev/null +++ b/quality/CONCEPTS.md @@ -0,0 +1,121 @@ +# Quality concepts + +E2E, performance, and trajectory skills use these semantics to identify the subject, operate a +project capability, and interpret its evidence. Read this document before selecting a capability +or interpreting a run. Domain workflows and preparation live in the skills and their references. + +## Ownership + +- The framework provides execution, collection, evaluation, reduction, and reporting mechanisms. +- The project owns Cases, adapters, profiles, recordings, annotations, judgment criteria, target + policy, and the canonical entrypoint. Use that entrypoint when one exists. +- The skill discovers and operates the capability, explains evidence, and improves project-owned + assets within the user's request. Framework presence alone does not establish usable coverage. + +These concepts align with the [quality-harness kernel](https://github.com/compforge/quality-harness/blob/main/docs/kernel.md). +Map native project artifacts to their semantic roles; shared classes, field names, and file formats +are not prerequisites. Check the project's actual SDK and artifact contracts before invoking an API +or assuming that a model has been persisted. + +## Subject and runtime identity + +| Concept | Meaning and evidence to resolve | +|---|---| +| Repository | Source identity within a Forge, such as GitHub or GitLab; record its revision separately. | +| Component | A unit in a Repository that can be independently built or released. | +| Environment | A named deployment and runtime environment; identify its scope and configuration. | +| Host | An optional environment access host. A cluster access host does not prove where an application process ran. | +| Service | A Component's named runtime presence in an Environment; resolve the deployed revision independently. | +| Workload | A named platform carrier declared by a Service; discover its actual instances and state at execution time. | +| Operation | A named capability exposed by a Service; protocol details belong to its adapter and access configuration. | + +A logical Service may use several Workloads. Neither a declared Workload nor a configured endpoint +proves readiness, and a logical Service is not necessarily a Kubernetes Service resource. Bind +evidence to the relevant service, workload instances, and time instead of inferring identity from +an address or display label. + +The **target** is the subject selected for the task. For live testing, identify its Service, +Environment, revision, and scope; for offline analysis, identify the recorded source and cohort. +Record the quality command's execution location and connection path as run conditions. Moving the +test process or changing its connection need not change Environment identity, but changes the +conditions under which results can be compared. See [environment preparation](references/environment.md). + +## Intent and execution facts + +A **Case** is reusable test intent with stable identity. Canonical Case assets belong to spec-case +when the project uses it; profiles select Cases and experiment-local weights without rewriting +their inputs or expected behavior. An **Experiment** names reproducible verification intent. + +```text +ExperimentRun + → Execution + → OperationRun + → Outcome + +Reducer(recorded run facts) → Artifact → Report / Verdict +``` + +An ExperimentRun is one actual execution. Each domain owns its grouping and lifecycle: E2E can use +CaseRun, while perf uses ArmRun for one resource/load configuration. An OperationRun records one +real service invocation and owns its raw Outcome. A multistep Case may produce several calls; a +perf Case can be selected repeatedly. Keep configuration identity, execution identity, and call +identity distinct when joining evidence. + +A **Reducer** derives **Artifacts** from recorded facts without calling the subject again. A +**Report** renders those artifacts. Inspect native run IDs, call IDs, manifests, and raw records; +do not reconstruct identity from report labels. Analysis of existing traces or trajectories does +not require inventing an Experiment, Case, or service call. + +## Evidence and evaluation + +| Concept | Semantic role | +|---|---| +| Observation | What execution or collection actually observed, with source identity and provenance. Outcome and recorded trajectory data are examples. | +| Unit | The domain's addressable evaluation grain, such as a request, window, Case, or trajectory. | +| Annotation | Supervision already available for evaluation, such as a human label or reference, with producer and provenance. | +| Dataset | Reusable, versioned Unit facts and their Case, Observation, and Annotation relationships. | +| EvaluationRun | One evaluation of those facts under selected components, versions, configuration, and policy. | +| Finding | An evidenced pattern or anomaly; it may motivate investigation but is not a quality verdict or proven cause. | +| Measurement | A factual value derived from evidence, such as latency, tokens, or resource use. | +| Evaluation | A judgment against an explicit criterion, produced by a Judge, Verifier, assertion, or other native evaluator. | +| Worksheet | The evaluation's rows of Unit facts and results, including missing or errored cells. This is a semantic view, not a required file format. | +| Verdict | The run-level, machine-consumable decision under the declared policy. | + +Preserve the source facts when judgment fails. Missing evaluation is not an observed product +failure. Read execution health and result coverage before interpreting scores or aggregates; a +Finding or Measurement affects a gate only through an explicit judgment or policy. + +Keep one grain and stable key per analysis table. Request, window, and run aggregates answer +different questions. Use native artifacts when available rather than inventing Dataset, +EvaluationRun, or Worksheet files that the project does not produce. + +## Reuse and comparison + +Record subject or generator revision, evidence identity/version, and evaluation configuration +separately. Changing source facts, Cases, or annotations changes the dataset. Changing judgment +rules or analysis configuration can reuse the same facts in a distinct evaluation when the native +capability supports it. Rendering a report neither recollects evidence nor reruns judgment. + +Comparing subject behavior requires separate executions with aligned inputs and controlled run +conditions. Comparing judgment components requires fixed source evidence. Identify unmatched +factors, missing observations, and coverage denominators before attributing differences to a +revision. Domain skills define the remaining comparability requirements. + +## Capability, health, and decision + +Keep three questions separate: + +1. **Can this task run?** `ready` means its selected action has the necessary capability and inputs; + `blocked` means a prerequisite is unavailable; `no_capability` means no usable project-owned + entrypoint or sufficient evidence exists for the requested action. Offline analysis can be ready + without a live target; live execution requires verified application connectivity. +2. **Did it execute correctly?** Preserve errors, cancellations, missing observations, partial + evaluation coverage, and cleanup failures. Configured components may not have executed. +3. **What passed?** Preserve native verdict states and their scope. Skipped or missing judgments + are not passes; analysis without a gate policy does not prove release readiness. Passing + assertions or SLOs do not erase an execution or cleanup error. + +Report the decision with its subject and provenance, realized conditions, command or evidence +source, coverage, native result, and artifact paths. Keep run-specific facts and reusable project +operating knowledge beside the project's assets; keep credentials, private data, and volatile +environment details out of the plugin. diff --git a/quality/README.md b/quality/README.md index bd36483..d784a34 100644 --- a/quality/README.md +++ b/quality/README.md @@ -13,55 +13,36 @@ The bundled skills currently cover three independent quality views: - `trajectory` evaluates agent decisions and actions, compares effect and cost, finds the next evidence-backed problem, and organizes controlled tuning experiments. -Each skill connects three layers without merging their ownership: - -1. a framework such as case-harness may provide reusable execution, load, evaluation, measurement, - artifact, and reporting primitives; -2. each project owns its real cases, workloads, profiles, recordings, adapters, labels, Evaluators, - Measurers, acceptance criteria, and canonical entrypoints; -3. the skill discovers and operates that project capability, interprets its evidence, helps evolve - project-owned assets when requested, and preserves project-specific knowledge beside them. - -During assessment, if no runnable project-owned capability exists, a skill returns `no_capability` -rather than inventing tests or evaluations, or presenting source review as execution evidence. -Explicit coverage work may evolve an existing project-owned capability; it does not make an absent -suite look executable. +The project owns the tests, profiles, recordings, adapters, and judgment criteria. A framework +such as quality-harness supplies execution and analysis mechanisms; these skills discover and +operate the project's canonical entrypoints, interpret evidence, and preserve useful operating +knowledge beside the project assets. ## Operating model -The skills share a small quality loop while preserving each domain's native semantics: +The skills share [Quality concepts](CONCEPTS.md): subject and environment identity, execution +facts, evidence and evaluation, comparison, and the distinction between capability, execution +health, and quality decisions. Each skill retains its domain's lifecycle and judgment rules. ```text -understand the target and runner path - → discover project-owned capability - → gate on executability - → run the canonical entrypoint - → interpret native evidence - → report conclusions and unknowns - → retain reusable operating knowledge +identify the question and subject + → discover a project capability or sufficient recorded evidence + → prepare live execution or select existing evidence + → use the canonical entrypoint + → interpret results and coverage + → report the conclusion and next supported action ``` -E2E and performance execution share one target-confirmation boundary: identify the system under -test and revision, decide whether it is local or remote, locate the runner, and verify the -application data-plane path between them. Kubernetes API access or a configured endpoint is not by -itself target readiness. Each skill then adds its own preparation: E2E prepares cases, fixtures, -and scenario controls; performance additionally prepares the resource profile, load generator, -dependency capacity, observations, safety limits, and cooldown. - -Execution facts, judgments, and missing evidence remain distinct. A failed assertion is not a -runner error; an unavailable capability is not a pass; a useful analysis is not automatically a -release verdict. Skills may organize evidence and propose the next experiment, but they do not -claim causality or mutate the evaluated system unless the user asks. - -## Skills first, orchestration later - -Quality intentionally starts at the skill layer. E2E, performance, trajectory, and other quality -views should first mature as independent operators with clear inputs, evidence, outcomes, and -ownership. This keeps real project workflows visible while their common shape is still emerging. +Live E2E and performance runs follow [environment preparation](references/environment.md): select +the target and revision, locate the test process, verify application connectivity, prepare +conditions, and retain cleanup evidence. Offline analysis reuses recorded facts without requiring +a live target. No usable capability or sufficient evidence is reported as `no_capability`; +an unavailable prerequisite is `blocked`. -Only after several skills demonstrate stable contracts and repeated composition needs might an -agent- or orchestration-level Quality Harness coordinate them. That is a future consumer of these -skills, not a dependency or implementation goal of this plugin today. +Execution facts and judgment remain separate. A passed assertion or SLO covers only its realized +conditions; missing observations, skipped checks, and execution errors remain visible. Skills +can propose the next experiment or improve project-owned assets when requested. They neither +invent a missing suite nor supply cross-skill execution orchestration. ## Install @@ -101,7 +82,7 @@ For this high-token trajectory problem, identify likely causes and controlled ex ``` The skills operate existing project-owned quality assets. Assessment does not invent missing tests, -workloads, profiles, recordings, labels, or evaluators. When the user asks to grow coverage, changes +load profiles, recordings, labels, or judgment components. When the user asks to grow coverage, changes remain in the project-owned capability and move toward broader coverage incrementally. The skills do not deploy environments without authorization, modify agent behavior unless asked, or turn unavailable coverage into a pass. diff --git a/quality/references/environment.md b/quality/references/environment.md index 3992bd4..6ec0f7d 100644 --- a/quality/references/environment.md +++ b/quality/references/environment.md @@ -1,121 +1,95 @@ -# Environment +# Environment preparation -An **Environment** is the complete observable state required to execute a quality capability -against the intended system. It may be local, dev, or another remote environment; it does not need -to be a dedicated test environment. E2E and performance runs share this contract even though they -prepare different domain-specific conditions. +Read [Quality concepts](../CONCEPTS.md) for Environment, Service, Workload, and execution identity. +This contract prepares live E2E and performance runs. Offline analysis of existing evidence does +not require a live target or connection preparation. -```text -Environment = Target + Runner + Connection - + Preconditions + Fixtures + Controls - + Cleanup + Evidence -``` +Run conditions include the selected target, test-process location, connection, prerequisites, +fixtures, temporary controls, and cleanup. Record them separately from Environment identity so a +changed network path or test location remains visible without inventing another environment. -- **Target** identifies the system under test, revision, configuration, and environment. It may be a - local process or a remote environment such as Kubernetes. -- **Runner** is where the quality command and its traffic generator execute. -- **Connection** is the application data-plane path from Runner to Target. -- **Preconditions** are target or dependency facts that must already hold. -- **Fixtures** are records, identities, files, or other state owned by the run. -- **Controls** are temporary changes such as fault injection, resource selection, or load. -- **Cleanup and Evidence** prove what was restored and what actually ran. +The project owns startup and access mechanisms, credentials, fixtures, controls, and cleanup. +Use its canonical entrypoint and runbook to prepare and verify these conditions. -The project owns target names, startup and access mechanisms, credentials, fixtures, controls, and -cleanup. A quality skill discovers and operates that knowledge; it does not replace it with generic -environment assumptions. +## Resolve the target and execution location -## Select the Target before choosing the Connection +Resolve in order: -Resolve the Environment in order: +1. **Target:** identify the Service or system boundary, Environment, deployed revision, + configuration, namespace or other scope, and whether it is dedicated or shared. +2. **Execution location:** decide where the quality command and traffic generator execute after + the target is fixed. Record the test-process revision separately from the target revision. +3. **Connection:** select and verify the application data-plane path from that process to the target. -1. **Target:** identify the exact environment, subject revision, configuration, namespace or other - scope, and whether it is dedicated or shared. -2. **Runner:** decide where the quality command executes after the Target is fixed. -3. **Connection:** choose and verify the application data-plane path from that Runner to that Target. +Do not infer the target from whichever endpoint happens to respond. Resolve the endpoint from +project configuration and target state, then probe its exact host, port, and protocol from the +actual execution location. Confirm the deployed revision and health independently when applicable. -Do not start from whichever endpoint happens to respond and infer the Target from it. A convenient -connection must not silently change the selected environment, revision, or scope. - -Common shapes include: - -- **Local Target:** start or discover it through the canonical project entrypoint, then probe the - same host, port, and protocol used by the quality workload. -- **Remote Target:** use a project-owned application path. For Kubernetes, API or `kubectl` access - proves control-plane reachability only; it does not prove that the Runner can exchange application - traffic with a Service or Pod endpoint. - -Resolve the endpoint from target state instead of guessing a hostname, namespace, address, or port. -Probe it from the actual Runner and, when relevant, independently confirm the deployed revision and -health from the Target side. A configured URL, successful deployment command, or control-plane -login is not readiness evidence by itself. +For a local target, use the project's startup or discovery mechanism. For Kubernetes, API or +`kubectl` access proves control-plane reachability only. Neither it nor a configured Service or +Workload proves that the test process can exchange application traffic with the target. ### Choose the Kubernetes connection by priority -After selecting a Kubernetes Target, use the first applicable authorized option when the connection -mechanism is not itself under test: - -1. **Run-owned port-forward:** prefer a project-owned port-forward that the quality run starts, - probes, observes, and stops. Use the project's wrapper when one exists; otherwise bind the - forward to the resolved Service or Pod and target port, choose a run-scoped local port, preserve - logical service authority such as TLS SNI or HTTP `Host`, retain logs, and own readiness and - shutdown. -2. **Direct application endpoint:** use project-owned ingress or a `ClusterIP:port` that is routable - without an ambient connection helper. For a normal Service, prefer ClusterIP over Service DNS to - avoid local DNS and VPN DNS behavior without bypassing the Service data plane. Preserve required - TLS SNI, HTTP `Host`, service-mesh routing, or other logical authority through project-owned - Runner configuration; use Service DNS when the ClusterIP is not routable or DNS is part of the - intended connection path or behavior under test. -3. **In-environment Runner:** move the Runner into the Target environment when a local connection is - unavailable or unsuitable, and report the changed Runner explicitly. -4. **Ambient VPN or shared tunnel:** use this only when earlier options are unavailable or the - project requires that path; verify it for this run rather than trusting existing host state. - -Do not substitute a port-forward when Service DNS, ingress, VPN behavior, or another network path is -part of the behavior being verified. For performance runs, also treat the forwarder as Runner -capacity: use it only when it can carry the declared profile without becoming an unmeasured -bottleneck. Otherwise use an authorized direct path or in-environment Runner and report the changed -Connection. - -If connection setup fails, distinguish an unavailable Target, a broken Runner-to-Target path, and a -Runner error. An alternate endpoint, tunnel, or Runner is a changed Environment: verify it, -preserve the first failure, and report the changed execution conditions rather than silently -retrying for green. - -## Prepare the environment - -Keep these preparation kinds visible because they have different failure and cleanup semantics: - -1. **Pre-existing conditions** come from the selected Target, such as a deployed revision, enabled - capability, dependency, quota, or resource policy. -2. **Run-owned fixtures** are created for the run. Give them stable run identity, isolate them from - concurrent work, and clean them through project-owned mechanisms. -3. **Transient controls** deliberately alter behavior for a bounded period. Apply them only after - steady-state readiness is proven, verify activation, and always restore them. - -Do not infer readiness from setup exit status. Observe each condition at the layer the scenario -depends on. An absent required condition makes the run `blocked` or `error`; it is not a product -pass and must not be silently removed from the realized Environment. +When the connection mechanism is not itself under test, use the first applicable authorized option: + +1. **Run-owned port-forward:** prefer a project-owned forward that the run starts, probes, + observes, and stops. Use the project's wrapper when available; otherwise resolve the Kubernetes + Service or Pod and target port, choose a run-scoped local port, preserve TLS SNI or HTTP `Host`, + and own readiness, logs, and shutdown. +2. **Direct application endpoint:** use project-owned ingress or a routable `ClusterIP:port`. + For a normal Kubernetes Service, prefer ClusterIP over Service DNS to avoid ambient DNS and VPN + dependencies while retaining the Service data plane. Preserve TLS SNI, HTTP `Host`, service-mesh + routing, and other logical authority through project configuration. Use Service DNS when + ClusterIP is not routable or DNS is required by the intended path or behavior. +3. **In-environment execution:** move the test process into the target Environment when local + access is unavailable or unsuitable, and report the changed execution location. +4. **Ambient VPN or shared tunnel:** use this when earlier options are unavailable or the project + requires that path; verify it for this run rather than trusting existing host state. + +Do not substitute a port-forward when Service DNS, ingress, VPN behavior, or another network path +is under test. For performance work, assess the connection helper's capacity as part of the load +path. If it could bottleneck the declared capacity, stress, or soak profile, use an authorized +direct path or in-environment execution and record the chosen connection. + +If setup fails, distinguish an unavailable target, a broken connection, and a test-process error. +Verify any alternate path, preserve the first failure, and report changed execution conditions. +An endpoint or execution-location change is not automatically a change in Environment identity. + +## Prepare conditions and effects + +Keep preparation kinds visible because they have different failure and cleanup semantics: + +1. **Pre-existing conditions** include the target revision, enabled capabilities, dependencies, + quotas, and resource policy. +2. **Run-owned fixtures** are records, identities, files, or other resources created for this run. + Give them stable run identity, isolate concurrent work, and clean them through project mechanisms. +3. **Transient controls** deliberately alter behavior for a bounded period, such as fault injection, + load, or a resource setting. Verify steady state first, then activation, and always restore them. + +Observe each required condition at the layer the scenario depends on. Setup exit status alone is +not readiness evidence. If a required condition is missing, preserve the `blocked` or `error` state +and identify that condition rather than silently removing it from the run. ## Follow the lifecycle 1. Read the project runbook nearest to the selected capability. -2. Resolve the Target, then the Runner and prioritized Connection, then conditions, effects, and - evidence. -3. Check authorization before deploying, creating durable data, changing resources, injecting - faults, or generating material load. -4. Prepare preconditions and fixtures through canonical mechanisms. -5. Verify Target health and application connectivity from observable state. +2. Resolve the target, execution location, connection, required conditions, and expected effects. +3. Check existing authorization before deployment, durable-data creation, resource changes, fault + injection, or material load; obtain authorization for actions outside it. +4. Prepare prerequisites and fixtures through canonical mechanisms. +5. Verify target health and application connectivity from observable state. 6. Apply and verify transient controls. -7. Execute without silently repairing or changing the Environment during measurement. -8. Remove controls, clean fixtures, stop connection helpers, and verify restoration even after a - failed run. +7. Execute under the declared conditions; record deviations instead of silently repairing the + target or connection during measurement. +8. Remove controls, clean fixtures, stop connection helpers, and verify restoration even after failure. -Cleanup is evidence, not a business verdict. A capability may pass its assertions or SLOs and still -finish with a cleanup error; report both outcomes. +Keep assertions or SLO results and cleanup health separately visible. A successful assertion does +not erase a cleanup error or override the native run-level Verdict. -## Preserve evidence safely +## Preserve run evidence -Retain the project and Runner revision, Target identity, connection path, endpoint identity, -application readiness, conditions and controls, fixtures, start/end time, cleanup, native Verdict, -and artifact paths. Keep credentials, private endpoint details, and volatile environment facts out -of reusable plugin guidance. +Retain the target and test-process revisions, Environment and Service identity, relevant Workload +instances, execution location, connection and endpoint identity, readiness observations, conditions, +controls, fixture identities, start/end times, and cleanup. Include native results and artifact +paths under the provenance and privacy rules in Quality concepts. diff --git a/quality/skills/e2e/SKILL.md b/quality/skills/e2e/SKILL.md index 9dde56e..2a2e484 100644 --- a/quality/skills/e2e/SKILL.md +++ b/quality/skills/e2e/SKILL.md @@ -11,17 +11,12 @@ read-only; when the user explicitly asks to add or fix coverage, change the proj under its development rules and validate them through the canonical entrypoint. Comprehensive coverage is the long-term direction, not a requirement to close every gap in one task. -## Keep the owners separate +Read [Quality concepts](../../CONCEPTS.md) before discovery or interpretation. It defines ownership, +subject identity, execution facts, evaluation, comparison, and result states shared by the skills. -1. Treat case-harness or another framework as infrastructure that may provide runners, lifecycle, - assertions, evidence, and verdict primitives. -2. Treat the project as owner of its E2E cases, fixtures, adapters, acceptance criteria, canonical - commands, and operating notes for environments, retries, cleanup, and evidence. -3. Treat this skill as the operator that discovers and runs those assets, interprets their evidence, - exposes remaining coverage, and helps add bounded project-owned cases when requested. - -Do not call a lower-level framework directly when the project exposes a wrapper. Framework code -without runnable project-owned tests is `no_capability`, not an executable E2E suite. +The project owns E2E Cases, fixtures, adapters, acceptance criteria, canonical commands, and +operating notes. Preserve its test grouping and call evidence: a CaseRun may contain several +OperationRuns. Framework code without runnable project-owned tests cannot establish live coverage. ## Discover and select @@ -31,7 +26,7 @@ Read the project's AGENTS.md and README first, then locate: - E2E, acceptance, integration, or system-test directories and their nearest operating notes; - the canonical command, exact case registry or listing mechanism, configuration, fixtures, and recent result artifacts; -- the runner revision and the identity of the system under test, including the requested target or +- the test-process revision and the identity of the system under test, including the requested target or environment, where the runner executes, and how it reaches that target; - project-declared prerequisites, material execution conditions, side effects, cleanup, and available evidence; @@ -41,32 +36,18 @@ Prefer the business-facing suite when several lower-level runners exist. Resolve from project registries, listing commands, or documentation instead of guessing them. Project-local knowledge takes precedence over generic framework habits. -Classify discovery as: - -- `ready`: an executable suite, selected cases, required inputs, and a verified connection from the - runner to the intended target are available; -- `blocked`: the suite exists but a required target, credential, dependency, service, device, or - other prerequisite is unavailable; -- `no_capability`: no executable project-owned E2E suite can be found. +Classify the requested action as `ready`, `blocked`, or `no_capability` using the shared concepts. +A live suite requires selected cases, inputs, prerequisites, and verified application connectivity. +Existing result artifacts can support an offline report without a reachable target. Report ambiguity when multiple plausible suites or targets remain. Do not turn framework presence, source review, or a compile-only check into execution evidence. ## Prepare and control the environment -Before a live run, read the shared -[Environment contract](../../references/environment.md) completely. It defines local and remote -Targets, Runner-to-Target data-plane readiness, preparation kinds, authorization, lifecycle, -cleanup, and evidence. - -First fix the Target environment and subject revision; only then select the Runner and use the first -applicable authorized Connection in the contract's priority order. For a local Runner reaching -Kubernetes, that normally means a run-owned port-forward when the network path is not itself under -test, with readiness, logs, and cleanup owned by the run. - -When using a direct Kubernetes Service endpoint instead, follow the contract's IP-first selection, -preserve required logical service authority, and do not bypass Service DNS when name resolution is -part of the E2E behavior being verified. +Before a live run, read [environment preparation](../../references/environment.md) completely. +Resolve the target Service, Environment, and deployed revision first, then the test-process location +and application connection. Follow the shared connection selection and readiness checks. When selected cases require fixtures, temporary conditions, fault injection, chaos experiments, or restoration, also read @@ -76,7 +57,7 @@ authorized Target and blast radius. ## Run and interpret -Use the project-owned entrypoint and preserve the requested runner revision, system-under-test +Use the project-owned entrypoint and preserve the requested test-process revision, system-under-test identity, target, and execution policy. Run the smallest sufficient selection by default; use a full suite only when the user, a project gate, or the affected boundary requires it. @@ -88,9 +69,9 @@ suite only when the user, a project gate, or the affected boundary requires it. reports, cleanup outcomes, or other project-declared evidence. - Treat target-connection failures as environment `error` or `blocked`, not product assertion failures. If a verified alternate connection path is used, preserve the first error and report the rerun as - execution under a changed Environment rather than silently retrying for green. -- Use native result semantics. Keep `passed`, `failed`, `skipped`, `error`, `blocked`, and - `no_capability` distinct; an online case that skipped is not a live pass. + execution under changed run conditions rather than silently retrying for green. +- Preserve native verdict states such as `passed`, `failed`, `skipped`, and `error`, separately + from capability discovery states. An online case that skipped is not a live pass. - Separate verdict from coverage. `passed` means the executed assertions passed under the realized conditions; skipped, gated, unselected, or unknown areas remain unverified. @@ -124,9 +105,9 @@ behavior. Lead with the decision-relevant result, then report: -- the project, runner revision, system-under-test identity, target, and E2E boundary; -- the runner location, target connection path, and application-level connectivity evidence; -- the prepared Environment, fault or chaos conditions, and readiness evidence relevant to the verdict; +- the project, test-process revision, system-under-test identity, target, and E2E boundary; +- the execution location, target connection path, and application-level connectivity evidence; +- the prepared run conditions, fault or chaos controls, and readiness evidence relevant to the verdict; - the canonical entrypoint and exact cases executed; - the native verdict and supporting evidence; - executed, skipped, gated, unselected, and unknown or unverified areas; @@ -134,6 +115,6 @@ Lead with the decision-relevant result, then report: - execution errors, blocked prerequisites, side effects, and cleanup outcomes; - reusable project-specific operating knowledge observed during the work. -Persist run-specific evidence and project-specific lessons beside the project's E2E assets. Promote -an operating pattern into this skill only after it proves reusable across projects. Do not copy -private data, credentials, volatile environment facts, or project-only commands into the plugin. +Persist evidence and reusable operating knowledge beside the project's E2E assets under the shared +provenance and privacy rules. Promote an operating pattern into this skill only after it proves +reusable across projects. diff --git a/quality/skills/e2e/references/scenario-environment.md b/quality/skills/e2e/references/scenario-environment.md index 89c8a89..d7a8d12 100644 --- a/quality/skills/e2e/references/scenario-environment.md +++ b/quality/skills/e2e/references/scenario-environment.md @@ -1,6 +1,6 @@ # E2E scenario preparation -Read the [shared Environment contract](../../../references/environment.md) first. This +Read the [environment preparation](../../../references/environment.md) first. This reference adds the E2E-specific preparation for fixtures, scenario controls, fault injection, and chaos experiments. @@ -44,6 +44,6 @@ mechanism and keep fault activation, business assertion, and cleanup as separate ## Retain E2E evidence -In addition to the shared Environment evidence, retain selected cases, fixture identities, +In addition to the shared run evidence, retain selected cases, fixture identities, control or fault activation, steady-state and recovery observations, executed/skipped coverage, assertion results, and cleanup outcomes. diff --git a/quality/skills/perf/SKILL.md b/quality/skills/perf/SKILL.md index 0a9a5ba..f1549ad 100644 --- a/quality/skills/perf/SKILL.md +++ b/quality/skills/perf/SKILL.md @@ -5,146 +5,110 @@ description: Discover, run, interpret, compare, and incrementally extend a proje # Performance Quality -Assess how a deployed system behaves under a declared resource profile and load. Operate the -project's existing performance capability; do not replace its load framework, workload semantics, -target policy, or service-level objectives. - -## Keep the owners separate - -1. Treat case-harness or another performance framework as infrastructure for load scheduling, - lifecycle, observation, metric reduction, artifacts, reports, and Verdicts. -2. Treat the project as owner of its Workloads, Cases, Experiment/Profile configuration, target - adapters, Probes, SLOs, safety limits, canonical command, and comparison baselines. -3. Treat this skill as the operator that discovers those assets, selects and runs the relevant - experiment, checks evidence validity, interprets or compares results, and helps evolve bounded - project-owned coverage when requested. - -Do not call a lower-level framework directly when the project exposes a wrapper. Framework code -without a project-owned workload and runnable profile is `no_capability`. - -## Discover the capability - -Read the project's AGENTS.md and README first, then locate: - -- canonical commands and directories for performance, load, capacity, stress, soak, scalability, - or regression testing; -- the subject boundary and target environment, including whether it is a local process or remote - system such as Kubernetes, the deployed revision, where the load runner executes, and how it - reaches the application endpoint; -- workload adapters, stable Cases or stimuli, case mix, and protocol-specific success judgment; -- named profiles or experiments, resource profiles, open- or closed-loop load, stages, duration, - concurrency or arrival rate, and request/time limits; -- SLOs, breakers, abort conditions, cleanup, target authorization, and operating notes; -- request and resource observations, run models, Verdicts, reports, and historical baselines. - -Choose the smallest existing profile that answers the user's question. Do not silently replace a -capacity scan or soak test with a smoke profile and present the narrower result as equivalent. - -Classify discovery as `ready`, `blocked`, or `no_capability`. `ready` requires a verified data-plane -connection from the load runner to the intended target, not only a configured URL or Kubernetes -control-plane access. A profile with an unavailable target, credential, dependency, quota, load -runner, or required observer is `blocked`, not an executable pass. Report ambiguity instead of -guessing when repository evidence cannot select a profile or target. - -## Prepare a live run safely - -Before generating live load or changing a performance Target, read both the shared -[Environment contract](../../references/environment.md) and -[performance load-environment preparation](references/load-environment.md) completely. Inspection and -offline re-analysis of an existing run do not require Environment preparation. - -Prepare three explicit layers in order: - -1. **Target environment:** identify the local or remote Target, subject revision, scope, and - declared resource profile before choosing how to reach it. -2. **Runner and connection:** select the Runner, use the first applicable authorized Connection in - the Environment contract's priority order, then probe the exact endpoint from that Runner. Keep - the path stable across compared runs; switching between Service DNS and ClusterIP is an - Environment change and must be reported. -3. **Load environment:** establish load-generator capacity, - dependency quotas, fresh request/resource observations, initial steady state, safety limits, - cooldown, and cleanup path. - -For a local Runner reaching Kubernetes, the first Connection is normally a run-owned port-forward. -For capacity, stress, or soak work that could saturate it, skip that option and use an authorized -direct path or in-environment Runner so the connection helper does not become an unmeasured -bottleneck. - -Preserve the requested revision, target, profile, workload, resource envelope, load model, case mix, -and SLO policy. Do not deploy, install dependencies, create credentials, resize resources, or switch -targets unless the user authorized that action. Shared and production environments require -authorization appropriate to the declared load and blast radius. - -## Run and preserve evidence - -Use the project-owned entrypoint and its native lifecycle. Record the exact command and realized -configuration, not only the profile filename. Respect warmup, ramp, measurement, cooldown, breaker, -graceful-stop, cleanup, and retry semantics; do not edit a profile midway or retry a failed run only -to obtain green output. - -Keep these facts distinct: - -- load is an experimental input, not an observed metric; -- request Outcomes and resource samples are observations; -- per-request judgment and run-level SLOs are evaluation policy; -- framework or environment errors are not SLO failures. - -Treat target-connection and load-generator failures as environment `error` or `blocked`, not product -latency or error-rate regressions. An alternate runner, endpoint, tunnel, or in-cluster execution is -a changed Environment and must be reported as such. - -Capture the native run model, raw request/resource facts, Verdict, report, logs, traces, and cleanup -evidence exposed by the project. When the runner supports offline report or SLO recomputation, use -the persisted run rather than generating load again. +Assess how a system behaves under declared resources and load. Operate the project's performance +capability and explain the capacity, latency, resource use, and limitations supported by its facts. + +Read [Quality concepts](../../CONCEPTS.md) before discovery or interpretation. It defines ownership, +subject identity, execution facts, evaluation, comparison, and result states shared by the skills. + +The project owns its Cases, protocol Runners, Judges, resource/load profiles, Probes, SLOs, safety +limits, and baselines. A perf Runner invokes a service protocol; Workload identifies a Service's +platform carrier. Record where the load generator executes separately from both. + +## Discover and select + +Read the project's AGENTS.md and README, then locate: + +- the canonical performance, capacity, stress, soak, or regression commands and operating notes; +- the target Service, Environment, deployed revision, Workloads, load-generator location, and + application connection path; +- Cases or stimuli, case mix, protocol Runner, raw completion signals, and independent Judge; +- named experiments and resource/load profiles, both request rate and maximum inflight limits, + phase schedules, duration, arrival distribution, and request/time budgets; +- Probes, SLOs, breakers, stop/cancellation behavior, cooldown, cleanup, and target authorization; +- native run schema, raw requests and resource observations, evaluation records, reports, and + comparison baselines. + +Classify the requested action as `ready`, `blocked`, or `no_capability` using the shared concepts. +An executable load test requires a project-owned adapter, runnable profile, and verified target +connection. Existing artifacts can support offline analysis without a reachable target. + +Choose the smallest existing profile that answers the question. Preserve a requested capacity scan +or soak test's scope; a smoke run cannot stand in for it. Resolve ambiguity from project evidence +and ask when the intended profile or target remains unclear. + +## Prepare and run + +Before live load or target changes, read [environment preparation](../../references/environment.md) +and [load-environment preparation](references/load-environment.md) completely. Resolve the target, +then execution location and connection, then load-generator capacity, dependency quotas, fresh +observations, steady state, safety limits, and cleanup. + +Before selecting or interpreting load, read [load and evidence](references/load-and-evidence.md) +completely. Use the project's actual scheduling contract: inspect what happens at the inflight +limit, how phases end, and which events determine windows and statistics. + +Use the canonical entrypoint and native lifecycle. Preserve the requested revision, resource +envelope, load, case mix, and SLO policy. Record the exact command and realized configuration; +respect warmup, hold, cooldown, breaker, stop, cancellation, and cleanup behavior. Do not edit a +profile during measurement or retry a failed run only to obtain green output. + +Deploying, installing dependencies, creating credentials, resizing resources, switching targets, +or generating material load requires authorization covering that action and target. Record changed +execution conditions and preserve the first failure when an alternate connection is used. + +Capture the native execution model, raw requests and resource samples, independent evaluations, +Verdict, report, logs, traces, and cleanup evidence. Use persisted facts for supported offline +judgment, SLO recomputation, or report rendering; changing a judgment is not a reason to generate +load again. A Judge error must remain visible even when a completed Outcome is available. ## Validate and interpret Check data health before drawing a performance conclusion: -1. Verify the subject revision, target, resource profile, load model, case mix, and comparison - cohort are the intended ones. -2. Check completed measurement windows, stop reasons, sample counts, error and drop counts, - cancellations, probe errors, missing series, and skipped SLOs. -3. Treat latency as a fact only for completed requests. Account for open-loop drops and closed-loop - coordinated-omission caveats rather than comparing percentiles in isolation. -4. Read request and resource facts over the same realized window. Ramp, hold, measurement, and - cooldown answer different questions and must not be flattened into one aggregate. -5. Read throughput, latency distribution, errors, drops, saturation, resource use, restarts, and - scaling behavior together. Capacity claims require complete stable-load windows. -6. Compare runs only after checking workload, load, resources, environment, and SLO comparability. - Describe unmatched factors instead of attributing every difference to the code revision. - -A passed SLO means the declared gate passed under the realized experiment; it does not prove an -untested capacity. A regression is an observed relationship, not a root-cause diagnosis. Profiling, -tracing, or source investigation is separate evidence. - -Preserve native outcomes: `passed`, `failed`, `skipped`, `error`, `blocked`, and `no_capability` are -distinct. Missing strict observations or incomplete windows cannot be converted into a pass. +1. Verify the target and revision, resources, both load axes, case mix, and realized phase windows. +2. Check stop reasons, completed samples, missing judgments, drops, interruptions, inflight state, + time limited by the inflight cap, Probe errors, missing series, and skipped SLOs. +3. Distinguish configured rate, actual dispatch rate, and completion throughput. No drops does not + prove the configured rate was sustained. Read scheduling and coordinated-omission caveats. +4. Align request and resource windows. Interpret latency by the native request cohort and + throughput by actual completion events; inspect warmup, hold, and cooldown separately. +5. Confirm capacity only from complete stable-load windows with sufficient requests and passing + applicable SLOs. A finite-rate window blocked by its inflight cap cannot confirm that rate's + capacity; use the native eligibility rules in the evidence reference. +6. Compare within a declared scan axis while holding the other axis, resources, scheduling shape, + and execution conditions fixed. Separate incomparable points instead of fitting one curve. + +A passed SLO supports its declared gate under the realized experiment. It does not establish +untested capacity or a root cause. Correlate latency, throughput, errors, saturation, resources, +restarts, and scaling; use profiling or trace evidence for causal investigation. + +Preserve useful completed windows after later failures while reporting the run's error or cleanup +state. Keep missing evidence, interrupted measurement, and skipped gates visible. ## Grow coverage deliberately -When the user explicitly asks to add or strengthen performance coverage: +When the user asks to add or strengthen performance coverage: -1. Map the existing workload, profiles, metrics, and baselines; choose one bounded missing question. -2. Reuse project adapters and lifecycle. Keep stable input and judgment in project Case assets; - keep load weights, stages, resources, and SLO selection in the Experiment/Profile. -3. Add protocol facts in the Workload and observations in Probes before adding derived claims to a - report. Make a metric affect pass/fail only through an explicit project-owned SLO. -4. Validate configuration and workload behavior offline, then run the smallest relevant live +1. Map existing Cases, adapters, profiles, observations, and baselines; choose one missing question. +2. Keep stable input and expected behavior in project Case assets. Select case weights, resources, + load phases, and SLOs in the Experiment/Profile. +3. Capture protocol facts in the Runner and resource facts in Probes. Implement per-request + judgment independently; make observations affect pass/fail through explicit project SLOs. +4. Validate configuration and adapter behavior offline, then run the smallest relevant live experiment when its target and impact are authorized. -5. Do not weaken thresholds, shorten measurement until it loses meaning, or drop missing data merely - to make the new profile pass. +5. Preserve meaningful windows and thresholds. Missing observations need explanation or repair, + not deletion to make a profile pass. -Improve a shared harness only when the capability is genuinely reusable across projects. Project -protocols, target locations, credentials, load limits, and learned thresholds stay in the project. +Reuse project lifecycle and clients. Improve a shared harness only for a reusable mechanism; +project protocols, endpoints, credentials, budgets, and thresholds stay in the project. ## Report and retain learning -Lead with the decision-relevant result, then report the project and revision, target and subject, -runner location and connection path, profile and exact command, resource/load/case-mix realization, -safety controls, native Verdict, data-health caveats, SLO results, key request and resource -observations, baseline comparison, artifact paths, cleanup, and unverified areas. +Lead with the supported performance conclusion. Include target and generator revisions, execution +location and connection, exact command, realized resource/load/case mix, phase boundaries, stop +and cleanup state, native Verdict, SLO coverage, key request/resource facts, comparison conditions, +and artifact paths. State the observed limit separately from any extrapolation and its assumptions. -Persist run-specific evidence and reusable project operating knowledge beside the project's perf -assets. Do not copy credentials, private endpoints, volatile measurements, or project-only commands -into this skill. +Persist evidence and reusable operating knowledge beside the project's perf assets under the +shared provenance and privacy rules. diff --git a/quality/skills/perf/references/load-and-evidence.md b/quality/skills/perf/references/load-and-evidence.md new file mode 100644 index 0000000..dbe178b --- /dev/null +++ b/quality/skills/perf/references/load-and-evidence.md @@ -0,0 +1,114 @@ +# Performance load and evidence + +Use this reference when selecting load, interpreting a run, or comparing experiments. Shared +identity and evidence semantics are defined in [Quality concepts](../../../CONCEPTS.md). + +## Establish the native load contract + +Inspect the project's wrapper, installed framework version, profile, and persisted schema together. +Use its native scheduler and result rules. The quality-harness LoadPlan contract below provides +concrete interpretation when that SDK is in use; other frameworks can have different admission, +drop, queue, and phase behavior. Preserve those differences in the report. + +Load has two independent axes: + +- `request_rate`: the configured request initiation rate; +- `max_inflight`: the maximum simultaneous requests across their full lifecycle, including reading + a complete SSE response. + +For quality-harness, finite rate controls initiation while slots are available; a full inflight +limit pauses the source. There is no pending queue, fabricated arrival/drop, or catch-up burst. +`inf` replenishes available slots. Lowering the cap pauses new calls without canceling existing +ones. An inflight-limited run can show zero drops and acceptable latency while sending less than +the configured rate. Check `limited_s`, dispatch rate, and completion throughput together. + +For a framework with open-loop admission, account for offered arrivals and drops; for feedback +load, explain how slow responses reduce offered traffic. Never compare latency percentiles while +hiding a change in the admission model or coordinated-omission caveats. + +## Follow phases and realized windows + +Quality-harness uses `setup → warmup → hold → cooldown → cleanup`: + +- Warmup steps toward the configured rate until it reaches that rate or the inflight cap. +- Hold starts when actually entered and uses the configured target rate as its replenishment + pace, even if warmup reached the cap at a lower rate. Its duration is `hold_s`. +- Cooldown stops initiation and waits up to `cooldown_timeout_s`, then cancels and joins remaining + calls before releasing clients. Optional `cooldown_s` extends resource observation after + deactivation; it is separate from the wait for inflight requests. +- Explicit stages own their duration, rate, inflight limit, and warmup/hold/ramp kind. Default + warmup and hold durations are not added to an explicit schedule. + +Stages are plans; Windows record actual boundaries. Read `end_reason`, `complete`, and `limited_s` +with the stop record. Measurement begins at the first actual hold; drain time or a later Judge +exception must not extend it. Requests and resource samples use the same realized boundaries, +while warmup, individual stages, aggregate measurement, and cooldown retain distinct meanings. + +Respect project-specific phase behavior when another framework is used. Never flatten warmup, +measurement, and recovery into a single apparent steady-state result. + +## Join execution and evaluation evidence + +An Arm names a resource/load configuration. An ArmRun is one actual execution containing request +scheduling records, OperationRuns with raw Outcomes, and independent per-request evaluations. +Use execution and operation IDs for joins. Display labels or configuration hashes do not identify +repeated executions. Check for ambiguous or duplicate identities before aggregating runs. + +With quality-harness, inspect: + +| Artifact | Evidence | +|---|---| +| `run.json` | Native schema, execution identities, Arm configuration, realized Windows, phase errors, and stop state. | +| `requests.jsonl` | Scheduling records and actual OperationRuns with their unique raw Outcomes; a drop has no call. | +| `evaluations.json` | Request judgments keyed by ArmRun and OperationRun identity. | +| `timeseries.csv` | Resource observations, when supplied by the implementation. | +| `verdict.json` | The native run-level decision and evidence references. | + +Verify the reader supports the artifact schema. Python provides resource observation, SLO +evaluation, and HTML reporting; the TypeScript perf SDK supplies request execution and evidence, +with resource observation and gates supplied by the consumer. Discover the consumer's actual +capabilities rather than assuming SDK parity or a generic EvaluationRun/Worksheet artifact. + +Preserve completed Outcomes when a Judge raises. Treat absent judgments as incomplete evaluation, +not observed product errors. Interrupted calls have their own records and do not enter latency +distributions or get judged as completed responses. Keep stop-time inflight counts, interruption +census, phase errors, and cleanup results alongside valid completed evidence. + +## Interpret rates, latency, and business success + +For quality-harness's half-open Windows `[start_s, end_s)`: + +- Arrivals count actual accepted initiation opportunities by scheduled time; dispatches count + actual sends by dispatch time. +- Completion throughput and success throughput count finished calls by finish time. +- Latency and request error rate use completed requests from the dispatch cohort, including those + that finish during cooldown. Their sample count differs from completions inside the Window. +- Inflight peak/end include requests carried in from earlier Windows. + +Drops and interruptions are not latency samples. Missing judgments, few samples, and native +`incomplete`, `co_biased`, or `high_drop` caveats constrain the conclusion. Transport success alone +does not prove SSE business completion; inspect the project's Judge and completion signals. +`first_byte_ms` measures first byte, not automatically first token; token timing needs an explicit +protocol fact from the Runner. + +## Confirm capacity and compare curves + +Quality-harness confirms capacity only from complete hold Windows with requests, no drops, +interruptions or missing judgments, and applicable SLOs that all pass. A finite-rate Window with +`limited_s > 0` cannot confirm configured-rate capacity or support resource-curve extrapolation. +An otherwise healthy run without SLOs is `skipped`, not confirmed capacity. A later failure can leave earlier complete +hold evidence usable; retain the run's failure separately. + +For finite-rate sweeps, scan request rate and fix the inflight schedule. For `inf` sweeps, scan the +inflight limit. Fix resources, case mix, target and generator conditions, arrival distribution, +seed, durations, warmup, cooldown limits, and breakers. Stage shape along the scanned axis must be +proportional; matching peaks alone do not make schedules comparable. + +Separate different fixed conditions. In particular, fixed finite rate with varying inflight caps +does not form a rate-capacity curve. Require at least two distinct levels within a comparable group +for a response curve or slope, label the axis and units, and distinguish measured capacity from +extrapolation. Use the native shared comparison grouping when available. + +The source contract is [Perf Harness](https://github.com/compforge/quality-harness/blob/main/spec/perf-contract.md). +Resolve API details and schema support from the version used by the project; this skill does not +authorize an SDK upgrade or a profile conversion. diff --git a/quality/skills/perf/references/load-environment.md b/quality/skills/perf/references/load-environment.md index 1ba07ac..8af71ab 100644 --- a/quality/skills/perf/references/load-environment.md +++ b/quality/skills/perf/references/load-environment.md @@ -1,63 +1,65 @@ # Performance load-environment preparation -Read the [shared Environment contract](../../../references/environment.md) first. This -reference adds the performance-specific resource, load-generator, observation, guardrail, and -cooldown requirements. +Read [environment preparation](../../../references/environment.md) first. This reference adds +performance-specific resource, generator, observation, guardrail, and cooldown preparation. +Interpret scheduling and result semantics through [load and evidence](load-and-evidence.md). ## Establish the load contract Resolve before a live run: -- the selected resource profile and whether the Harness applies it or only records it; -- open- or closed-loop load, stages, maximum arrival rate or concurrency, inflight/request limits, - duration, case mix, and expected durable-data growth; -- load-generator location and capacity; +- the selected resource profile and whether the harness applies it or only records it; +- request rate and maximum inflight count, admission behavior at that limit, arrival distribution, + phase schedule, measurement duration, request/time budgets, case mix, and durable-data growth; +- load-generator location, revision, and capacity; - required model, database, sandbox, queue, storage, or other downstream quotas; -- request and resource observations, clocks, SLOs, breaker and abort conditions; -- deactivation, cooldown, cleanup, and restoration checks. +- request and resource observations, clocks, SLOs, breakers, and abort conditions; +- stop, drain, cancellation, deactivation, cooldown observation, cleanup, and restoration behavior. -Do not infer missing values from an example profile. Project profiles and Target policy are the -authority. Never run material load against production or a shared Target merely because a profile -and credential are available. +Project profiles and target policy are authoritative. Do not fill missing limits from an example +profile. A profile and credentials alone do not authorize material load against a shared or +production target. ## Prepare the performance environment Verify that: -- the subject's resources, replicas, worker floor, autoscaling, and other capacity policy match the - selected profile; -- the load generator has enough CPU, memory, file descriptors, sockets, and network capacity for - the declared arrival rate or concurrency; -- the load path does not contain an unintended proxy, tunnel bottleneck, or local contention; +- target resources, replicas, worker floor, autoscaling, and other capacity policy match the profile; +- the generator has enough CPU, memory, file descriptors, sockets, and network capacity for both + the declared initiation rate and full-lifecycle inflight count; +- the application path does not contain an unintended proxy, tunnel bottleneck, or local contention; - dependencies have known authorized capacity for the experiment; -- request and resource Probes produce fresh samples for the intended components over aligned time; +- request and resource observations produce fresh samples for the intended Service and actual + Workload instances over aligned time; - current errors, restarts, backlog, saturation, scaling state, and retained data are understood. -When a local Runner shares a machine with the Target, record the contention and resource isolation. -When a remote Kubernetes Target is used, separately verify deployed images/resources, application -connectivity, and Kubernetes observation. If these conditions are uncontrolled, the run may -characterize the Runner, connection, or downstream quota rather than the subject. +When the test process shares a machine with the target, record contention and resource isolation. +For Kubernetes, independently verify deployed images/resources, application connectivity, and +resource observation. Uncontrolled conditions may measure the generator, connection, or downstream +quota rather than the subject's capacity. ## Bound and execute load -Prefer a project-declared smoke or lowest load arm before higher-impact arms when that is part of -the canonical experiment. Keep explicit maximum inflight/request/time limits and a project-defined -breaker. Abort when its condition fires; the abort is evidence, not a reason to disable the guard -and retry. +Use a project-declared smoke or lowest Arm before higher-impact Arms when it is part of the canonical +experiment. Keep explicit inflight/request/time limits and a project-owned breaker. Its activation +is evidence to retain, not a reason to disable the guard and retry. -Run the declared lifecycle without changing the Target or profile during measurement: +Run the declared lifecycle with observable readiness, phase boundaries, stop behavior, and recovery: ```text -prepare fixtures → verify steady state → warm up → ramp/hold measurement - → deactivate load → cooldown observation → cleanup → verify restoration +prepare fixtures and verify steady state + → warmup and hold/stage measurement + → stop initiation, drain or cancel and join inflight calls + → deactivate, finish cooldown observation, clean up, verify restoration ``` -Load activation, measurement, SLO judgment, and cleanup remain separate outcomes. Do not shorten a -measurement window, discard missing Probes, or remove a failed arm while retaining the original -experiment identity. +Confirm that requests have exited before releasing their clients or connections. Keep request +completion and optional resource-recovery observation distinct. Preserve the declared profile +throughout measurement; shortened windows, missing Probes, or removed Arms change what ran. ## Retain performance evidence -In addition to the shared Environment evidence, retain the realized resource and load -configuration, case mix, measurement windows, stop/breaker outcome, dependency limitations, -request and resource facts, SLO results, cooldown, and cleanup. +In addition to shared run evidence, retain the realized resource/load configuration, case mix, +actual phase Windows, inflight limits and time spent limited, stop/breaker and interruption census, +dependency limitations, request/resource observations, independent judgments, SLO coverage, +cooldown, and cleanup. diff --git a/quality/skills/trajectory/SKILL.md b/quality/skills/trajectory/SKILL.md index 0c61bd1..63da848 100644 --- a/quality/skills/trajectory/SKILL.md +++ b/quality/skills/trajectory/SKILL.md @@ -9,25 +9,19 @@ Turn an agent's recorded decisions and actions into an evidence-backed next impr project's existing trajectory capability; do not replace its framework, business data, labels, or judgment criteria. -## Keep the owners separate - -1. Treat case-harness or another trajectory framework as infrastructure that quantifies and - localizes trajectory problems, then accumulates their evidence through normalization, Dataset - and Run artifacts, Findings, Evaluations, Measurements, Worksheets, metrics, and reports. -2. Treat the project as owner of its RecordingSource, Loader, annotations, Detectors, Evaluators, - Measurers, report projection, canonical command, and optimization targets. -3. Treat this skill as the operator that discovers those assets, runs them, turns their evidence into - one bounded next problem and change surface, and organizes a controlled tuning experiment. - -The framework answers where a problem appears, with what evidence, magnitude, and trend. This skill -uses project context to decide what to change next and how to verify it. - -Do not call a lower-level framework directly when the project exposes a wrapper. Framework code -alone is `no_capability`; require either a usable project entrypoint that can produce trajectory -evidence or an existing project-owned artifact sufficient for the requested analysis. A missing -specialized Detector or Evaluator alone is not `no_capability`: when normalized trajectories are -available, inspect representative and counterexample rows for an open-ended candidate problem. -Record it as project evidence, not as a fabricated framework result or Verdict. +Read [Quality concepts](../../CONCEPTS.md) before discovery or interpretation. It defines ownership, +subject identity, execution facts, evaluation, comparison, and result states shared by the skills. + +The project owns recording sources, loaders, annotations, Detectors, Verifiers, Measurers, report +projection, and optimization targets. Trajectory and deterministically derived Measurements are +inputs to discovery and verification. A Detector discovers patterns; a Verifier checks an explicit +criterion. Both can concern cost or effect and use hard or soft rules. Map the project's native +component names to these roles without assuming that a specific API or artifact exists. + +Require either a usable project entrypoint that produces trajectory evidence or existing evidence +sufficient for the requested analysis. A missing specialized Detector or Verifier alone does not +make the capability absent: inspect representative and counterexample trajectories for candidate +problems, clearly recording them as analyst observations rather than fabricated framework results. ## Discover the capability @@ -40,7 +34,7 @@ Read the project's AGENTS.md and README first, then locate: version, exposed tool-contract version, loop/compact configuration, model, and orchestration when those can affect behavior; - human or external annotations and their coverage; -- the available Detectors whose `detect` operations return zero or more Findings, Evaluators for +- the available Detectors whose `detect` operations return zero or more Findings, Verifiers for effect or contract judgments, and Measurers for factual values such as cost; - the component specs and configuration actually selected by the requested Run, plus each component's execution status, output, and coverage; @@ -50,9 +44,9 @@ Keep three component states separate: available in the repository, selected for successfully executed with usable output. A definition found in source does not prove that the current Run used it or that it covers the relevant trajectories. -Classify discovery as `ready`, `blocked`, or `no_capability`. Report ambiguity when multiple +Classify the requested action using the shared capability states. Report ambiguity when multiple plausible entrypoints remain; do not guess. Missing labels, history, usage data, a dedicated -Detector or Evaluator, or a verdict policy may limit a comparison without making an otherwise valid +Detector or Verifier, or a verdict policy may limit a comparison without making an otherwise valid analysis run disappear. ## Run and interpret @@ -65,16 +59,15 @@ Interpret artifacts in this order: 1. Check source freshness, build issues, unmatched annotations, execution coverage, measurement coverage, and cohort comparability. -2. Reconcile the Run's selected Detector, Evaluator, and Measurer specs with persisted results. +2. Reconcile the Run's selected Detector, Verifier, and Measurer specs with persisted results. Report components that were configured but did not run, returned errors, were not applicable, or produced partial coverage. 3. Check effect regression across generator versions before subtler smells. Do not confuse the - Dataset version or Evaluator version with the identity of the agent configuration that generated + Dataset version or Verifier version with the identity of the agent configuration that generated a trajectory. -4. Read Findings, Evaluations, and Measurements separately. A Finding is the output of `detect` and - describes a behavior pattern without implying a verdict; an Evaluation is a contract or effect - judgment; a Measurement is a factual value. A `DetectionResult`, when present, is only the - Detector's execution envelope around zero or more Findings. +4. Apply the shared distinction among Findings, Evaluations, and Measurements. Read Verifier + results as criterion judgments and `DetectionResult`, when present, as the execution envelope + around zero or more Findings; neither an envelope nor a discovered pattern is a release gate. 5. Compare both totals and normalized values. Volume can increase total token or time while unit cost improves, and the reverse can also happen. 6. Preserve the project's dimensions. `target` identifies what is evaluated; `category` identifies @@ -108,7 +101,7 @@ reproduced and stated independently of its assumed cause. For an agent change, freeze the same Case/input workload and evaluation semantics, then let each generation configuration produce its own Trajectories. Align the resulting cohorts by stable Case or input identity; do not pretend that changed observations are the same TrajectoryDataset. Reuse one -fixed TrajectoryDataset only when comparing Detectors, Evaluators, Measurers, policies, or report +fixed TrajectoryDataset only when comparing Detectors, Verifiers, Measurers, policies, or report projections. Benchmark a deterministic adapter, result filter, CLI output compressor, or other pure transform by @@ -127,13 +120,12 @@ evaluation assets; implement them only when the user asks for that mutation. ## Report and retain learning Report the evaluated project and revision, Dataset identity, current and comparison generation -provenance and runs, data health, the available and actually selected Detectors, Evaluators, and +provenance and runs, data health, the available and actually selected Detectors, Verifiers, and Measurers with their execution coverage, quantified effect and cost, the exact evidence location, -selected Findings and Evaluator judgments, the next prioritized problem and change surface, +selected Findings and Verifier judgments, the next prioritized problem and change surface, supporting Worksheet examples, unknowns, and artifact paths. Lead with the decision-relevant summary; keep per-trajectory evidence in drill-down artifacts. -Persist run-specific evidence and project-specific lessons beside the project's evaluation assets. -Promote a problem pattern or optimization tactic into this skill's references only after it proves -reusable across projects. Do not copy private samples, volatile weekly numbers, credentials, or -project-only commands into the plugin. +Persist evidence and reusable operating knowledge beside the project's evaluation assets under the +shared provenance and privacy rules. Promote a problem pattern or optimization tactic into this +skill's references only after it proves reusable across projects. diff --git a/quality/skills/trajectory/references/problem-discovery.md b/quality/skills/trajectory/references/problem-discovery.md index 67efe07..96cce24 100644 --- a/quality/skills/trajectory/references/problem-discovery.md +++ b/quality/skills/trajectory/references/problem-discovery.md @@ -31,7 +31,7 @@ Before ranking agent problems, verify: - build failures, unmatched annotations, and unknown execution outcomes are visible; - label coverage and the exact denominator accompany every quality rate; - model-usage and duration coverage are sufficient for cost claims; -- evaluator comparisons reuse the same TrajectoryDataset, while generator comparisons use aligned +- judgment-component comparisons reuse the same TrajectoryDataset, while generator comparisons use aligned Case/input cohorts and explicitly control their differences; - target, model, tool version, feature flags, and other material dimensions are available. @@ -55,7 +55,7 @@ environment, and coverage. A week-over-week decline is an operational alert, not automatically a version regression: workload mix and annotation coverage may have changed. Group by generation provenance within or across the time -windows. Escalate the smell when the same effect metric declines under the same Evaluator and policy, +windows. Escalate the smell when the same effect metric declines under the same Verifier and policy, especially when completion or cost also regresses. ## Search in priority order @@ -67,7 +67,7 @@ especially when completion or cost also regresses. 3. **Action errors:** malformed model output, invalid tool arguments, rejected calls, tool errors, and repair loops, including errors that a later retry hides from the final outcome. 4. **Effect regression within a version:** wrong, unsupported, missed, repeat, low-score, or - low-completion outcomes, using the project's own Evaluators and annotations. + low-completion outcomes, using the project's own Verifiers and annotations. 5. **Pareto regression:** effect is unchanged or worse while normalized token, time, model calls, or tool calls increase. 6. **Cost hotspot:** effect is acceptable, but a high-volume target or cohort dominates avoidable