Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,13 @@ built-harness Chrome regression, every public edit operation, and generated
media/audio rendering. Real-GPU effects remain an explicit operator/release
gate because hosted PR CI does not provide a portable WebGPU adapter.

### QA Gates

Every PR to `staging` runs the full QA gate set (head binding, static checks,
provenance, consumer smoke, browser QA, redaction) and posts a canonical QA
report; a PR stays DRAFT until a trusted judge returns PASS. See
[docs/qa/README.md](docs/qa/README.md).

### Performance Checks

- `npm run dev` is best for correctness and iteration, but includes React/Vite dev overhead, HMR, and debug instrumentation.
Expand Down
97 changes: 97 additions & 0 deletions docs/qa/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# FreeCut QA Gates

FreeCut's QA gate set mirrors the CodePress delivery gates. Every PR to
`staging` must run the full gate set at its **exact head** and post a
[canonical QA report](./canonical-report.md) as a PR comment. A PR stays
**DRAFT** until a trusted judge returns PASS — no self-merge, no report
comment or evaluator may mark a PR ready.

## Docs-first rule

Before touching any area, read its doc: `docs/` for packaging/provenance and
render decomposition, `packages/freecut-editor/README.md` for the published
editor surface, `headless/` sources for the browser harness, and this
directory for the QA gates themselves. Cross-feature couplings that break
silently are recorded in [verification-graph.md](./verification-graph.md) —
check your blast radius there before and after a change.

## The gate set

Run everything at the PR head, from a clean tree, in this order:

| # | Gate | Command | Notes |
|---|------|---------|-------|
| 1 | Head binding | `npm run qa:binding -- --check` | Emits/validates the exact base/head + dirty-tree block for the report |
| 2 | Type check | `npm run check` | Focused: `npx vp check --no-fmt <paths>` |
| 3 | Lint | `npm run lint` | Focused: `npx vp lint <paths>`. Needs `packages/freecut-editor/dist` — run `npm run build:editor-surface` first on a fresh tree, else a pre-existing TS2307 on the self-referencing `@quantfive/freecut-editor-surface` import (present at base too) |
| 4 | Unit tests | `npm run test:run` | Selective: `npx vp test run <file>` |
| 5 | Build | `npm run build` | |
| 6 | Feature boundaries | `npm run check:boundaries` | |
| 7 | Deps contract boundaries | `npm run check:deps-contracts` | |
| 8 | Legacy lib imports | `npm run check:legacy-lib-imports` | |
| 9 | Deps wrapper health | `npm run check:deps-wrapper-health` | |
| 10 | Unused exports | `npm run check:unused-exports` | Allowlists are ratchet baselines — never bulk-fix |
| 11 | Unused class members | `npm run check:unused-class-members` | Same ratchet rule |
| 12 | Changed-health | `npm run check:changed-health` | |
| 13 | Edge budgets | `npm run check:edge-budgets` | |
| 14 | Provenance/reproducibility | `npm run verify:provenance` | Verifies baseline manifest, license/notices, dependency + asset inventory SHA256s. `npm run package:reproducible` additionally rebuilds and writes a deterministic tarball (never publish from QA) |
| 15 | Editor-surface package build | `npm run build:editor-surface` | |
| 16 | Installed consumer smoke | `npm run test:editor-surface:consumer` | Packs `@quantfive/freecut-editor-surface` into a tarball, installs it into a fresh temp consumer project, and runs the consumer test — no publish |
| 17 | Headless contract tests (Node) | `npm run headless:test:node` | |
| 18 | Browser QA | `npm run qa:browser -- --skip-build` | Discovers a browser session (system Chrome, then Playwright chromium), renders a frame, writes screenshot/log artifacts + manifest to `artifacts/qa/`. Exit 3 = BLOCKED (no browser) — see below |
| 19 | Full headless browser suite | `npm run headless:test:chrome` | Render/edit/frame/layout contract checks in a real browser |
| 20 | Redaction | `npm run check:qa-redaction` | QA docs + artifacts must carry no secrets, tokens, absolute local paths, or embedded media bytes; the command also runs the fail-closed self-test (`scripts/qa-redaction-check.test.mjs`) proving the gate rejects the negative fixtures |

`npm run verify` aggregates gates 2 and 4–13 (type check, unit tests, build,
and all boundary/deps/fallow/edge gates) plus the portable headless suite
(`headless:test:portable` = gates 17 + 19 plus the media harness tests). Lint
(gate 3) and the remaining gates run on demand.

## Browser QA and the BLOCKED rule

Gates 18–19 need a real browser. Discovery order: system Chrome
(`channel: 'chrome'`), then the Playwright-bundled chromium. If neither
exists, `qa:browser` exits **3** and prints a BLOCKED notice — that is a hard
environment blocker, not a failure of the code, but it also means **no
browser evidence exists**. A QA report must record the gate as `⚠️ BLOCKED`
with the exact reason; it must never claim browser evidence that was not
produced, and a missing browser session never waives the gate — it blocks
readiness until re-run in an environment that has one. GPU tuning goes
through `FREECUT_CHROME_ARGS` / `FREECUT_CHROME_ARGS_REPLACE` (see
`headless/lib/cli.mjs`).

## Baseline failures are disclosed, never hidden

If a gate fails on `staging` before your change (an inherited baseline
failure — e.g. full-suite jsdom/localStorage failures), record it as FAIL in
the report with the note `inherited from base <sha>` and reproduce it at the
base SHA. Never edit allowlists, skip lists, or config to make an inherited
failure disappear.

## Draft-to-ready discipline

1. Open the PR as **DRAFT** at the exact head you verified.
2. Run the full gate set at that head from a clean tree.
3. Post the canonical QA report (format: [canonical-report.md](./canonical-report.md))
as a PR comment, including the revision-binding block from `qa:binding`.
4. Any new commit re-arms every gate: re-run and re-post.
5. Only a **trusted judge** PASS transitions the PR out of draft. Verdicts
are binary and fail closed. The verifier's strongest self-verdict is
`PASS — PENDING JUDGE`; only the judge may say `READY TO MERGE`.

## Privacy

QA artifacts and reports contain no secrets, tokens, cookies, raw absolute
local paths (use repo-relative paths), or embedded media bytes. Screenshots
and logs live in gitignored `artifacts/qa/` and are referenced by
repo-relative path or uploaded to a durable URL before being cited.
`npm run check:qa-redaction` enforces this over `docs/qa` and `artifacts/qa`:
a generic absolute-path pattern (any Unix root or Windows drive, not a list of
known roots), any media data URI regardless of payload length, and the secret
patterns. Browser logs are sanitized with the same generic path pattern before
being written. Intentional examples in docs may carry the per-line marker
`qa-redaction:allow` (secrets are never allowlisted). The gate is proven
fail-closed by `scripts/qa-redaction-check.test.mjs`: the negative fixtures in
`scripts/fixtures/qa-redaction/` (a tmp path, a generic absolute path, a short
media data URI, a Windows path) must each fail the gate, and the clean fixture
set must pass.
52 changes: 52 additions & 0 deletions docs/qa/canonical-report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Canonical QA Report

Posted as a PR comment once the full gate set has run at the exact head.
Verdicts per gate: `✅ PASS`, `❌ FAIL`, `⚠️ BLOCKED` (BLOCKED is reserved for
hard environment blockers — missing browser session, missing credentials,
outage; anything else is FAIL).

Template:

```
## Canonical QA Report

### Verifier revision binding
base: <40-hex> (origin/staging)
head: <40-hex>
git rev-parse HEAD -> <40-hex>
git status --porcelain -> (empty)

### Gate results
| # | Gate | Result | Evidence |
|---|------|--------|----------|
| 1 | Head binding | ✅ PASS | clean tree, head descends from base |
| ... | ... | ... | one line each: command + key output |

### Inherited baseline failures
- <gate>: FAIL at base <sha> with <one-line reason> — reproduced at base, not caused by this PR

### Environment blockers
- <gate>: BLOCKED — <exact reason and what re-run requires>

### Visual / browser artifacts
- artifacts/qa/browser-<short-sha>/manifest.json (+ frame.png, harness.png, console.log)

### Overall
✅ PASS — PENDING JUDGE (or: ❌ FAIL — NOT READY TO MERGE)

No merge until trusted judge PASS.
```

Rules:

- The revision-binding block comes verbatim from `npm run qa:binding -- --check`.
A dirty tree forces FAIL/BLOCKED; there are no exemptions.
- Evidence lines are bounded (one line per gate) and redacted — no tokens,
no absolute local paths, no media bytes (`npm run check:qa-redaction`).
- The verifier never writes `READY TO MERGE`. The strongest self-verdict is
`PASS — PENDING JUDGE`. The judge's verdict is binary and fails closed:
`READY TO MERGE` (PASS) or `NOT READY TO MERGE` (FAIL).
- The report is valid only for the exact head in the binding block. Any new
commit re-arms the gate set; the report must be regenerated and re-posted.
- Inherited baseline failures are disclosed in their own section and
reproduced at the base SHA — never hidden by allowlist or config edits.
57 changes: 57 additions & 0 deletions docs/qa/verification-graph.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# FreeCut verification graph

Cross-feature couplings that break **silently** — i.e. not caught by the
typechecker or imports alone. Format per edge:
`<source> -> <target> | mechanism | how it breaks | check that catches it`.
Blast radius of an area: grep its name below. When a change touches any edge,
run the named check at the PR head.

## Edges

- `packages/freecut-editor/src -> src/features/editor/host` | the published
package re-exports the real host surface (`FreeCutEditorSurface`,
`EditorHostProvider`, `EditorHost` contract); a signature drift compiles in
the repo but breaks installed consumers | `npm run test:editor-surface:consumer`
- `packages/freecut-editor package.json exports -> vite.editor-package.config.ts`
| `main`/`module`/`types`/`./style.css` point at `dist/` names produced by
the package build; renaming an output breaks consumers without any in-repo
error | `npm run build:editor-surface && npm run test:editor-surface:consumer`
- `standalone app -> host surface` | the same editor surface runs with the
standalone bootstrap (router, workspace, local project storage) that the
package deliberately excludes; a surface change that silently depends on
the bootstrap breaks the package, and vice versa | `npm run build` +
`npm run build:editor-surface` + consumer smoke
- `src/headless (window.freecut API) -> headless/*.mjs harness` | the harness
waits on `window.freecut.ready` and calls `renderTimeline`, `renderProject`,
`renderFrame`, `dumpLayout`, `editProject`, `probeMedia` by name; a rename
or readiness-change times out with no compile error | `npm run headless:test:chrome`,
`npm run qa:browser`
- `codec capability override -> headless tests` | tests force codec fallback
via the `globalThis.__freecutSupportedCodecsOverride` seam; removing or
renaming the seam makes fallback coverage silently untestable |
`node headless/test.mjs`
- `project schemaVersion -> project JSON fixtures` | saved projects, headless
fixtures, and the consumer smoke fixture carry `schemaVersion`; a migration
that changes semantics without bumping deserializes stale projects without
error | `npm run test:run` + `node headless/test.mjs`
- `provenance/freecut-baseline.json -> package.json + package-lock.json +
LICENSE + notices + tracked assets` | the manifest pins SHA256s of the
upstream revision, dependency inventory, license/notice files, and asset
roots; any drift fails only when the provenance gate runs |
`npm run verify:provenance`
- `packages/freecut-editor consumer-smoke fixtures -> vite.editor-package.test.config.ts`
| `scripts/test-editor-surface-consumer.mjs` copies the smoke
test/setup/style-declaration plus the test vite config into a temp consumer;
renaming any of those files breaks the smoke with a missing-file error only
at run time | `npm run test:editor-surface:consumer`
- `headless/server.mjs harness contract -> headless tests + qa:browser` |
`createHarnessServer({ distDir, resolveMedia })` returns
`harnessUrl`/`mediaUrl(id)`; tests and the browser QA script both consume
that shape | `npm run headless:test:node` + `npm run qa:browser`
- `fallow allowlists (scripts/*.allowlist.json) -> unused-export/member gates`
| the allowlists are ratchet baselines; editing them to silence a new
finding hides real dead code | `npm run check:unused-exports`,
`npm run check:unused-class-members`
- `feature-edge budgets -> feature boundaries` | `check:edge-budgets` caps
cross-feature import edges; growing an edge count past the budget fails
only when the budget gate runs | `npm run check:edge-budgets`
11 changes: 7 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@
"package:reproducible": "node scripts/package-reproducible.mjs",
"package:editor-surface": "node scripts/package-editor-surface.mjs",
"build:perf": "vp build --mode perf",
"lint": "vp lint src packages/freecut-editor/src packages/freecut-editor/consumer-smoke.test.tsx packages/freecut-editor/consumer-smoke.setup.ts packages/freecut-editor/consumer-smoke-style.d.ts scripts/package-editor-surface.mjs scripts/test-editor-surface-consumer.mjs headless vite.config.ts vite.editor-package.config.ts vite.editor-package.test.config.ts",
"lint:fix": "vp lint src packages/freecut-editor/src packages/freecut-editor/consumer-smoke.test.tsx packages/freecut-editor/consumer-smoke.setup.ts packages/freecut-editor/consumer-smoke-style.d.ts scripts/package-editor-surface.mjs scripts/test-editor-surface-consumer.mjs headless vite.config.ts vite.editor-package.config.ts vite.editor-package.test.config.ts --fix",
"format": "vp fmt src packages/freecut-editor/src packages/freecut-editor/consumer-smoke.test.tsx packages/freecut-editor/consumer-smoke.setup.ts packages/freecut-editor/consumer-smoke-style.d.ts scripts/package-editor-surface.mjs scripts/test-editor-surface-consumer.mjs headless vite.config.ts vite.editor-package.config.ts vite.editor-package.test.config.ts package.json packages/freecut-editor/package.json .oxlintrc.json .oxfmtrc.json",
"format:check": "vp fmt src packages/freecut-editor/src packages/freecut-editor/consumer-smoke.test.tsx packages/freecut-editor/consumer-smoke.setup.ts packages/freecut-editor/consumer-smoke-style.d.ts scripts/package-editor-surface.mjs scripts/test-editor-surface-consumer.mjs headless vite.config.ts vite.editor-package.config.ts vite.editor-package.test.config.ts package.json packages/freecut-editor/package.json .oxlintrc.json .oxfmtrc.json --check",
"lint": "vp lint src packages/freecut-editor/src packages/freecut-editor/consumer-smoke.test.tsx packages/freecut-editor/consumer-smoke.setup.ts packages/freecut-editor/consumer-smoke-style.d.ts scripts/package-editor-surface.mjs scripts/test-editor-surface-consumer.mjs scripts/qa-head-binding.mjs scripts/qa-browser-check.mjs scripts/qa-redaction-check.mjs scripts/qa-redaction-check.test.mjs headless vite.config.ts vite.editor-package.config.ts vite.editor-package.test.config.ts",
"lint:fix": "vp lint src packages/freecut-editor/src packages/freecut-editor/consumer-smoke.test.tsx packages/freecut-editor/consumer-smoke.setup.ts packages/freecut-editor/consumer-smoke-style.d.ts scripts/package-editor-surface.mjs scripts/test-editor-surface-consumer.mjs scripts/qa-head-binding.mjs scripts/qa-browser-check.mjs scripts/qa-redaction-check.mjs scripts/qa-redaction-check.test.mjs headless vite.config.ts vite.editor-package.config.ts vite.editor-package.test.config.ts --fix",
"format": "vp fmt src packages/freecut-editor/src packages/freecut-editor/consumer-smoke.test.tsx packages/freecut-editor/consumer-smoke.setup.ts packages/freecut-editor/consumer-smoke-style.d.ts scripts/package-editor-surface.mjs scripts/test-editor-surface-consumer.mjs scripts/qa-head-binding.mjs scripts/qa-browser-check.mjs scripts/qa-redaction-check.mjs scripts/qa-redaction-check.test.mjs headless vite.config.ts vite.editor-package.config.ts vite.editor-package.test.config.ts package.json packages/freecut-editor/package.json .oxlintrc.json .oxfmtrc.json",
"format:check": "vp fmt src packages/freecut-editor/src packages/freecut-editor/consumer-smoke.test.tsx packages/freecut-editor/consumer-smoke.setup.ts packages/freecut-editor/consumer-smoke-style.d.ts scripts/package-editor-surface.mjs scripts/test-editor-surface-consumer.mjs scripts/qa-head-binding.mjs scripts/qa-browser-check.mjs scripts/qa-redaction-check.mjs scripts/qa-redaction-check.test.mjs headless vite.config.ts vite.editor-package.config.ts vite.editor-package.test.config.ts package.json packages/freecut-editor/package.json .oxlintrc.json .oxfmtrc.json --check",
"check:boundaries": "node scripts/check-feature-boundaries.mjs",
"check:deps-contracts": "node scripts/check-deps-contract-boundaries.mjs",
"check:legacy-lib-imports": "node scripts/check-legacy-lib-imports.mjs",
Expand All @@ -27,6 +27,9 @@
"check:changed-health": "node scripts/check-fallow-changed-health.mjs",
"report:deps-wrapper-health:json": "node scripts/check-deps-wrapper-health.mjs --json",
"check:edge-budgets": "node scripts/check-feature-edge-budgets.mjs",
"qa:binding": "node scripts/qa-head-binding.mjs",
"qa:browser": "node scripts/qa-browser-check.mjs",
"check:qa-redaction": "node scripts/qa-redaction-check.mjs && node --test scripts/qa-redaction-check.test.mjs",
"report:feature-edges": "node scripts/report-feature-edges.mjs",
"report:feature-edges:json": "node scripts/report-feature-edges.mjs --json",
"check": "vp check --no-fmt src packages/freecut-editor/src headless vite.config.ts vite.editor-package.config.ts vite.editor-package.test.config.ts",
Expand Down
2 changes: 1 addition & 1 deletion provenance/dependency-inventory.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"generatedFrom": "package.json",
"packageName": "freecut",
"packageVersion": "0.0.0",
"packageJsonSha256": "331cb27e2fa2e50dbf9759e625e6d2b9c97db2205ee5870c8157ffbb2b14f971",
"packageJsonSha256": "d585b800f4a701f45a5415e588c282fbb9067cbebc78316add16397052a6a493",
"lockfile": {
"path": "package-lock.json",
"lockfileVersion": 3,
Expand Down
11 changes: 2 additions & 9 deletions provenance/freecut-baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
],
"dependencies": {
"packageJson": "package.json",
"packageJsonSha256": "331cb27e2fa2e50dbf9759e625e6d2b9c97db2205ee5870c8157ffbb2b14f971",
"packageJsonSha256": "d585b800f4a701f45a5415e588c282fbb9067cbebc78316add16397052a6a493",
"lockfile": "package-lock.json",
"lockfileVersion": 3,
"lockfileSha256": "b4a86741ce7891da1f63df01b6fdd4ed507e8887d5097c6a0f93fc2ea6f3420e",
Expand Down Expand Up @@ -79,14 +79,7 @@
"packageCommand": "npm run package:reproducible",
"artifactPattern": "artifacts/freecut-<source-revision>.tar.gz",
"archiveFormat": "deterministic tar.gz with sorted paths, zeroed mtimes, and uid/gid 0",
"contents": [
"dist/",
"LICENSE",
"notices/",
"package.json",
"package-lock.json",
"provenance/"
]
"contents": ["dist/", "LICENSE", "notices/", "package.json", "package-lock.json", "provenance/"]
},
"ciVerification": {
"workflow": ".github/workflows/reproducible-package.yml",
Expand Down
11 changes: 11 additions & 0 deletions scripts/fixtures/qa-redaction/clean/example.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Clean fixture

QA artifacts are referenced by repo-relative path only:
`artifacts/qa/browser-<short-sha>/manifest.json` and `docs/qa/README.md`.

URLs are not local paths: https://example.com/qa/artifacts/manifest.json is fine.
Single-segment URL paths like /headless.html are fine too.

Intentional examples can be allowlisted per line. The next line demonstrates a
path example that a doc may legitimately show:
Run the checker against /tmp/example-output/ to see a finding. qa-redaction:allow
1 change: 1 addition & 0 deletions scripts/fixtures/qa-redaction/negative-generic-path.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Pipeline config loaded from /opt/acme/pipeline/config.yaml at startup.
1 change: 1 addition & 0 deletions scripts/fixtures/qa-redaction/negative-short-data-uri.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Inline icon: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///
1 change: 1 addition & 0 deletions scripts/fixtures/qa-redaction/negative-tmp-path.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
QA scratch log was written to /tmp/freecut-qa-run-9f2c/output.log before cleanup.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Build output at C:/Users/builder/freecut/dist/bundle.js was archived.
1 change: 1 addition & 0 deletions scripts/fixtures/qa-redaction/negative-windows-path.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Build output at C:\Users\builder\freecut\dist\bundle.js was archived.
Loading
Loading