Skip to content

fix(evals): correct skia reference apps and clip-rect-and-path spec - #388

Open
mikolajadamowicz wants to merge 1 commit into
callstackincubator:mainfrom
mikolajadamowicz:skia-fix
Open

mikolajadamowicz wants to merge 1 commit into
callstackincubator:mainfrom
mikolajadamowicz:skia-fix

Conversation

@mikolajadamowicz

@mikolajadamowicz mikolajadamowicz commented Sep 3, 2026 •

Copy link
Copy Markdown

Summary

The skia eval category was never typechecked against the real @shopify/react-native-skia package, since it was never added as a dependency anywhere in the repo. Once added, tsc found four reference solutions that call APIs that don't exist or are called wrong. This PR adds the dependency and fixes all four.

What changed

  • testbench/package.json: added @shopify/react-native-skia@^2.11.2 as a dependency, so the skia evals can actually be typechecked and run against the real API surface.
  • Eval 16 (clip-rect-and-path): the prompt, requirements, and reference solution were built around ClipRect and ClipPath components. Neither exists in @shopify/react-native-skia. Clipping is done through the clip prop on Group (<Group clip={rect(...)}> or <Group clip={somePath}>). Rewrote the reference implementation and updated the prompt and requirements to describe the real API.
  • Eval 01 (canvas-fill-background) and eval 08 (text-rendering): both destructured const { width, height } = useCanvasSize(), but useCanvasSize() returns { ref, size: SkSize }, not flat width/height. Fixed the destructuring in both reference files.
  • Eval 24 (picture-save-restore): called canvas.rotate(degrees) with one argument. rotate takes a required pivot point: rotate(rotationInDegrees, rx, ry). Fixed both call sites to pass (deg, 0, 0).
  • evals/skia/README.md: rule 13 in the best-practice inventory also described the fake ClipRect/ClipPath API, and the API coverage table separately listed Group clip / invertClip as "omitted" even though it's exactly what eval 16 now tests. Fixed both.

Why

The skia category was added in #377 with 20 evals and no dependency ever installed to check them against, so the LLM judge scored eval 16 at 100% despite it referencing components that don't exist. Adding the dependency and running tsc -p evals/tsconfig.json surfaced this immediately, along with three more reference files with real bugs that had been shipping silently.

Verification

bun install
bun tsc -p evals/tsconfig.json --noEmit

Clean except for two pre-existing, intentional exclusions unrelated to this PR (async-state/12, expo-router/01), both of which typecheck failures are the deliberate point of those evals.

Also ran bunx prettier --check on all touched files.

Not included

No CI wiring. tsc -p evals/tsconfig.json still isn't run automatically anywhere (only bun run typecheck, which excludes evals/, runs in CI), so a similar bug in this category could ship again without someone running it by hand. Working on a follow-up PR that should apply tsconfig to every eval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant