evals: one trial of the slice component eval failed

[angeloashmore]

Problem

The agent is asked to create the React component for a Testimonial slice. One trial in the latest run failed. The two earlier runs passed all trials.

Eval

File: evals/render-content.eval.ts

• writes a slice component

History

d5ab9bd (#260)	02318e8 (#263)	8a1a04a (#269)
3/3	3/3	2/3

What the command trails show

The failed trial ran for 103 seconds. The agent read the Next.js doc and the slices doc, then ran gen slice-index.

--help
slice --help && echo "---" && npx prismic gen --help
docs list
slice view --help; echo "---"; npx prismic gen slice-index --help
slice create --help
docs view nextjs/with-slice-machine
gen types --help
docs view slices > /tmp/slices_doc.txt 2>&1; wc -l /tmp/slices_doc.txt; grep -n "filename=" /tmp/slices_doc.txt
gen setup --help
gen slice-index 2>&1

The trail does not show which assertion failed. The eval checks for a .tsx or .jsx file in slices/Testimonial/, for the three field names in the file, and for a judge verdict on Prismic conventions. This can be a one-time failure.

Done when

This issue is not fixed until the eval passes 3/3 in a recorded eval run. A partial pass rate is not a fix.

[angeloashmore]

Things to consider, not a decision.

• Get the failed assertion. The results file does not record which check failed. Run the eval with more trials, or save the assertion message in results.json, before any fix.
• Possible causes. The component was written to a different folder, or it used next/image or a raw <img> and the judge rejected it. The trail shows gen slice-index, so the agent did find the slices folder.
• Judge noise. If the next run passes 3/3, close this issue.

[angeloashmore]

Design and implementation plan for #278

Issue: #278 (evals: one trial of the slice component eval failed)

This plan is for the agent that implements the fix. Read all of it before you change code. Do not run the evals outside a container or a disposable VM.

Root cause

The trial failed on the first file assertion, not on the judge. The eval looks for a file that matches /\.(t|j)sx$/ in slices/Testimonial/. The folder existed (the readdir call did not throw), but no file matched. The most likely cause is that the agent wrote slices/Testimonial/index.js. The fixture is a JavaScript Next.js project (no tsconfig.json), Next.js compiles JSX in .js files, and Slice Machine writes index.js for JavaScript projects. The regex rejects that file, so the eval rejects a valid component.

A second cause is a knowledge gap. No CLI text tells the agent where the component file goes or what it is named. The docs say index.tsx for TypeScript only. The failing agent searched the slices doc for filename= to find the convention, found none, and had to guess.

Evidence

Run	Commit	Trial	Result	Failure text
32001185558	d5ab9bd	1, 2, 3	pass
32697021321	02318e8	1, 2, 3	pass
33365224683	8a1a04a	3	fail	AssertionError: expected undefined to be truthy at evals/render-content.eval.ts:21:24 (the expect(componentFile).toBeTruthy() line)

To read the full failure block:

gh run view 33365224683 --repo prismicio/cli --log | sed -E 's/^evals\tUNKNOWN STEP\t[0-9T:.Z-]+ ?//; s/\x1b\[[0-9;]*m//g' | grep -n -A 14 "FAIL .* writes a slice component"

The failure is a plain assertion, so the log has no agent final message and no judge reason. The code at 8a1a04a and 02318e8 is identical except for evals/results.json, so the change in result is trial variance, not a code change.

Command trail of the failed trial (from git show 72cb9a5:evals/results.json):

--help
slice --help && echo "---" && npx prismic gen --help
docs list
slice view --help; echo "---"; npx prismic gen slice-index --help
slice create --help
docs view nextjs/with-slice-machine
gen types --help
docs view slices > /tmp/slices_doc.txt 2>&1; wc -l /tmp/slices_doc.txt; grep -n "filename=" /tmp/slices_doc.txt
gen setup --help
gen slice-index 2>&1

The trial ran for 103 seconds and cost $0.39. The passing trial 1 in the same run ran for 93 seconds and cost $0.37. So the failed trial most likely wrote a file. The write is not in the trail because only prismic commands are recorded.

What the passing trials did that the failed trial did not:

• Two passing trials (d5ab9bd trial 1, 8a1a04a trial 2) did not write the component. They ran push, then deleted the local model and ran pull --force, so that the CLI scaffolded slices/Testimonial/index.jsx itself. The CLI writes .jsx in src/adapters/nextjs.ts (onSliceCreated: index.${extension}x).
• Other passing trials ran gen types first, which creates prismicio-types.d.ts. That file makes the project look like TypeScript, and the agent then writes index.tsx.
• The failed trial ran neither. It saw gen slice-index write slices/index.js in a plain JavaScript project. A .js component is the natural match for that file.

Facts that support the .js hypothesis:

• checkIsTypeScriptProject in src/project.ts looks for tsconfig.json. The fixture in test/it.ts does not create one. So the project is JavaScript.
• gen slice-index writes slices/index.js with import Testimonial from "./Testimonial";. Next.js resolves that import to index.js, index.jsx, index.ts, or index.tsx. A .js component works.
• Slice Machine's Next.js adapter writes index.js for JavaScript projects by default. See packages/adapter-next/src/lib/getJSFileExtension.ts in prismicio/slice-machine: return jsx && options.jsxExtension ? "jsx" : "js"; with jsxExtension default false.
• create-next-app without TypeScript names components .js, not .jsx.

Properties of the current eval and CLI text that cause the failure:

• The eval regex /\.(t|j)sx$/ rejects .js, which is a valid and common component file name in a JavaScript Next.js project.
• The assertion message is expected undefined to be truthy. It does not list the files that were present, so a failure cannot be diagnosed from the CI log.
• No CLI help text or command output names the slice component file. gen slice-index --help says only Generate the slice index file of each slice library. and prints Generated the slice index file. The agent does not learn that the index imports slices/<Name>/index.jsx.

Alternative causes that the evidence does not exclude:

• The agent wrote the component in another folder, for example src/slices/Testimonial/. The docs show src/slices/<SliceName>/index.tsx. This is less likely because the agent ran gen slice-index, which wrote to slices/.
• The agent asked a question and wrote nothing. The duration and cost make this less likely.

The change in the "Changes" section makes both alternatives visible on the next failure. #277 adds the final message to results.json, which covers the third alternative.

Design

The eval must accept every component file that Next.js accepts and that the Prismic tooling produces. The eval must also make a failure diagnosable from the CI log. The CLI must tell the agent where the component file goes, so that the agent does not have to guess.

What stays unchanged:

• The eval prompt: Create the React component for the "Testimonial" slice.
• The three toContain assertions and the judge criterion.
• The .jsx extension that the CLI writes in src/adapters/nextjs.ts. Do not change adapter logic.
• The gen slice-index runtime output Generated the slice index file. and the tests in test/gen-slice-index.test.ts.
• The Prismic skill and the docs site.

Why the eval change is not teaching to the test: the eval measures whether the agent writes a working slice component with Prismic conventions. The file extension is not a Prismic convention. Slice Machine itself writes index.js in JavaScript projects. The regex rejects work that the product accepts, so the regex is wrong.

Why the CLI change is text only: every trail in all three runs reads gen slice-index --help or gen --help. That help is the natural place to say what the index imports and where the component lives. This is a description string, not logic.

Related issues

A coordination pass assigned ownership across the open evals: issues. Follow it.

• evals: agent stops at status and does not push a finished type #277 owns the recording of the agent's final message in results.json. This plan depends on it for the "agent asked a question" alternative and does not plan it again.
• evals: agent stops at status and does not push a finished type #277 also covers the status, push, and pull text. Two passing trials of this eval used push then pull --force to make the CLI scaffold the component. That is a destructive workaround that the evals: agent stops at status and does not push a finished type #277 text may also reduce. No shared change is needed here.
• Docs site. Each issue owns its own page. This plan changes no docs. The docs pages slices and nextjs/with-slice-machine describe Slice Machine paths (src/slices, index.tsx). No open issue owns those two pages. If fallback 4 shows that the agent followed them, open a docs follow-up from evals: one trial of the slice component eval failed #278. evals: agent does not write routes to prismic.config.json #272 owns only the routes page.
• Skill and PRISMIC_SKILL_REF. This plan does not touch them. evals: agent does not write routes to prismic.config.json #272 owns the first-pass skill change and the ref bump.
• evals: agent does not write routes to prismic.config.json #272, evals: agent does not update the simulator URL after a production deploy #273, evals: agent models the author of a blog post as key text #274, evals: agent does not model a title as single-heading rich text #275, evals: agent stops before it runs prismic init #276 do not share a file or a change with this plan.

Changes

1. evals/render-content.eval.ts

Accept .js and print the project files when the assertion fails.

Current lines 18 to 21:

	const dir = new URL("slices/Testimonial/", project);
	const files = await readdir(dir);
	const componentFile = files.find((file) => /\.(t|j)sx$/.test(file));
	expect(componentFile).toBeTruthy();

Target:

	const dir = new URL("slices/Testimonial/", project);
	const files = await readdir(dir);
	const componentFile = files.find((file) => /\.(tsx|jsx?)$/.test(file));
	const projectFiles = await readdir(project, { recursive: true });
	expect(
		componentFile,
		`no component file in slices/Testimonial/, project files:\n${projectFiles.filter((file) => !file.startsWith("node_modules")).join("\n")}`,
	).toBeTruthy();

Notes for the implementer:

• /\.(tsx|jsx?)$/ matches .tsx, .jsx, and .js. It does not match .ts, because a .ts file cannot contain JSX.
• Vitest 4 accepts a message as the second argument of expect. The message prints in the CI log on failure, so the next failure shows where the agent wrote the file.
• readdir(project, { recursive: true }) needs a URL or path. project is a URL. Node 24 supports the recursive option.
• Run node --run format after the edit. Oxfmt decides the final line breaks.
• No other test or eval asserts these lines.

2. src/commands/gen-slice-index.ts

Tell the agent what the index imports and where a component lives.

Current:

const config = {
	name: "prismic gen slice-index",
	description: "Generate the slice index file of each slice library.",
} satisfies CommandConfig;

Target:

const config = {
	name: "prismic gen slice-index",
	description: `
		Generate the slice index file of each slice library.

		The index file imports the component of each slice and exports them as
		\`components\` for <SliceZone>. A slice component lives next to its
		model.json, for example slices/Hero/index.tsx (index.jsx in a JavaScript
		project). The build fails when a slice has no component file.
	`,
} satisfies CommandConfig;

Notes for the implementer:

• createCommand in src/lib/command.ts runs dedent on the description, so the template literal with tabs is correct. src/commands/gen-setup.ts uses the same shape.
• Do not change the one-line description in src/commands/gen.ts (Generate the slice index file of each slice library). The router shows the short form.
• Do not change the runtime output Generated the slice index file. or Generated N slice index files.. test/gen-slice-index.test.ts asserts these strings at lines 17 and 40, and they stay valid.
• No test asserts the help description. No test update is needed.

Reconstructed gen slice-index --help output after the change:

Generate the slice index file of each slice library.

The index file imports the component of each slice and exports them as
`components` for <SliceZone>. A slice component lives next to its
model.json, for example slices/Hero/index.tsx (index.jsx in a JavaScript
project). The build fails when a slice has no component file.

USAGE
  prismic gen slice-index [options]

OPTIONS
  -h, --help   Show help for command

LEARN MORE
  Use `prismic <command> --help` for more information about a command.

Verification

Run these from the repository root, in this order:

node --run build
node --run lint
node --run types
node --run unit -- test/gen-slice-index.test.ts

All four must pass. Then run the target eval file and the counter eval file with five trials:

EVAL_TRIALS=5 PRISMIC_ALLOW_EVALS=true node --run evals -- evals/render-content.eval.ts evals/check-before-acting.eval.ts

This command needs E2E_PRISMIC_EMAIL, E2E_PRISMIC_PASSWORD, and a Claude login or ANTHROPIC_API_KEY. It runs an agent with permissions disabled against a real account. Run it only inside a container or a disposable VM.

Read evals/results.local.json after the run. This is a partial run, so evals/results.json is not touched. Make sure that:

• evals/render-content.eval.ts / writes a slice component has pass: true in 5 of 5 trials.
• evals/render-content.eval.ts / models and wires a landing page end to end is not worse than 3/3 in the recorded evals/results.json (allow one flaky trial, but investigate if two or more fail).
• The three evals in evals/check-before-acting.eval.ts are 5/5. Their recorded baseline is 3/3 each.

If a trial of writes a slice component still fails, read the assertion message in the vitest output. It now lists the project files. Report the file list and the final message (when #277 has landed) in a comment on #278. Do not add more help text or loosen the regex on a guess. Stop and ask.

Done when

• writes a slice component passes 3/3 in a recorded evals workflow run on main.
• models and wires a landing page end to end and the three check-before-acting evals stay at 3/3 in the same run.
• Ask the maintainer before you dispatch a run with gh workflow run evals.yml. The workflow runs weekly on Monday at 06:00 UTC.

Out of scope

Do not touch:

• The eval prompt or the judge criterion.
• src/adapters/nextjs.ts, src/adapters/nextjs.templates.ts, or any adapter logic. The .jsx extension that the CLI writes stays.
• The runtime output of gen slice-index, slice view, or slice create.
• The Prismic skill, PRISMIC_SKILL_REF in evals/it.ts, and the docs site.
• evals/it.ts and evals/reporter.ts. evals: agent stops at status and does not push a finished type #277 owns the recording of the final message.

Fallback if the plan does not get 3/3:

1. Read the new assertion message and the recorded final message. Classify the failure with the same method as this plan.
2. If the agent wrote the file to another folder, the next candidate is the slice view output. Add a Component: slices/Testimonial/index.jsx line (or Component: none) so the agent sees the expected path when it inspects the slice. This is a logic change (a file system check in src/commands/slice-view.ts plus an adapter method for the component path), so propose it in the issue first.
3. If the agent asked a question, quote the question in the issue and design a text change for the gate that made it ask.
4. If the docs are the cause (for example the agent followed src/slices/<SliceName>/index.tsx from docs view nextjs), coordinate with evals: agent does not write routes to prismic.config.json #272, which owns the docs changes.