evals: agent does not model a title as single-heading rich text

[angeloashmore]

Problem

The agent is asked to add a title and a Bluesky handle to a blog_post type. The agent adds the title as key text, or as rich text with no single-heading limit. Prismic convention is rich text limited to one heading.

Eval

File: evals/design-sensible-models.eval.ts

• models a title as single-heading rich text and a social media handle as key text

History

d5ab9bd (#260)	02318e8 (#263)	8a1a04a (#269)
1/3	0/3	1/3

What the command trails show

In five of six recorded failures the agent reads only field add text --help. It does not read field add rich-text --help. Then it adds the title with field add text. One trial adds field add rich-text title with no --single or --allow flag.

Example trail:

--help
type --help && echo ---- && npx prismic field --help
type create --help && echo ---- && npx prismic field add --help
field add text --help
type create "Blog Post" --format page --id blog_post
field add text title --to-type blog_post --label "Title" && npx prismic field add text author_bluesky_handle --to-type blog_post --label "Author Bluesky Handle" --placeholder "e.g. @handle.bsky.social"
type view blog_post

The Bluesky handle passes in every trial. In the blog post eval (models a page type with sensible field types) the same model adds the title with --allow heading1 --single in every trial. The convention is known to the model but is not applied when the prompt only says "title".

Done when

This issue is not fixed until the eval passes 3/3 in a recorded eval run. A partial pass rate is not a fix.

[angeloashmore]

Things to consider, not a decision.

• Help text. In field add text --help say that a title is rich text with a single heading, and point to field add rich-text --single --allow heading1.
• Command output. Print a hint when a text field is named title or heading.
• Skill or doc change. State the convention in the content modeling doc and test with PRISMIC_DOCS_HOST.
• Cross-check. The blog post eval (evals: agent models the author of a blog post as key text #274) uses the convention every time. Compare the two prompts to see what makes the agent read the rich text help there but not here.

[angeloashmore]

Plan for #275: agent does not model a title as single-heading rich text

Eval: evals/design-sensible-models.eval.ts, test name
models a title as single-heading rich text and a social media handle as key text.

Prompt the agent gets: Set up the "blog_post" type: it needs a title and the author's Bluesky handle.

Bar: 3/3 in a recorded evals workflow run. A partial pass rate is not a fix.

1. Root cause

The agent lacked knowledge. The only place in the CLI that states the title convention is the EXAMPLES section of prismic field add rich-text --help. The agent maps the word "title" to the text subcommand, reads prismic field add text --help, and never opens the rich text help. field add --help lists text and rich-text with equal weight and no guidance, and field add text --help says nothing about titles. Both passing trials read field add rich-text --help before they added the title. Every failed trial did not.

No trial asked for permission. No Claude Code rule is involved.

Evidence

Seven failed trials in three recorded runs. Judge reasons are shortened.

Run	Commit	Trial	Read rich-text --help?	Title command	Judge reason
32001185558	d5ab9bd	1	No	field add text title --to-type blog_post --label "Title"	"type": "Text" is key text, not StructuredText
32001185558	d5ab9bd	2	No	field add text title --to-type blog_post --label "Title"	"type": "Text" is key text, not StructuredText
32697021321	02318e8	1	No	field add text title --to-type blog_post --label "Title"	"type": "Text" is key text, not StructuredText
32697021321	02318e8	2	No	field add text title --to-type blog_post --label "Title"	"type": "Text" is key text, not StructuredText
32697021321	02318e8	3	No	field add text title --to-type blog_post --label "Title"	"type": "Text" is key text, not StructuredText
33365224683	8a1a04a	2	No	field add rich-text title --to-type blog_post --label "Title"	StructuredText with multi and all block types, no single limit
33365224683	8a1a04a	3	No	field add text title --to-type blog_post --label "Title"	"type": "Text" is key text, not StructuredText

The two passing trials (d5ab9bd trial 3, 8a1a04a trial 1) both ran field add rich-text --help and then
field add rich-text title --to-type blog_post --allow heading1 --single --label "Title".

The Bluesky handle passed in all nine trials. Only the title judge fails.

To read a full failure block:

gh run view 33365224683 --repo prismicio/cli --log > /tmp/run-33365224683.log
sed -E 's/^evals\tUNKNOWN STEP\t[0-9T:.Z-]+ ?//; s/\x1b\[[0-9;]*m//g' /tmp/run-33365224683.log | grep -n -A 3 "FAIL .*single-heading"

To read the command trails. Note the offset: the results file for a run lives in the next recording commit.

# run at 8a1a04a
git show 72cb9a5:evals/results.json | jq '.evals["evals/design-sensible-models.eval.ts / models a title as single-heading rich text and a social media handle as key text"][] | {pass, calls}'
# run at 02318e8
git show 8a1a04a:evals/results.json | jq '.evals["evals/design-sensible-models.eval.ts / models a title as single-heading rich text and a social media handle as key text"][] | {pass, calls}'
# run at d5ab9bd
git show 1c8fd5f:evals/results.json | jq '.evals["evals/design-sensible-models.eval.ts / models a title as single-heading rich text and a social media handle as key text"][] | {pass, calls}'

Comparison with the blog post eval (#274)

The eval models a page type with sensible field types in the same file gets the title right in all nine recorded trials. Its prompt names a "body". The word "body" sends the agent to field add rich-text --help, and that help holds the example
prismic field add rich-text title --to-type blog_post --allow heading1 --single. The agent then applies the example to the title. The convention is not in the model prior. It comes from the rich text help. When the prompt has no word that points to rich text, the agent does not read that help.

Properties of the current text that cause the failure

Every failed trail runs both field add --help and field add text --help. These two surfaces are what the agent sees before it decides.

1. prismic field add --help lists rich-text Add a rich text field and text Add a text field as peers. Nothing says which one holds a title.
2. prismic field add text --help says only Add a text field to a slice or custom type. It does not say what key text is for, and it does not say what it is not for.
3. prismic field add rich-text --help says only Add a rich text field to a slice or custom type. in its description. The title guidance is in an EXAMPLES section at the bottom. The agent never gets there when it does not open this help.
4. The public docs push the same way. prismic docs view content-modeling shows npx prismic field add text title --to-type page, and prismic docs view fields/text shows npx prismic field add text title --to-slice MySlice. No trail in this eval read the docs, so this is not a cause of the recorded failures. It is a related problem (see Out of scope).

Precedent in the same router: the content-relationship and link entries already carry a rule and a pointer:
Add a content relationship field for fetching data from related documents (not for navigation -- use link).

2. Design

Principle: put the rule where the agent already reads. The agent reads field add --help and field add text --help in every trial. Both must say that a title is rich text with one heading block, and both must give the reason and the exact command. Then the agent does not need to open field add rich-text --help to find the convention.

What stays unchanged:

• The eval prompt and the judge criteria.
• All command logic. No new options, no new output at runtime.
• The EXAMPLES section of field add rich-text --help.
• The USAGE lines. The existing help tests assert them.

What the text must do:

• Name key text for what it is: a short plain string with no formatting, such as a name, a username, or a label.
• State that a title or a heading is not key text.
• Give the reason in one sentence.
• Give the exact command for a title. The line must paste as valid shell.
• Keep the router entries short. They are one table row each.

Reuse this command line verbatim wherever the plan shows it:
prismic field add rich-text title --to-type blog_post --allow heading1 --single

Do not name Bluesky or a social media handle in the help text. The handle already passes in every trial, and naming it teaches to the test.

3. Changes

3.1 src/commands/field-add.ts

Two router descriptions change. No other line changes.

Current:

		"rich-text": {
			handler: fieldAddRichText,
			description: "Add a rich text field",
		},

Target:

		"rich-text": {
			handler: fieldAddRichText,
			description: "Add a rich text field for titles, headings, and body copy",
		},

Current:

		text: {
			handler: fieldAddText,
			description: "Add a text field",
		},

Target:

		text: {
			handler: fieldAddText,
			description:
				"Add a key text field for short plain strings like names, usernames, and labels (not for titles or headings -- use rich-text)",
		},

Expected rows in prismic field add --help after the change:

  rich-text              Add a rich text field for titles, headings, and body copy
  text                   Add a key text field for short plain strings like names, usernames, and labels (not for titles or headings -- use rich-text)

3.2 src/commands/field-add-text.ts

The description becomes a multi-line template string. buildCommandHelp in src/lib/command.ts runs dedent on it, so the indented template renders flush left. The same pattern is in src/commands/field-add-content-relationship.ts.

Current:

	description: "Add a text field to a slice or custom type.",

Target:

	description: `
		Add a key text field to a slice or custom type.

		Key text holds a short plain string with no formatting, such as a
		name, a username, or a label. Do not use it for a title or a
		heading. Prismic models titles and headings as rich text limited to
		one heading block, so editors get a real heading and cannot add
		extra blocks:

		  prismic field add rich-text title --to-type blog_post --allow heading1 --single
	`,

Expected output of prismic field add text --help after the change:

Add a key text field to a slice or custom type.

Key text holds a short plain string with no formatting, such as a
name, a username, or a label. Do not use it for a title or a
heading. Prismic models titles and headings as rich text limited to
one heading block, so editors get a real heading and cannot add
extra blocks:

  prismic field add rich-text title --to-type blog_post --allow heading1 --single

USAGE
  prismic field add text <id> [options]

ARGUMENTS
  <id>   Field ID (required)

OPTIONS
      --to-slice string      ID of the target slice
      --to-type string       ID of the target content type
      --variation string     Slice variation ID (default: "default")
      --tab string           Content type tab name (default: "Main")
      --label string         Field label
      --placeholder string   Placeholder text
  -h, --help                 Show help for command

LEARN MORE
  Use `prismic <command> --help` for more information about a command.

3.3 src/commands/field-add-rich-text.ts

The description gains one paragraph. The BLOCKS and EXAMPLES sections do not change.

Current:

	description: "Add a rich text field to a slice or custom type.",

Target:

	description: `
		Add a rich text field to a slice or custom type.

		Use rich text for titles, headings, and body copy. For a title or a
		heading, limit the field to one heading block with --single and
		--allow (see EXAMPLES).
	`,

Expected first lines of prismic field add rich-text --help after the change:

Add a rich text field to a slice or custom type.

Use rich text for titles, headings, and body copy. For a title or a
heading, limit the field to one heading block with --single and
--allow (see EXAMPLES).

USAGE
  prismic field add rich-text <id> [options]

3.4 Tests

No test asserts the old description strings. These tests assert only the USAGE lines, which do not change:

• test/field-add.test.ts: toContain("prismic field add <command> [options]")
• test/field-add-text.test.ts: toContain("prismic field add text <id> [options]")
• test/field-add-rich-text.test.ts: toContain("prismic field add rich-text <id> [options]")

Do not add tests. Do not delete tests. All three files must still pass.

4. Verification

Run these from the repo root.

node --run build
node --run lint
node --run types
node --run unit -- test/field-add.test.ts test/field-add-text.test.ts test/field-add-rich-text.test.ts

Compare the rendered help with the expected blocks in section 3:

PRISMIC_TELEMETRY_ENABLED=false PRISMIC_SENTRY_ENABLED=false node dist/index.mjs field add --help
PRISMIC_TELEMETRY_ENABLED=false PRISMIC_SENTRY_ENABLED=false node dist/index.mjs field add text --help
PRISMIC_TELEMETRY_ENABLED=false PRISMIC_SENTRY_ENABLED=false node dist/index.mjs field add rich-text --help

Then run the target eval with the counter evals. This command runs an agent with all permissions skipped against a real Prismic account. Run it only inside a container or a disposable VM. It needs E2E_PRISMIC_EMAIL, E2E_PRISMIC_PASSWORD, and a Claude login or ANTHROPIC_API_KEY.

EVAL_TRIALS=5 PRISMIC_ALLOW_EVALS=true node --run evals -- evals/design-sensible-models.eval.ts evals/edit-models-precisely.eval.ts evals/check-before-acting.eval.ts

Why these counter evals:

• evals/design-sensible-models.eval.ts holds the target. It also holds models a testimonial slice from a text description, where the judge accepts key text for the author name and role, and models a page type with sensible field types (evals: agent models the author of a blog post as key text #274). The new text must not push names into rich text.
• evals/edit-models-precisely.eval.ts holds adds a field (an excerpt rich text field), is idempotent when the same task runs twice (a body rich text field), and uses the documented config for a single-heading constraint. The new rich text description must not change how the agent uses --single and --allow.
• evals/check-before-acting.eval.ts rewards the agent when it asks before an ambiguous or destructive action. The new text must not add a reason to stop and ask.

The filtered run writes evals/results.local.json and leaves evals/results.json untouched. Read it with:

jq -r '.evals | to_entries[] | "\([.value[]|select(.pass)]|length)/\(.value|length)  \(.key)"' evals/results.local.json

What must hold:

• The target eval is 5/5.
• No eval in the two counter files is lower than its value in evals/results.json at HEAD. Read the current values with node --run evals:report.

If the target eval is not 5/5, stop. Post the trails and judge reasons from results.local.json in #275. Do not add more text on a guess. #277 plans to record the agent's final message as text in results.json. Do not plan that again here.

5. Done when

• The eval models a title as single-heading rich text and a social media handle as key text is 3/3 in a recorded evals workflow run on main.
• The evals in evals/edit-models-precisely.eval.ts and evals/check-before-acting.eval.ts are not lower than in the run before.
• Ask the maintainer before you dispatch the workflow with gh workflow run evals.yml. Do not rerun or dispatch on your own.

6. Out of scope

Do not touch:

• The eval prompt, the fixture, or the judge criteria in evals/design-sensible-models.eval.ts.
• Command logic in src/commands/field-add-text.ts or src/commands/field-add-rich-text.ts. No new options. No runtime hints.
• The Prismic skill (prismicio/skills) and PRISMIC_SKILL_REF in evals/it.ts.
• The docs site.
• The EXAMPLES section of field add rich-text --help.

Fallbacks, in order, if the plan does not get to 3/3:

1. Print a hint after field add text when the field id matches /title|heading/i, with the same verbatim command. This is runtime output, so it comes after help text in the fix priority. It also lands after the wrong action, and the agent must then remove and re-add the field. Report the trails first and get a go from the maintainer.
2. Add a modeling rule to the Prismic skill, then bump PRISMIC_SKILL_REF. This is the second priority in the fix order and needs a change in prismicio/skills.

Separate follow-up, not part of this fix: the docs pages content-modeling and fields/text use npx prismic field add text title as their example command. This teaches the opposite of the convention. It did not cause the recorded failures, because no trail read the docs. Open a docs issue to change the example field to a plain string, for example author_name. Test with PRISMIC_DOCS_HOST only if a later eval shows the agent reads these pages.

Side observation, not part of this fix: type create with an --id that already exists overwrites the model file with no error (src/adapters/index.ts, createCustomType). Most trails in this eval ran type create "Blog Post" --format page --id blog_post and replaced the fixture type. This did not change the title result. It is tracked as a separate bug in #279. Do not fix it in this change.

Related issues

A coordination pass assigned ownership across the open evals: issues. Follow it.

• evals: agent models the author of a blog post as key text #274 (models a page type with sensible field types, same file). Different cause: the agent does not run type list, so it does not know that an author type exists, and it models the author as key text. evals: agent models the author of a blog post as key text #274 changes src/commands/type-create.ts only. evals: agent does not model a title as single-heading rich text #275 owns src/commands/field-add.ts, field-add-text.ts, and field-add-rich-text.ts. evals: agent models the author of a blog post as key text #274 does not touch them, so there is no wording to merge. evals: agent does not model a title as single-heading rich text #275 also owns any runtime hint in field add text keyed on the field id (fallback 1). If evals: agent models the author of a blog post as key text #274 needs a hint for an existing type id as its own fallback, it lands inside the evals: agent does not model a title as single-heading rich text #275 hint change, as one change. One local run of evals/design-sensible-models.eval.ts verifies both issues, so coordinate the run if both fixes are in progress.
• evals: agent does not write routes to prismic.config.json #272 (routes) adds a five-line ROUTES section to type --help. Every trial in this eval reads type --help. This is not a conflict, but note it if the eval behaves differently after evals: agent does not write routes to prismic.config.json #272 lands.
• Skill fallback. evals: agent does not write routes to prismic.config.json #272 owns the only first-pass skill change and the PRISMIC_SKILL_REF bump in evals/it.ts. If this plan needs fallback 2, add the line to the evals: agent does not write routes to prismic.config.json #272 skills PR when that PR is still open, or open a new skills PR after it merges. Only one PRISMIC_SKILL_REF bump is in flight at a time.
• Docs site. Each issue owns its own page. evals: agent does not model a title as single-heading rich text #275 owns the follow-up on content-modeling and fields/text. evals: agent does not write routes to prismic.config.json #272 owns routes. evals: agent does not update the simulator URL after a production deploy #273 owns previews. evals: agent stops before it runs prismic init #276 owns the nextjs page.
• evals: agent does not update the simulator URL after a production deploy #273 (simulator URL), evals: agent stops before it runs prismic init #276 (init stops), evals: one trial of the slice component eval failed #278 (slice component): different commands and files. No overlap.
• evals: agent stops at status and does not push a finished type #277 (final message recording): referenced in section 4. Not planned again here.