feat(agents): Move agent-template intelligence from the card prompt into the build-an-agent skill - #5188
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR defines and implements an agent-template playbook system, including authoring guidance, 28 generated playbooks, build-an-agent routing, frontend catalog expansion, overlay-aware onboarding, registry validation, and workflow-version normalization. ChangesAgent template playbook delivery
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Home-page template cards seed prompts the builder agent cannot execute. This workspace plans the fix: per-use-case playbooks shipped as build-an-agent skill references, short card seeds, a 28-template inventory for the beachhead personas, a playbook-writing repo skill, and Sonnet verification. Six open questions gate the frontend and scope decisions. Claude-Session: https://claude.ai/code/session_016MYos6VaKu8AjhcLFkaMUk
18ed2ae to
7aa13cc
Compare
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 8
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: aae5ffc9-831b-4ac0-866c-72b329e3cf34
📒 Files selected for processing (9)
docs/design/agent-workflows/projects/agent-templates/README.mddocs/design/agent-workflows/projects/agent-templates/context.mddocs/design/agent-workflows/projects/agent-templates/exemplar.mddocs/design/agent-workflows/projects/agent-templates/open-questions.mddocs/design/agent-workflows/projects/agent-templates/plan.mddocs/design/agent-workflows/projects/agent-templates/playbook-spec.mddocs/design/agent-workflows/projects/agent-templates/research.mddocs/design/agent-workflows/projects/agent-templates/status.mddocs/design/agent-workflows/projects/agent-templates/template-inventory.md
| - **Measure first.** Ship with five categories, watch which tabs users click, split to six | ||
| later. Lowest risk, slowest to the full grouping. | ||
|
|
||
| **Recommendation.** Measure first: ship five categories by folding Monitoring into Engineering |
…d home catalog Template cards seeded one-line prompts the builder agent could not execute: it had no playbook for the use case, so it asked the wrong questions, wired the wrong tools, and stalled. - build-an-agent now routes through references/agent-templates/index.md; 28 per-use-case playbooks ship as generated SkillFiles from the new agent_templates package (one module per category, import-time validation, drift + FE-parity tests). - Home registry grows 6 -> 28 templates across 5 categories with short 'Build a <X> that ...' seeds, a display-only logoSlugs field, PROVIDERS 6 -> 22, and a registry-invariant test. - New write-template-playbooks repo skill encodes the playbook format and the prompting checklist (unignored via scoped .gitignore negations). - Fixes found by live verification: static workflow version 'v1' was rejected by the FE zod schema (killing every __ag__* retrieve), the first builder turn auto-sent before the build-kit overlay resolved, and the newrelic logo slug hit the CDN fallback. Decisions from the PR #5188 review are recorded in the workspace (open-questions.md); the elicitation default field is tracked in #5190. Claude-Session: https://claude.ai/code/session_016MYos6VaKu8AjhcLFkaMUk
mmabrouk
left a comment
There was a problem hiding this comment.
Inline review guide from the implementation session: one comment per load-bearing spot (router, package design, exemplar fidelity, display/required split, categories, the two pre-existing FE bugs fixed, gitignore scoping, drift guard). The PR body carries the verification results and the one pending QA cell (builder smoke, blocked on the dev-box FUSE mount another session is fixing).
| fewest calls and the least time. A simple no-tool ask is two actions: write better | ||
| `instructions.agents_md`, then call `commit_revision`. | ||
|
|
||
| Before anything else, check `references/agent-templates/index.md` for a playbook matching the |
There was a problem hiding this comment.
Review guide: the router. This is the whole change to the skill body: 3 lines inserted after the role statement. It sends the builder to the index before the generic loop and explicitly keeps approval stops. Everything else in the body is untouched; the 29 new reference files are appended to BUILD_AN_AGENT_SKILL.files via build_agent_template_skill_files() further down.
| from ...skills import SkillFile | ||
|
|
||
|
|
||
| class TemplateEntry(NamedTuple): |
There was a problem hiding this comment.
Review guide: the package design. One module per category so parallel authors never conflict. The index table is GENERATED from these entries (it cannot drift from the files that exist) and _validate_entries raises at import on duplicate keys or table-breaking characters (reviewer findings H1/L4). Playbook markdown lives in the category modules as string constants, the same pattern as the existing reference constants in agenta_builtins.py.
| # The hand-written seed playbook. Body sourced from the workspace exemplar; it layers the | ||
| # changelog use case onto the generic build loop and never repeats the loop or the config | ||
| # schema. Keep it 1-2 KB: a bigger playbook hides what is relevant from a Sonnet-class model. | ||
| _CHANGELOG_WRITER_BODY = """\ |
There was a problem hiding this comment.
Review guide: the exemplar. This changelog-writer body is your hand-written playbook from the plan (exemplar.md), verbatim except the three factual corrections recorded there: test_run explores uncommitted tools via its delta (no commit-and-stop needed), the trigger test is Lightning "Test event" / Play "Run" (not a flask), and request_input has no default field so proposed values ride the field descriptions with an enum-first "figure it out" option (real defaults tracked in #5190). The other 27 playbooks were authored against this exemplar plus the write-template-playbooks skill, then quality-swept (2 medium fixes: a no-match fabrication clause in 3 knowledge bots, and crm-updater's cross-run approval reframed to a single-run approval-gated flow).
| * Every integration the use case might touch, shown as card/chip logos. Display-only — it | ||
| * never gates Create. `requiredIntegrations` is the separate, functional connect/tools list. | ||
| */ | ||
| logoSlugs?: string[] |
There was a problem hiding this comment.
Review guide: the display/required split (decision #5). logoSlugs is display-only brand marks; requiredIntegrations stays the functional connect gate + drawer tools list, narrowed to one SOLID integration per template. CHECK-confidence integrations (datadog, pagerduty, confluence, intercom, gitlab...) appear only in logoSlugs per decision #3, so no card demands an unverified connection. Cards render via templateProviderSlugs(), which prefers logoSlugs and falls back to requiredIntegrations.
| export const TEMPLATE_CATEGORY_ORDER = ["Engineering", "Support", "Ops", "Docs"] as const | ||
| /** Canonical chip order; only categories present in the templates render. Five visible categories | ||
| * (Monitoring folds into Engineering) — see open-questions.md #1; revisit once there's click data. */ | ||
| export const TEMPLATE_CATEGORY_ORDER = [ |
There was a problem hiding this comment.
| // the static catalog serves reserved __ag__* workflows as "vN" (e.g. "v1"). Strip the | ||
| // "v" prefix before coercing so both shapes parse to a number instead of NaN. | ||
| version: z | ||
| .preprocess( |
There was a problem hiding this comment.
Review guide: pre-existing bug fixed on the FE read path. The static catalog stamps reserved ag* revisions with version "v1"; z.coerce.number() rejected it, the whole response failed validation, and retrieveWorkflowRevision returned null, so EVERY static workflow retrieve (including the build kit) was silently dropped. This preprocess strips the v prefix; output stays number|null|undefined and downstream consumers were checked. 5 new unit tests, including the real static response envelope.
| // pending, wait at most 10s for the overlay, then send anyway (kit-less) with a warning. | ||
| const [overlayWaitElapsed, setOverlayWaitElapsed] = useState(false) | ||
| useEffect(() => { | ||
| if (!firstRunPrompt || overlayReady) return |
There was a problem hiding this comment.
Review guide: the turn-1 race fix. Before: the seed auto-send fired as soon as the model unblocked, while the build-kit overlay fetch was still in flight, so the FIRST builder turn of every card flow ran kit-less (verified live: turn 1 carried tools: [] and no skill). Now the send waits on a derived overlay-ready atom mirroring the overlay atom's own resolution, with a 10s escape hatch that sends anyway and console.warns, so a broken overlay endpoint cannot wedge the send forever.
| .agents/* | ||
| !.agents/skills/ | ||
| .agents/skills/* | ||
| !.agents/skills/write-template-playbooks/ |
There was a problem hiding this comment.
Review guide. The repo ignores all dotdirs (.* above); the three existing repo skills were force-added long ago. These scoped negations re-include only the new write-template-playbooks skill (a dir-by-dir re-include chain, because git cannot re-include inside an excluded dir); everything else under .agents/ and .claude/ stays ignored exactly as before.
| _validate_entries([entry]) | ||
|
|
||
|
|
||
| def test_frontend_and_sdk_template_keys_match(): |
There was a problem hiding this comment.
Review guide: the cross-repo drift guard. Parses the AGENT_TEMPLATES keys out of templates.ts by regex and asserts 1:1 parity with the SDK entries, so a card without a playbook (or a playbook without a card) fails the SDK suite. Skips with a clear message when the web tree is absent (SDK-only distribution).
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
web/oss/src/components/pages/agent-home/assets/templates.ts (1)
141-1179: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSuggest extending the registry test to catch
toolsSummary/integration-label drift.
templates.test.tscurrently only asserts thatlogoSlugs/requiredIntegrationsslugs exist inPROVIDERS, which is why the HubSpot/Gmail mismatch above wasn't caught. A cheap addition — assertingtoolsSummarymentions the label oftemplate.requiredIntegrations[0].slug— would guard against future drift as more templates are added.sdks/python/agenta/sdk/agents/adapters/agent_templates/__init__.py (1)
52-67: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd
categoryto the table-breaking character validation.
_validate_entrieschecksnameandmatchfor|and\nbecause they render in the index Markdown table, butcategoryalso renders in the table at line 95 and is not validated. A future contributor adding a category name containing|or a newline would silently break the generatedindex.mdtable. Thekeyfield (line 96) has the same exposure, though it's also used as a filename so the risk is doubly relevant.♻️ Proposed fix
- for field_name in ("name", "match"): + for field_name in ("name", "category", "match"):If
keyshould also be guarded (it renders in the table and doubles as a filename), add it too:- for field_name in ("name", "match"): + for field_name in ("key", "name", "category", "match"):
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6f947c9e-8553-461c-a859-a00f3792bb34
📒 Files selected for processing (29)
.agents/skills/write-template-playbooks/SKILL.md.agents/skills/write-template-playbooks/references/prompting-checklist.md.agents/skills/write-template-playbooks/references/worked-example.md.claude/skills/write-template-playbooks.gitignoredocs/design/agent-workflows/documentation/tools.mddocs/design/agent-workflows/projects/agent-templates/open-questions.mddocs/design/agent-workflows/projects/agent-templates/plan.mddocs/design/agent-workflows/projects/agent-templates/playbook-spec.mddocs/design/agent-workflows/projects/agent-templates/status.mdsdks/python/agenta/sdk/agents/adapters/agent_templates/__init__.pysdks/python/agenta/sdk/agents/adapters/agent_templates/engineering.pysdks/python/agenta/sdk/agents/adapters/agent_templates/knowledge.pysdks/python/agenta/sdk/agents/adapters/agent_templates/monitoring.pysdks/python/agenta/sdk/agents/adapters/agent_templates/ops.pysdks/python/agenta/sdk/agents/adapters/agent_templates/sales.pysdks/python/agenta/sdk/agents/adapters/agent_templates/support.pysdks/python/agenta/sdk/agents/adapters/agenta_builtins.pysdks/python/oss/tests/pytest/unit/agents/test_agenta_builtins_reference_files.pyweb/oss/src/components/AgentChatSlice/AgentChatPanel.tsxweb/oss/src/components/pages/agent-home/PlaygroundOnboarding/OnboardingConfigPanel.tsxweb/oss/src/components/pages/agent-home/assets/templates.test.tsweb/oss/src/components/pages/agent-home/assets/templates.tsweb/oss/src/components/pages/agent-home/components/TemplatesSection/TemplateCategoryChips.tsxweb/packages/agenta-entities/src/workflow/core/schema.tsweb/packages/agenta-entities/src/workflow/index.tsweb/packages/agenta-entities/src/workflow/state/index.tsweb/packages/agenta-entities/src/workflow/state/store.tsweb/packages/agenta-entities/tests/unit/workflowVersionSchema.test.ts
💤 Files with no reviewable changes (1)
- web/oss/src/components/pages/agent-home/components/TemplatesSection/TemplateCategoryChips.tsx
✅ Files skipped from review due to trivial changes (5)
- .claude/skills/write-template-playbooks
- .agents/skills/write-template-playbooks/SKILL.md
- docs/design/agent-workflows/projects/agent-templates/playbook-spec.md
- docs/design/agent-workflows/projects/agent-templates/open-questions.md
- docs/design/agent-workflows/projects/agent-templates/plan.md
🚧 Files skipped from review as they are similar to previous changes (1)
- docs/design/agent-workflows/projects/agent-templates/status.md
…d home catalog Template cards seeded one-line prompts the builder agent could not execute: it had no playbook for the use case, so it asked the wrong questions, wired the wrong tools, and stalled. - build-an-agent now routes through references/agent-templates/index.md; 28 per-use-case playbooks ship as generated SkillFiles from the new agent_templates package (one module per category, import-time validation, drift + FE-parity tests). - Home registry grows 6 -> 28 templates across 5 categories with short 'Build a <X> that ...' seeds, a display-only logoSlugs field, PROVIDERS 6 -> 22, and a registry-invariant test. - New write-template-playbooks repo skill encodes the playbook format and the prompting checklist (unignored via scoped .gitignore negations). - Fixes found by live verification: static workflow version 'v1' was rejected by the FE zod schema (killing every __ag__* retrieve), the first builder turn auto-sent before the build-kit overlay resolved, and the newrelic logo slug hit the CDN fallback. Decisions from the PR #5188 review are recorded in the workspace (open-questions.md); the elicitation default field is tracked in #5190. Claude-Session: https://claude.ai/code/session_016MYos6VaKu8AjhcLFkaMUk
4627d7b to
a4c6dfd
Compare
…d home catalog Template cards seeded one-line prompts the builder agent could not execute: it had no playbook for the use case, so it asked the wrong questions, wired the wrong tools, and stalled. - build-an-agent now routes through references/agent-templates/index.md; 28 per-use-case playbooks ship as generated SkillFiles from the new agent_templates package (one module per category, import-time validation, drift + FE-parity tests). - Home registry grows 6 -> 28 templates across 5 categories with short 'Build a <X> that ...' seeds, a display-only logoSlugs field, PROVIDERS 6 -> 22, and a registry-invariant test. - New write-template-playbooks repo skill encodes the playbook format and the prompting checklist (unignored via scoped .gitignore negations). - Fixes found by live verification: static workflow version 'v1' was rejected by the FE zod schema (killing every __ag__* retrieve), the first builder turn auto-sent before the build-kit overlay resolved, and the newrelic logo slug hit the CDN fallback. Decisions from the PR #5188 review are recorded in the workspace (open-questions.md); the elicitation default field is tracked in #5190. Claude-Session: https://claude.ai/code/session_016MYos6VaKu8AjhcLFkaMUk
…ate feature Codex blockers: static workflow versions now compare tolerantly (v1 == 1 == 1) in the static catalog and the static-ref consistency check, with a round-trip test, so an FE that parses 'v1' to 1 can never send back a version the backend refuses; every card's requiredIntegrations now lists exactly the connections its playbook hard-requires (13 rows became two-integration), with the rule documented on the field. Codex mediums: the first-turn overlay wait resets on entity/seed change and arms only when the send is blocked solely on the overlay; the FE-parity test bracket-scans the array body and fails loudly on a missing marker; template keys must be kebab slugs. CodeRabbit round: replaced every 'press Enter to accept' field description (request_input has no prefill) with 'Recommended: X' / 'Leave empty to use X' across all playbooks and the authoring skill; support-reply-drafter's card required integration matches its playbook (zendesk); inventory persona and CHECK counts corrected by recount (F=27 J=13 R=14; 7 CHECK / 21 SOLID). Claude-Session: https://claude.ai/code/session_016MYos6VaKu8AjhcLFkaMUk
a4c6dfd to
c188226
Compare
Context
The agent home page shows a strip of template cards. Clicking one opens a builder agent
seeded with a single outcome sentence, like "Create an agent that turns merged pull requests
into clean release notes and publishes them to our docs page." That sentence describes the
result but never says how to reach it. The builder (a Sonnet-class model running the
build-an-agentskill) had no playbook for a changelog writer, so it guessed: it asked theuser questions they could not answer, wired read tools but never the write tool that does the
real work, or started committing on wrong priors. The prompt was the wrong place for the
per-use-case intelligence.
This PR moves that intelligence into the skill. The card prompt shrinks to a pointer, and the
depth lives in a playbook the builder reads in full once it recognizes the use case.
Changes
Four parts move together.
Skill routing + playbook package.
BUILD_AN_AGENT_SKILL(
sdks/python/agenta/sdk/agents/adapters/agenta_builtins.py) gained a three-line routingparagraph: before the generic build loop, the builder checks
references/agent-templates/index.mdfor a playbook matching the ask, and follows it when onematches. The 28 playbooks plus a generated
index.mdcome from a new package,agent_templates/, one module per category (TemplateEntry(key, name, match, category, body)). The package validates its entries at import time (unique keys, no|or newline intable cells) and generates the index from the entries, so the router table can never drift
from the files that exist.
Frontend registry.
web/oss/src/components/pages/agent-home/assets/templates.tsgrew from6 to 28 templates.
TEMPLATE_CATEGORY_ORDERis now[Engineering, Support, Sales, Knowledge, Ops](Monitoring folds into Engineering per decision#1). A new display-only
logoSlugsfield carries the brand marks a card shows;requiredIntegrationsstays the separate functional connect/tools list.PROVIDERSgrew from6 to 22 entries.
seedMessage === builderMessageshort seeds ("Build a that ...").OnboardingConfigPanelcaps at 6 cards,TemplateCategoryChipsis deleted, and a newregistry-invariant test guards the shape.
Authoring skill. A new repo skill,
.agents/skills/write-template-playbooks/(SKILL.md +prompting-checklist + a worked example), teaches how to write one playbook file so the catalog
can grow consistently.
Tests. An SDK drift + FE-parity test file and a frontend registry-invariant test (details
below).
Fixes found by live verification. Three bugs surfaced while verifying against the running
stack, two of them pre-existing but fatal to this feature's smoke path:
version: "v1"(
web/packages/agenta-entities/src/workflow/core/schema.ts), so every__ag__*staticworkflow retrieve was silently discarded and the build kit never loaded through the primary
path. The schema now parses
vNstrings to numbers (5 new unit tests).resolved, so turn 1 always ran without the kit even when the fetch worked. The auto-send now
waits on a derived overlay-ready atom, with a 10-second escape hatch that sends anyway and
logs a warning.
newrelicComposio logo slug returned the grey fallback placeholder; the working slug isnew_relic.Concrete before/after for the changelog-writer card:
Before, the card's builder message was the only thing the builder received.
After, the seed is a short pointer, and the builder reads the playbook for the depth.
On that seed the builder now reads
references/agent-templates/index.md, matches thechangelog-writer row, opens
references/agent-templates/changelog-writer.md, and follows areal playbook: discover the GitHub read tools,
test_runthem through a delta, then ask theuser one
request_inputform for the repo and where to publish, before it commits.Scope / risk
No wire or protocol change. The playbook files ride the existing
SkillFilemechanism, andthe playground still embeds the
build-an-agentskill by slug exactly as before. The agentconfig schema is unchanged.
Docs-only surfaces are frozen. The full flag-off surface delete is deferred (decision #4), so
the old card-prompt gallery path stays behind the flag rather than being removed here. The
CHECK integration marks are display-only:
logoSlugsrenders brand marks and never gatesCreate;
requiredIntegrationsis the only functional connect gate. Analytics vocabularychanged per decision #6.
Tests / notes
sdks/python/oss/tests/pytest/unit/agents/test_agenta_builtins_reference_files.py: 16passed. It pins the routing paragraph, the generated index, per-playbook file existence, and
parity between the SDK template keys and the frontend registry.
web/oss/src/components/pages/agent-home/assets/templates.test.tsvia vitest: 17 passed.files for
__ag__build_an_agentwith no restart (dev hot-reload picked up the SDK change);the strip renders all 28 cards with exact tab counts (All 28, Engineering 10, Support 4,
Sales 5, Knowledge 5, Ops 4) and real brand logos; picking a card seeds the composer and
shows the provenance chip. Unit sweep: 606 passed / 2 pre-existing failures unrelated to this
change (
test_unknown_harness_is_permissive,test_run_selection_unknown_permission_falls_back_to_allow_reads— stale against a recentcapabilities/dtos change, flagged to that lane).
Changelog writer card: the auto-sent turn 1 carried the full build kit (17 tools plus the
build-an-agent skill embed, proving the zod fix and the overlay wait live); the agent read
SKILL.md, thenreferences/agent-templates/index.md, thenchangelog-writer.md; itsfirst reply was a
request_inputform (repository, where to publish, how releases work)with "Figure it out" / "Use your best judgment" as leading enum options, no free-form
questioning. One environment caveat: the test project's OpenAI and Anthropic keys are out
of credit, so the passing run used the subscription (self-managed) credential path; the
default card experience needs a funded key in the project vault.
Review rounds folded in
fixed in the follow-up commit. (1) Version round-trip: the FE parses the static catalog's
"v1"to1, so a later reference round-trip could send a version the backend's exactstring compare refuses; the static catalog and the static-ref consistency check now compare
versions tolerantly (
"v1"=="1"==1) with a parametrized test. (2)requiredIntegrationsunder-declared what playbooks actually need; every card now listsexactly the connections its playbook hard-requires (13 rows became two-integration) and the
rule is documented on the field. Mediums also applied: the first-turn overlay wait resets on
entity/seed change and arms only when the send is blocked solely on the overlay; the
FE-parity test bracket-scans the array and fails loudly on a missing marker; template keys
must be kebab slugs. Deferred with notes in the workspace status: allOf/oneOf requirement
groups,
TemplateEntry/overlayReadyrenames.that leaked into shipped content are fixed ("press Enter to accept" field descriptions
replaced with "Recommended: X" / "Leave empty to use X" across all playbooks and the
authoring skill; support-reply-drafter's card matches its playbook, zendesk). Inventory
persona and CHECK counts corrected by recount (F=27, J=13, R=14; 7 CHECK / 21 SOLID).
parity test, table-injection guard, all applied) and a 27-playbook quality sweep (two medium
content fixes: a no-match fabrication clause in three knowledge bots, and crm-updater's
cross-run approval reframed to a single-run approval-gated flow).
How to QA
Prerequisites: the local dev stack (
load-envfor your edition, thenbash ./hosting/docker-compose/run.sh <flags> --build). No extra setup.Steps:
28 cards across them.
requests into release notes and publishes them." and a template chip appears.
references/agent-templates/index.md, then openreferences/agent-templates/changelog-writer.md, then reply with arequest_inputformasking for the repo and where to publish.
Expected result: the builder follows the changelog-writer playbook instead of guessing. It
asks one grounded
request_inputform and proposes GitHub read tools plus the publish tool,not a blind commit.
Automated tests:
Edge cases:
deleted, so that surface renders as before.
logoSlugs) shows those brand marks without demandingthe user connect them. Only
requiredIntegrationsgates Create.Design docs: docs/design/agent-workflows/projects/agent-templates/ (see open-questions.md for the six decisions).
https://claude.ai/code/session_016MYos6VaKu8AjhcLFkaMUk