Skip to content

feat(simulate): require stable ids in scenarios.yaml, offer to add them - #970

Draft
u9g wants to merge 1 commit into
mainfrom
jason/scenario-yaml-ids
Draft

feat(simulate): require stable ids in scenarios.yaml, offer to add them#970
u9g wants to merge 1 commit into
mainfrom
jason/scenario-yaml-ids

Conversation

@u9g

@u9g u9g commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Depends on livekit/protocol#1785 (adds Scenario.id and ScenarioGroup.id). go.mod is pinned to that branch's commit so CI builds; bump to the merged version before merging this.

Why

Today nothing correlates runs of the same scenario over time: the label, instructions, and agent expectations can all be edited, and hashing the content would make every edit a new identity. Dashboards charting a scenario's success rate over time need a stable key.

What

  • scenarios.yaml now requires an id on the file and on every scenario. Loading rejects missing or duplicate ids.
  • A file missing ids is refused. Interactively the CLI offers to insert generated 8-character ids in place, as the first key of each entry, preserving comments, key order, and indentation (yaml.v3 node edit, no re-marshal). --yes inserts without asking; non-interactive without --yes errors with instructions.
  • Scenarios rendered by the CLI (generated runs saved to temp or via the save key, clipboard copy) are given ids so the resulting file runs as-is.
  • Ids are sent on ScenarioGroup / Scenario when creating the run. Run-level identity already exists as SimulationRun.id.

Example

id: 3k9fz2ab
name: drive-thru
scenarios:
  - id: q7x1m0dp
    label: order a burger
    instructions: ...

🤖 Generated with Claude Code

Labels, instructions, and expectations all change over a scenario's life,
so nothing correlates runs of the same scenario across time. Each scenario
and the file itself now carry an id that is sent with the run.

Ids are generated with protocol's shared guid utility, prefixed SCN_
(scenario) and SCNG_ (scenario group). When a file is missing ids, the CLI
offers to insert them in place, preserving comments, key order, and
indentation.
@u9g

u9g commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Ids now come from protocol's shared guid utility with stable prefixes: SCN_ for scenarios and SCNG_ for the scenario group (livekit/protocol#1785). Dropped the randomBase36 refactor commit since it is no longer needed.

@u9g
u9g force-pushed the jason/scenario-yaml-ids branch from c674dde to 77a0cbc Compare September 9, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant