Skip to content

[Feat] Persist typed decision rules for custom automations - #3090

Closed
roomote-roomote[bot] wants to merge 2 commits into
developfrom
feature/custom-automation-shadow-judgment-1h2p4ol59lt45
Closed

roomote-roomote[bot] wants to merge 2 commits into
developfrom
feature/custom-automation-shadow-judgment-1h2p4ol59lt45

Conversation

@roomote-roomote

@roomote-roomote roomote-roomote Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

​Opened on behalf of @daniel-lxs. Follow up by mentioning @roomote-roomote, in the web UI, or in Slack.

What changed

  • Custom automation create/edit now compiles and persists one versioned decision rule snapshot.
  • The rule declares two bounded typed inputs (goal from the saved prompt and result from a future completed outcome), one TypeSafe choice question, three allowed outcomes (addressed, needs_review, no_match), and a code-owned preserve_delivery consequence for every outcome.
  • The existing automation management surface and inspect API expose the saved question, goal snapshot, inputs, outcomes, and consequence.
  • Removed result-time judgment evaluation, judgment result storage, and advisory Results UI. Automation runs keep their existing behavior unchanged.

Why this change was made

The prototype now tests the structured decision-rule contract itself instead of introducing shadow execution or per-run timing. The saved rule is replaced on create/edit and is not consumed by a run in this slice, so there is no stale-spec race or model-controlled delivery path.

Impact

Authorized automation owners and admins can inspect the exact typed rule that would constrain a future judgment-model call. The model has no path to authorize, suppress, rewrite, retry, or reroute automation work. Focused rule/persistence/API/UI tests, package type checks, pnpm lint:fast, and pnpm check-types:fast passed; pnpm knip completed with the repository's existing apps/docs/mint configuration hint. The screenshot uses a temporary local automation row for presentation only; it was deleted after capture.

Screenshots

Custom automation editor showing typed decision-rule inputs, outcomes, and code-owned consequence

@roomote-community

roomote-community Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

No code issues found. See task

  • Snapshot the judgment specification per result so an automation edit cannot evaluate an already-completed run against a different goal.

Reviewed 1667d4d

Comment thread packages/sdk/src/server/lib/automation-result-preparation.ts Outdated
@roomote-roomote roomote-roomote Bot changed the title [Feat] Add shadow judgments to custom automation results [Feat] Persist typed decision rules for custom automations Sep 22, 2026
@roomote-roomote
roomote-roomote Bot marked this pull request as ready for review September 22, 2026 06:21
@roomote-roomote

Copy link
Copy Markdown
Contributor Author

Closing for now per @daniel-lxs. The direction is moving to declarative run_when judgment conditions that are evaluated at run time (with recorded judgments for threshold tuning) instead of a save-time rule that no run consumes. This branch is preserved and can be reopened if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant