Skip to content

feat: add string includes and prebuilt judge scorers - #26

Closed
TheGreatAxios wants to merge 1 commit into
cl-11499-run-judge-scorers-as-a-workflow-model-stepfrom
cl-11498-add-the-first-prebuilt-scorers
Closed

TheGreatAxios wants to merge 1 commit into
cl-11499-run-judge-scorers-as-a-workflow-model-stepfrom
cl-11498-add-the-first-prebuilt-scorers

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

  • Add includes, a string check: case-insensitive substring after NFKC, trim, and whitespace collapse. An empty needle fails closed.
  • Add faithfulness, answerRelevancy, correctness, and trajectoryAccuracy as one-step workflow factories, exported only from @corbits/evals/workflow.
  • parsePrebuilt rejects fenced or out-of-range judge output instead of clamping it.

Verification

  • bun run check exited 0 (typecheck, oxlint, oxfmt, unit tests, 60 e2e tests).
  • Review of 2f46622 found no defects.

Stacked on #25.

Fixes CL-11498

@linear-code

linear-code Bot commented Oct 26, 2026

Copy link
Copy Markdown

CL-11498

@corbits-triage

corbits-triage Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Triage: Fail

@TheGreatAxios, please address the following:

  • mixes unrelated changes, not part of feat: add string includes and prebuilt judge scorers

Stacked on #25: only this pull request's own changes were checked.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review · Approve

Adds includes and the prebuilt judge scorers as one workflow step.

Findings

No findings.

includes reads expectedOutput. Graded judges fail closed on partial credit and extra keys. Each prebuilt is one judge step with input: { from: "trigger.payload" }. includes is exported from the package root. faithfulness, answerRelevancy, correctness, trajectoryAccuracy, and parsePrebuilt are exported from the workflow entry. bun test e2e/prebuilt.e2e.test.ts plus a deleted probe exited 0 (8 pass).

@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Superseded by the 0.2.0 release: #32

Do not merge this pull request separately. The release branch cherry-picks the feature stack (except the 0.1.3 bump on #22) and sets @corbits/evals to 0.2.0 with @corbits/system-one ^0.4.0.

@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Superseded by the 0.2.0 release: #32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant