[andrewklatzke]

Summary

• Add launchdarkly-ai-typesafe, a judge handler that sends a variation's classifiers to Jev in one call.
• Each label becomes its own judge_results entry, tracked on that label's eventKey, with the selected label as the reason.
• A score is the expected rubric index divided by one less than the number of levels, then clamped to 0–1.

https://launchdarkly.atlassian.net/browse/AIC-3534

Test plan

• pytest packages/typesafe/tests packages/client/tests/test_judges.py packages/client/tests/test_conversation.py packages/client/tests/test_judge_scoring.py packages/client/tests/test_schema.py
• Run a TypeSafe judge variation and confirm one result per label, with the label value in response

Made with Cursor

[cursor]

Stale Bugbot comment from a previous run.

[jeffdupont]

Reviewed together with js-ai-sdk #122 and ai-sdks-monorepo #49. The handler matches the spec and the JS side. I checked it against typesafe_sdk 0.7.2: NoulAnswer.noul is a float, and ScoreAnswer.legend is keyed by integer levels starting at 0, so the score mapping is right. The _handler_for rewrite also fixes a real ordering bug: before, a wildcard listed first beat an exact provider match. The new skip for a judge with no matching handler matches JS runJudges.

Tests: 661 passed on this head. Merged into current main (6 conflicts: release-please files, pyproject.toml workspace members, uv.lock, and a docstring in types.py), packages/typesafe plus packages/client give 1586 passed, 1 skipped.

1. Should fix: deferred callers don't know to track metrics. run_judge now returns results / metrics / event_key, and the docstring says to track each metrics entry instead of evaluation_metric_key. But the only deferred guidance in the README is run_judge(task, handlers) on your own thread. A worker written the way it works today tracks evaluation_metric_key with result.score. For a TypeSafe judge, that tracks nothing (no metric key) or tracks the first label under the wrong key. JS #122 updated examples/judge-worker.ts with the pattern. Python needs the same, in the README or an example.

2. Should fix: create_typesafe_handler reports no helper usage. TESTING.md §3.27 makes report_usage(...) the first statement of every public factory, with an ID in api-surface/usage-ids.txt. Every handler on main does it (for example openai-messages/handler.py:231).

Nice to have: questions.py imports FORMATTING_INSTRUCTIONS from the internal module launchdarkly_ai_server.judge_scoring, while JS #122 makes it a public root export. The two languages should agree on whether it's public API.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3cd5635. Configure here.}

[jeffdupont]

Spec-wise this lines up with §1.12 / A.18 and the api-surface list 👍 One thing: judges.py:192/448 now pass input to every online judge, but TESTING.md:1476 still says online judges get message_history + response_to_evaluate "and nothing else". Probably just needs that line updated in monorepo #49 so the spec agrees with §1.12.