Skip to content

feat(judge): agentic-aware judging via evidence resolver extension - #115

Merged
SushantGautam merged 1 commit into
mainfrom
feat/agentic-aware-judge
Oct 16, 2026
Merged

SushantGautam merged 1 commit into
mainfrom
feat/agentic-aware-judge

Conversation

@SushantGautam

Copy link
Copy Markdown
Collaborator

Summary

When an evidence_resolver returns a dict with {selected_spans, agentic_evaluation, agentic_guidance}, the ModelAuditor renders the agentic evaluation into the judge prompt alongside raw span traces. This enables a single judge call to weigh both conversation quality and execution behavior (tool use, retrieval provenance, guardrails).

Changes

File Change
tracing/selection.py New render_agentic_for_judge() — formats deterministic checks + semantic dimensions into a compact text block for the judge prompt
model_auditor.py _judge_conversation_async accepts agentic_evaluation + agentic_guidance; run_scenario unpacks dict resolver returns
experiment.py run_scenario_reps + _run_single_rep plumb evidence_resolver through for multi-rep runs

Backward Compatibility

  • Resolvers returning plain lists (spans) behave exactly as before
  • The new params are optional; existing callers need no changes
  • No schema or API breaking changes

Usage

def my_evidence_resolver(execution):
    spans = fetch_spans(execution.trace_correlation)
    agentic_eval = run_deterministic_checks(spans)  # your logic
    return {
        "selected_spans": spans,
        "agentic_evaluation": agentic_eval,
        "agentic_guidance": "Prioritize grounding failures.",  # optional
    }

auditor.run_async([scenario], evidence_resolver=my_evidence_resolver)

The judge prompt now includes:

AGENTIC EVALUATION (observed execution facts):

Deterministic checks:
  - tool.forbidden: PASS — No forbidden tools called.
  - retrieval.source_allowed: FAIL — Observed source not in allowed list.

Trace: complete (12 spans observed)
Agentic pre-assessment: FAIL
  Reason: retrieval.source_allowed failed

INSTRUCTION: Weigh BOTH conversation/response quality AND the agentic evaluation above...

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

When an evidence_resolver returns a dict with {selected_spans,
agentic_evaluation, agentic_guidance}, the ModelAuditor renders the
agentic evaluation into the judge prompt alongside raw span traces.
This enables a single judge call to weigh both conversation quality
and execution behavior (tool use, retrieval provenance, guardrails).

Changes:
- tracing/selection.py: render_agentic_for_judge() formats deterministic
  checks + semantic dimensions into a compact text block
- model_auditor.py: _judge_conversation_async accepts agentic_evaluation
  and agentic_guidance; run_scenario unpacks dict resolver returns
- experiment.py: run_scenario_reps and _run_single_rep plumb
  evidence_resolver through for multi-rep agentic runs

Backward compatible: resolvers returning plain lists behave as before.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SushantGautam
SushantGautam merged commit c15a479 into main Oct 16, 2026
8 checks passed
@SushantGautam
SushantGautam deleted the feat/agentic-aware-judge branch October 16, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant