Skip to content

fix(evaluation): score CAR on groundtruth_eval.json concept recall #634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ShuxinLin merged 3 commits into from
Oct 14, 2026
Merged

fix(evaluation): score CAR on groundtruth_eval.json concept recall #634

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
83a4dfd
fix(evaluation): score CAR on groundtruth_eval.json concept coverage
ShuxinLin Oct 14, 2026
d9268d8
fix(evaluation): match regular plurals of CAR concept forms
ShuxinLin Oct 14, 2026
767a8b3
refactor(evaluation): score CAR concept answers on recall alone
ShuxinLin Oct 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 68 additions & 25 deletions docs/static-json-evaluation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,18 +100,36 @@ The evaluator then joins trajectories and ground truth by scenario id.

Scenario folders may also include `groundtruth_eval.json`, which carries
scorer-specific metadata alongside the canonical answer in `groundtruth.txt`.
The `static_json` scorer does not read it. For clarification-abstain-response
(CAR) scenarios the expected mode is the single key of `groundtruth.txt`, and
the required terms are extracted from the value under that key.
The loader exposes it as `Scenario.evaluation_metadata`. `static_json` reads
it only for clarification-abstain-response (CAR) scenarios, whose expected
mode is the single key of `groundtruth.txt`.

Example `groundtruth.txt`:
Example CAR `groundtruth.txt`:

```json
{
"clarification": "Which asset and which time window do you mean?"
}
```

Example CAR `groundtruth_eval.json`:

```json
{
"clarification": "Which asset do you mean by 'the main unit'?"
"mode": "clarification",
"required_terms": [
{"concept": "asset", "any_of": ["asset", "equipment"]},
{"concept": "time window", "any_of": ["time window", "date range"]}
],
"optional_terms": []
}
```

Each required concept lists the surface forms (`any_of`) that count as
covering it. A bare string in `required_terms` is a concept with that one
form. `mode` and `optional_terms` are not scored; the mode comes from
`groundtruth.txt`.

---

## How Matching Works
Expand Down Expand Up @@ -220,7 +238,7 @@ object.
| `extra_keys` | Keys present in model output but not in gold |
| `details` | Per-key comparison records |

The scorer marks a scenario as passed only when the structured answer is a strict exact match. Partial correctness is still available through `score`, `f1`, `partial_exact_match_accuracy`, and detailed key-level results.
The scorer marks a scenario as passed only when the structured answer is a strict exact match, and reports `f1` as `score`. Partial correctness is still available through `score`, `f1`, `partial_exact_match_accuracy`, and detailed key-level results. CAR scenarios with `groundtruth_eval.json` are the exception: see below.

### CAR metrics and pass criteria

Expand All @@ -232,20 +250,44 @@ exactly one of these top-level keys:
response | clarification | abstain
```

The required terms are extracted from the gold value: quoted phrases, asset or
fault identifiers (e.g. `PMP42144`), a leading `yes` / `no` stance, and a small
allowlist of domain terms. Matching is case-insensitive on normalised text.
Matching is case-insensitive on normalised text and on word boundaries, so
`asset` does not match inside `assetid`.

#### With `groundtruth_eval.json`

When the scenario's `groundtruth_eval.json` has `required_terms`, the score is
`recall`, in [0, 1]:

A CAR scenario passes when:
1. If the answer is not a JSON object with exactly one top-level key, or that
key is not the expected mode, recall is 0.
2. Otherwise recall is the fraction of required concepts covered. A
concept is covered when any one of its `any_of` forms, or that form's
regular plural (`asset` → `assets`, `switch` → `switches`, `battery` →
`batteries`), appears in the value under the key. Irregular forms such as
`criteria` / `criterion` still need listing.

1. the model returns a JSON object with exactly one top-level key,
2. that key matches the expected mode (case-insensitive), and
3. every required term appears in the value under that key.
The scenario passes when recall is at least 0.5 (`CAR_PASS_THRESHOLD`).
`precision` and `f1` are `null`: precision needs a count of what the model
asserted, and a free-text value gives none, so it could only repeat recall.
For the example above (two concepts):

The score is the F1 over the mode key plus each required term. Gold has
`1 + len(required_terms)` entries; the model side has the same plus any extra
top-level keys. For gold `{"clarification": "Which asset do you mean by 'the
main unit'?"}` (one required term, `main unit`):
| Model answer | Passed | Score (recall) |
| ------------------------------------------------------------- | ------ | -------------- |
| `{"clarification": "Which equipment, over what date range?"}` | yes | 1.0 |
| `{"clarification": "Which assets?"}` | yes | 0.5 |
| `{"clarification": "Can you say more?"}` | no | 0.0 |
| `{"abstain": "Which asset and time window?"}` | no | 0.0 |

#### Without `groundtruth_eval.json`

The required terms are extracted from the gold value: quoted phrases, asset or
fault identifiers (e.g. `PMP42144`), a leading `yes` / `no` stance, and a small
allowlist of domain terms. The scenario passes when the answer has exactly one
top-level key, that key is the expected mode, and every required term appears
in the value under it. The score is the F1 over the mode key plus each required
term. Gold has `1 + len(required_terms)` entries; the model side has the same
plus any extra top-level keys. For gold `{"clarification": "Which asset do you
mean by 'the main unit'?"}` (one required term, `main unit`):

| Model answer | Passed | Score |
| ---------------------------------------------------- | ------ | ----- |
Expand All @@ -255,13 +297,13 @@ main unit'?"}` (one required term, `main unit`):

Additional CAR report fields:

| Metric | Meaning |
| ---------------------- | ---------------------------------------------------------- |
| `mode_key_match` | 1.0 when the returned mode key matches the expected mode |
| `mode_exactly_one_key` | 1.0 when the answer has exactly one top-level key |
| `mode_required_terms` | Required terms extracted from the gold answer value |
| `mode_matched_terms` | Required terms found in the model answer value |
| `mode_term_coverage` | Fraction of required terms found |
| Metric | Meaning |
| ---------------------- | -------------------------------------------------------------------- |
| `mode_key_match` | 1.0 when the returned mode key matches the expected mode |
| `mode_exactly_one_key` | 1.0 when the answer has exactly one top-level key |
| `mode_required_terms` | Required concepts from `groundtruth_eval.json`, or extracted terms |
| `mode_matched_terms` | Required concepts or terms found in the model answer value |
| `mode_term_coverage` | Fraction of required concepts or terms found, regardless of the key |

---

Expand Down Expand Up @@ -421,13 +463,14 @@ The scorer itself only needs:
```text
gold_answer
model_answer
eval_metadata # groundtruth_eval.json, CAR only
```

The evaluation pipeline is responsible for:

```text
trajectory file → final answer + scenario_id/run_id
scenario folder → groundtruth.txt
scenario folder → groundtruth.txt (+ groundtruth_eval.json)
join trajectory and scenario
run scorer
write report
Expand Down
2 changes: 0 additions & 2 deletions src/evaluation/report.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@ def _aggregate_score_summary(results: list[ScenarioResult]) -> dict[str, Any]:
"mode_exactly_one_key",
"mode_term_coverage",
"mode_optional_term_coverage",
"car_score",
]

score_values: dict[str, list[float]] = {name: [] for name in metric_names}
Expand Down Expand Up @@ -151,7 +150,6 @@ def _aggregate_score_summary(results: list[ScenarioResult]) -> dict[str, Any]:
"mode_optional_term_coverage_avg": _avg(
score_values["mode_optional_term_coverage"]
),
"car_score_avg": _avg(score_values["car_score"]),
"missing_keys_total": missing_keys_total,
"extra_keys_total": extra_keys_total,
"detail_entries_total": detail_entries_total,
Expand Down
Loading
Loading