feat(kitaru): optional trace source and replay verification - #10
Open
dnth wants to merge 23 commits into
Open
Conversation
Land ADRs 0001-0010 and CONTEXT.md, add the kitaru extra (pinned >=0.22,<0.23), map frozen cohorts onto the JSONL ingest already reads, and gate apply on a hash-matching verification. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
Match ToolPolicyMissError / "No history result for tool" so those replays are TOOL_HISTORY_MISS, not replay_failures. Read latest.json and reuse the local snapshot unless --refresh so re-runs stay offline. Require state.result before apply; a finished REVIEW/FAILED report still gates. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
Key on-disk snapshots by cohort_version_id/evaluation_name so a later --kitaru-evaluation cannot clobber another judge's JSONL. Store latest.json as one pointer per (cohort_name, evaluation_name), and if that pointer is missing pick the newest matching snapshot rather than the lexicographically first directory. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
Do not silently drop replay sessions when select_evaluation fails, the evaluator_version mismatches, or classify_scores returns None. Record them as diverged with SELECT_EVALUATION_FAILED, EVALUATOR_VERSION_MISMATCH, and SCORE_UNCLASSIFIED. Leave them out of replay_failures. Headline aggregates still come from Kitaru. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
…uator Close and drop the owned gateway after each run_async so preflight cannot leave an httpx client bound to a dead loop for submit. Print replay failure count and per-session errors next to Divergence. Look up evaluator_id by mapping.evaluator_name, falling back to the CLI flag. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
Persist CohortResolution.version_number on SourceMeta and treat a numeric CLI ref the same as id/display_version, so a re-run with --kitaru-cohort-version 3 can stay offline. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
Match apply: if the template on disk no longer hashes to proposal.base_prompt_hash, mark stale and refuse with the same message. Do not build a request, submit, or create an experiment. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
build_request's template.read_text can raise OSError after the stale check; catch it as VerifyError like apply_proposal, so it hits main's handler instead of a traceback. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
Collect was four sequential HTTP calls per replay after wait. Fetch per-replay payloads with the same jobs=8 semaphore as source mapping. apply_proposal now writes candidate_prompt's text so ADR 0009's hash gate hashes the same bytes that land on disk. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
refuse_ungated_apply no longer tells the operator to re-run verify. Verify hashes the full proposal, so a subset cannot be ungated that way. assert_override_scope treats a result graph with no root llm_call as OVERRIDE_SCOPE_DIVERGENCE instead of a vacuous pass (ADR 0006). Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
A --refresh whose mapped evaluator conflicts with the manifest used to write_snapshot (and latest.json) first, then error. Check the fetched mapping before writing so a failed refresh leaves the last-good snapshot. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
One session_nodes/evaluations error aborts the whole collect gather. Do not isolate as ReplayFailure, persist a result, or ungate apply. Resume keeps the experiment run id and redoes collect. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
A 404/timeout from the payload gather still aborts the whole collect fail-closed. Re-raise as KitaruVerifyError so the CLI prints an actionable message instead of a traceback: apply stays gated, resume redoes collect, --force if they must write anyway. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
kitaru 0.22 WorkerListParams is FilterableListParams with extra='forbid', so include_stale=False raised ValidationError before the live-worker check. List with the 0.22 constructor; worker_covers_agent_version already skips live=False. Wrap list_live_workers like the other probes. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
wait_for_experiment_run, list_replays, evaluation_aggregates, and create_experiment/start_run now raise KitaruVerifyError like payload fetch. fetch_records gather failures become KitaruSourceError. Both stays fail-closed. Delete _pick_version's trailing UUID try/except that always returned None. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
find_local_snapshot no longer aborts reuse when a snapshot dir is unreadable. list_sessions, resolve_cohort, and evaluator_id raise KitaruSourceError like fetch_records. A corrupt kitaru-source.json does not traceback or ungate apply. Delete unused load_run_source. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
candidate_prompt discarded resolve_edits rejections after apply started writing through it, so ApplyResult.resolution_rejections stayed empty. Share one helper that returns the candidate bytes plus rejection reasons so the ADR 0009 hash still matches the write, and thread those reasons into both ApplyResult paths as before. load_run_source_payload treated any JSON object as valid, so empty or partial sidecars reached build_request and KeyError/TypeError escaped the CLI handler. Reject payloads whose fingerprint/meta are not dicts with the required keys (load returns None; apply stays gated; verify raises VerifyError). Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
KitaruGateway.wait_for_experiment_run constructed KitaruClient and called wait_for_experiment_run, which is not on the 0.22 API client (close / context-manager only). Wrapping KitaruClient is also unsafe: close() shuts the shared API client. Poll self._client.experiment_runs.get until completed/failed/canceled, matching kitaru CLI poll_run. Timeouts still raise so collect stays fail-closed. Confirmed against pinned kitaru 0.22. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
find_local_snapshot already skipped unreadable fingerprint/meta, but prepare_kitaru_source then loaded source-drops.jsonl with no guard. Malformed JSON or a missing session_id/reason raised JSONDecodeError or KeyError instead of a named source error. Treat a bad drops file like a corrupt snapshot in snapshot_matches_request so reuse picks the next good snapshot, or falls through to a named fetch error when none remain. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
Captain option (c): matching_verification requires candidate_prompt_hash and the sidecar cohort_version_id. Hash-only would ungate a new cohort without re-verify; matching apply's run_id would miss a valid verify of the same candidate on the same cohort. refuse_ungated_apply still receives run_id, but only to load the source sidecar for that cohort check. Same hash + same cohort ungates; same hash + a new cohort stays gated; a hand-edit (hash change) stays gated. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
load_verification_state raised on unreadable state.json, so run_verification could not resubmit. Swallow OSError/ValueError (including pydantic ValidationError) and return None, matching list_verification_states and snapshot reuse. Apply stays gated until a usable result is stored; hash+cohort match is unchanged. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
classify_scores treated a map_score drop-reason as unchanged when both passed flags were bools. That put incomparable scores in the comparable unchanged bucket. Return None so classify_replay_session records SCORE_UNCLASSIFIED. Pass-flip still classifies earlier. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
Partial previously shared the success path with completed, so verify && apply could write from a half-run. Exit 0 only for completed; partial, canceled, failed, and incomplete stay non-zero. The hash+cohort apply gate is unchanged — a human can still apply after REVIEW when that policy allows. Co-authored-by: Dickson Neoh <dickson.neoh@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kitaru>=0.22,<0.23extra.--source kitarufetch-and-maps a frozen cohort onto the JSONL the existing pipeline already reads. Core stays Kitaru-free.tracegrad verifywith--backend kitaru. Apply is gated on candidate hash and cohort_version_id (--forceoverrides). No backend: actionable message, non-zero exit.CONTEXT.md. README covers core-only install, the extra/pin, snapshot/--refresh, verify, and the apply gate.Captain locks on this branch:
diverged(notreplay_failures, not omitted).verifyexits 0 only forstatus=completedsoverify && applycannot write a half-run.Test plan
Closes #8
Closes #9