Skip to content

Add ASR transcription targets and reference-based faithfulness auditing - #116

Open
KushtrimVisoka wants to merge 1 commit into
SimulaMet:mainfrom
KushtrimVisoka:feat/asr-faithfulness
Open

KushtrimVisoka wants to merge 1 commit into
SimulaMet:mainfrom
KushtrimVisoka:feat/asr-faithfulness

Conversation

@KushtrimVisoka

Copy link
Copy Markdown
Contributor

What changes

SimpleAudit can now audit speech-to-text systems using plain-text references, without requiring timestamps. ASRTarget uploads one local audio file through the OpenAI-compatible multipart POST /audio/transcriptions endpoint, with OpenAI/OpenRouter helpers and a configurable base URL for compatible local servers. Existing transcriptions can be evaluated through CallableTarget.

The named asr_faithfulness judge checks scenario expectations against reference and transcription text, reports evidence from both texts, explains meaning changes and conditional potential consequences, and derives severity using the existing checklist rules. For example, dropping “not” from “the proposal was not approved” is recorded as a meaning-changing transcription error; faithfully transcribing a speaker's false or harmful statement is not itself an ASR failure.

  • Optional conservative normalization (Unicode NFC, case folding, whitespace); original transcription/reference text is retained.
  • References stay out of transcription requests, and audio attachments stay out of the text judge's input.
  • Invalid reference/rubric configuration is rejected before inference. Missing, uncertain or unverifiable observations are ungraded, rather than passes.
  • Single-turn target defaults avoid generated probes and preserve literal transcript content, including reasoning-tag text.
  • Saved reference settings are restored for judge-only rejudging and reframing.
  • Includes README usage, judge-hook documentation, and regression tests. Adds an explicit httpx dependency for the transcription client.

Validation

  • Full local suite: 1,609 passed, 4 skipped (Python 3.12).
  • Repository CI lint rules (ruff check --select E9,F821 .), focused lint/format checks, and git whitespace checks passed.
  • Wheel built successfully; smoke-tested importing the target and judge directly from the wheel.
  • Live Norwegian audio transcription via OpenRouter openai/whisper-1, with openai/gpt-4o judging through OpenRouter. Full upload → transcription → judgment → save/load completed in both normalization modes.
  • Live controls: identical text passes; altered quantities, reversed negation, complete omission and invented speech on a silence reference are detected; case/whitespace-only changes pass with normalization. A provider returned evidence_kind="insertion"; this known alias is canonicalized to direct only for insertion errors, with the original value retained.
  • Tests also cover malformed responses, reference isolation, invalid configuration, unknown/contradictory observations, original-text preservation, and save/load/rejudge/reframing round trips.

Scope and limitations

Direct authenticated OpenAI calls were not validated because the available OpenAI key was empty. Compatibility follows its documented multipart contract; live compatibility was established with OpenRouter. Other providers and optional timestamp/diarization formats were not live-tested. This is not a validated harm classifier or WER/CER implementation: consequences remain conditional, reference quality matters, and quote verification does not prove the judge's interpretation. The live sample and controls are smoke tests, not a benchmark of judge accuracy.

No credentials, private audio, source transcripts or live-result artifacts are included in this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant