diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml index e52f6fa..37e0b08 100644 --- a/.github/workflows/macos.yml +++ b/.github/workflows/macos.yml @@ -74,11 +74,29 @@ jobs: - name: Check async Clippy run: cargo clippy -p whisper-cpp-plus --all-targets --features async -- -D warnings + - name: Check log Clippy + run: cargo clippy -p whisper-cpp-plus --all-targets --features log -- -D warnings + - name: Check Metal Clippy run: cargo clippy -p whisper-cpp-plus --all-targets --features metal -- -D warnings env: MACOSX_DEPLOYMENT_TARGET: "14.0" + # docs.rs has no network access and builds from the published package, which only ships + # whisper.cpp's public headers. Simulate that so packaging/header regressions fail here. + - name: Check docs.rs build + env: + DOCS_RS: "1" + run: | + cargo package -p whisper-cpp-plus-sys --no-verify + mkdir -p "$RUNNER_TEMP/docsrs" + tar -xzf target/package/whisper-cpp-plus-sys-*.crate -C "$RUNNER_TEMP/docsrs" + cargo doc --no-deps \ + --manifest-path "$(echo "$RUNNER_TEMP"/docsrs/whisper-cpp-plus-sys-*/Cargo.toml)" \ + --target-dir "$RUNNER_TEMP/docsrs/target" + cargo doc --no-deps -p whisper-cpp-plus --all-features \ + --target-dir "$RUNNER_TEMP/docsrs/target" + - name: Cache test models uses: actions/cache@v5 with: @@ -133,6 +151,9 @@ jobs: - name: Test workspace run: cargo test --workspace -- --test-threads=1 + - name: Test log feature + run: cargo test -p whisper-cpp-plus --features log --test logging_log_crate -- --test-threads=1 + - name: Test Metal feature run: cargo test -p whisper-cpp-plus --features metal -- --test-threads=1 env: diff --git a/CHANGELOG.md b/CHANGELOG.md index 0bd5be9..5f39c3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,16 +9,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- Updated the pinned whisper.cpp fork to `rmorse/whisper.cpp` `stream-pcm` at `de8fb5fd` (tag `v1.9.4-dev-stream-pcm`, whisper.cpp `1.9.4-dev`), based on upstream `ggml-org/whisper.cpp` `master` after `v1.9.3`. This picks up upstream releases `v1.8.7` through `v1.9.3` and ggml `0.25.1`. +- Updated the pinned whisper.cpp fork to `rmorse/whisper.cpp` `stream-pcm` at `de8fb5fd` (tag `v1.9.4-dev-stream-pcm`), based on upstream `ggml-org/whisper.cpp` `master` after the `v1.9.4` release (`v1.9.4` plus 181 later upstream commits). This picks up upstream releases `v1.8.7` through `v1.9.4` and ggml `0.25.1`. - Upstream now re-seeds the decoder between calls (ggml-org/whisper.cpp#4025), so temperature-fallback output is deterministic across repeated transcriptions on the same state. - Upstream now rejects Silero VAD models whose encoder does not have exactly 4 layers (ggml-org/whisper.cpp#4064); loading such a model with `WhisperVadProcessor` now fails at load time. - On Apple Silicon, upstream's optional ANEForge encoder backend (ggml-org/whisper.cpp#3905) is activated by the `ANEFORGE_ENCODER` and `ANEFORGE_DYLIB` environment variables, which load a dynamic library from the given path when a state is created. It is inactive unless those variables are set. - NVIDIA Parakeet support is available in the bundled C library but is not yet exposed through the Rust API. - The new upstream VAD segment and VAD-mapped token timestamp accessors are not exposed: they are only populated by upstream's built-in `whisper_full` VAD, which does not run for per-state transcription (`whisper_full_with_state`, used by this crate; see ggml-org/whisper.cpp#3423). Use the crate's own VAD pipeline instead. +- The `whisper-cpp-plus-sys` package now includes whisper.cpp's public headers (`include/*.h`, `ggml/include/*.h`) and its `LICENSE`. docs.rs builds generate bindings from these headers instead of using hand-written stubs. Regular builds are unchanged: they still download the full pinned whisper.cpp source. +- `WhisperContext` no longer allocates whisper.cpp's default state, which the crate never used (all transcription runs on explicit `WhisperState`s). This saves that state's KV caches and compute buffers for every loaded context: about 146 MB with `ggml-tiny.en.bin` as reported by whisper.cpp, and considerably more for larger models. + +### Removed + +- **Breaking:** `WhisperState::full_parallel()`. It never returned correct results: whisper.cpp writes parallel results to the context's default state, which the method never read. Use `WhisperContext::full_parallel()`, which returns the merged `TranscriptionResult`. +- **Breaking:** `WhisperContext::n_len()`. It reported the mel length of the context's default state, which the crate never transcribes on. Use `WhisperState::n_len()` for the state you transcribed with. ### Added - `WhisperState::full_get_segment_no_speech_prob()`, previously only used internally by the temperature-fallback transcriber. +- `WhisperLog` (wrapping `whisper_log_set`) to control whisper.cpp's log output, which covers whisper.cpp, its VAD and the ggml backends: `WhisperLog::set()` routes messages to a Rust callback with a `LogLevel`, `WhisperLog::disable()` silences them, and `WhisperLog::reset()` restores the default stderr output. All three are safe to call at any time, including during transcription: whisper.cpp's log hook is unsynchronised global state, so the crate installs its own hook once, before its first call into whisper.cpp, and `WhisperLog` only changes where that hook sends messages. A hook set directly through `whisper-cpp-plus-sys` before the crate's first call is replaced. +- `log` feature: `WhisperLog::use_log_crate()` forwards whisper.cpp log output to the `log` crate with target `whisper_cpp`. +- `WhisperVadProcessor::detect_speech_no_reset()` and `reset_state()` (wrapping `whisper_vad_detect_speech_no_reset` / `whisper_vad_reset_state`) for streaming Silero VAD that keeps its state across chunks, plus `WhisperVadProcessor::WINDOW_SAMPLES` (512 samples per probability). +- `WhisperContext::full_parallel(params, audio, n_processors)`: splits audio into equal chunks, transcribes them concurrently on separate states, and returns the merged `TranscriptionResult` with times on the original timeline. Chunking follows whisper.cpp's `whisper_full_parallel`; in addition, segment times are clamped to their chunk, so whisper reporting a segment end past its audio can no longer push the next chunk's segments later. Words that straddle a chunk boundary may still be cut or misrecognised. +- `WhisperState::n_len()`: mel length of the last transcription on the state (`whisper_n_len_from_state`). ### Fixed @@ -26,6 +38,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Fixed a use-after-free in `FullParams::suppress_regex()`: the regex string was freed immediately after being set, so whisper.cpp read freed memory during transcription. - Fixed `FullParams::prompt_tokens()` storing a borrowed pointer that dangled once the caller's slice was dropped or the params were moved or cloned. The tokens are now copied into the params. - **Breaking:** `WhisperState` result getters now validate segment and token indices instead of passing them to whisper.cpp, which does not bounds-check (out-of-range indices were undefined behaviour). `full_get_segment_text()` and `full_get_token_text()` return `WhisperError::InvalidParameter`, `full_get_token_data()` returns `None`, and the plain-value getters (`full_get_segment_timestamps()`, `full_get_segment_speaker_turn_next()`, `full_n_tokens()`, `full_get_token_id()`, `full_get_token_prob()`) panic, like slice indexing. +- Improved Silero VAD accuracy in `WhisperStreamPcm`. Each 200 ms probe was evaluated from a freshly reset model with a zero-padded partial window, so speech onsets and short words were often misclassified: on `jfk.wav`, 16 of 55 probe decisions differed from a full-file Silero pass, cutting "Ask not" short (transcribed as "Ask, knock!") and splitting a sentence. The model state is now carried across probes (reset only when the stream starts) and only whole 32 ms windows are evaluated, which matches the full-file pass. +- Fixed the `whisper-cpp-plus-sys` documentation on docs.rs, which was generated from out-of-date hand-written stubs: it was missing functions, listed functions that no longer exist, and showed some wrong signatures and types. It now matches the real bindings. + +### Documentation + +- Clarified that `WhisperVadProcessor::detect_speech()` returns whether the computation succeeded, not whether speech was found; speech probabilities come from `get_probs()`. +- Documented that `PcmReaderConfig::buffer_len_ms` drops the oldest samples on overflow, so sources faster than real time (files, in-memory buffers) need a buffer that holds the whole input. ## [0.1.5] - 2026-06-12 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ee81963..a75406f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -97,6 +97,13 @@ cargo clippy -p whisper-cpp-plus --all-targets --features async -- -D warnings cargo test -p whisper-cpp-plus --features async -- --test-threads=1 ``` +For logging (`log` feature) changes: + +```bash +cargo clippy -p whisper-cpp-plus --all-targets --features log -- -D warnings +cargo test -p whisper-cpp-plus --features log --test logging_log_crate -- --test-threads=1 +``` + For macOS or Metal-sensitive changes: ```bash diff --git a/Cargo.toml b/Cargo.toml index 4f940ba..00b3dc8 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,4 +1,4 @@ -# Pinned to whisper.cpp 1.9.4-dev (fork: rmorse/whisper.cpp, branch: stream-pcm, commit de8fb5fd) +# Pinned to whisper.cpp post-v1.9.4 (fork: rmorse/whisper.cpp, branch: stream-pcm, commit de8fb5fd) [workspace] members = ["whisper-cpp-plus-sys", "whisper-cpp-plus", "xtask"] resolver = "2" diff --git a/README.md b/README.md index 3998e4f..46da7c2 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # whisper-cpp-plus -> **Pinned to whisper.cpp 1.9.4-dev** (fork: [`rmorse/whisper.cpp`](https://github.com/rmorse/whisper.cpp), branch: `stream-pcm`, commit [`de8fb5fd`](https://github.com/rmorse/whisper.cpp/commit/de8fb5fda8b25837a2ba0034c8c24223a6fd6c6c), based on upstream `ggml-org/whisper.cpp` `master` after `v1.9.3`) +> **Pinned to whisper.cpp post-v1.9.4** (fork: [`rmorse/whisper.cpp`](https://github.com/rmorse/whisper.cpp), branch: `stream-pcm`, commit [`de8fb5fd`](https://github.com/rmorse/whisper.cpp/commit/de8fb5fda8b25837a2ba0034c8c24223a6fd6c6c), based on upstream `ggml-org/whisper.cpp` `master` after the `v1.9.4` release) Safe Rust bindings for [whisper.cpp](https://github.com/ggerganov/whisper.cpp) with real-time PCM streaming and VAD support — OpenAI's Whisper speech recognition model. @@ -35,6 +35,7 @@ fn main() -> Result<(), Box> { - **Async** — `tokio::spawn_blocking` wrappers (feature = `async`) - **Cross-platform** — Windows (MSVC), Linux, macOS (Intel & Apple Silicon) - **Quantization** — model compression via `WhisperQuantize` (feature = `quantization`) +- **Logging control** — redirect, silence, or forward whisper.cpp's log output via `WhisperLog` (`log` crate integration with feature = `log`) - **Hardware acceleration** — SIMD auto-detected, GPU via feature flags ## Installation @@ -60,6 +61,7 @@ whisper-cpp-plus = { version = "0.1.5", features = ["quantization"] } # Model q whisper-cpp-plus = { version = "0.1.5", features = ["async"] } # Async API whisper-cpp-plus = { version = "0.1.5", features = ["cuda"] } # NVIDIA GPU whisper-cpp-plus = { version = "0.1.5", features = ["metal"] } # macOS GPU +whisper-cpp-plus = { version = "0.1.5", features = ["log"] } # Forward logs to the `log` crate ``` ### CUDA GPU Acceleration @@ -205,7 +207,7 @@ Notes: - `PcmReader` does not decode WAV/MP3, resample audio, or convert stereo to mono. Your `Read` source must already be normalized to the format described by `PcmReaderConfig`. - `WhisperStreamPcm::new(...)` uses fixed-step mode or simple built-in VAD depending on `use_vad`. -- `WhisperStreamPcm::with_vad(...)` uses an explicit `WhisperVadProcessor` (Silero VAD) and is the recommended path when you want Silero-based segmentation. +- `WhisperStreamPcm::with_vad(...)` uses an explicit `WhisperVadProcessor` (Silero VAD) and is the recommended path when you want Silero-based segmentation. Silero's state is carried across probes for the whole stream (it is reset when the stream is created), so each probe is judged in context. - In VAD mode, `no_context` is forced internally to match `stream-pcm.cpp`. - In VAD mode, `run()` emits the next completed speech chunk in chronological order, and callers can usually append those segments directly. - In fixed-step mode, callbacks are produced from overlapping windows, so callers that build a cumulative transcript may need to reconcile repeated text across callbacks. @@ -254,6 +256,30 @@ let result = ctx.transcribe_with_params_enhanced(&audio, params)?; // Automatically retries with higher temperatures if quality thresholds aren't met ``` +**Controlling whisper.cpp log output:** + +whisper.cpp prints model-loading and processing details to stderr by default. `WhisperLog` changes where they go, and can be called at any time (usually once at startup): + +```rust +use whisper_cpp_plus::{LogLevel, WhisperLog}; + +// Silence whisper.cpp entirely +WhisperLog::disable(); + +// ...or route messages to your own handler +WhisperLog::set(|level, message| { + if level >= LogLevel::Warn { + eprintln!("[whisper.cpp {:?}] {}", level, message); + } +}); + +// ...or, with feature = "log", forward to the `log` crate (target "whisper_cpp") +// WhisperLog::use_log_crate(); + +// Restore the default stderr output +WhisperLog::reset(); +``` + More examples in [`whisper-cpp-plus/examples/`](./whisper-cpp-plus/examples/). ## Enhanced Features diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 6a672b0..c3c9f18 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -65,4 +65,4 @@ Zero-copy where possible: `&[f32]` slices passed directly to C++ via `.as_ptr()` ## Build system -`cmake` crate compiles whisper.cpp sources. whisper.cpp vendored as git submodule at `whisper-cpp-plus-sys/whisper.cpp` (excluded from crates.io package - downloaded on demand for consumers). Prebuilt library caching available via `cargo xtask prebuild`. +`cmake` crate compiles whisper.cpp sources. whisper.cpp vendored as git submodule at `whisper-cpp-plus-sys/whisper.cpp`. The crates.io package ships only its public headers and license; consumers download the full pinned source on demand, and docs.rs (no network) generates bindings from the packaged headers. Prebuilt library caching available via `cargo xtask prebuild`. diff --git a/docs/PUBLISHING_GUIDE.md b/docs/PUBLISHING_GUIDE.md index ab73926..0df5e2b 100644 --- a/docs/PUBLISHING_GUIDE.md +++ b/docs/PUBLISHING_GUIDE.md @@ -52,23 +52,24 @@ Update `CHANGELOG.md` with a dated release entry before publishing. ### 3. Test docs.rs Build Locally -docs.rs runs in a **network-isolated container** - it cannot download dependencies at build time. Our `build.rs` detects `DOCS_RS=1` and generates stub bindings instead of compiling whisper.cpp. +docs.rs runs in a **network-isolated container**, so it cannot download whisper.cpp at build time. The sys crate package therefore ships only whisper.cpp's public headers (`whisper.cpp/include/*.h`, `whisper.cpp/ggml/include/*.h`) plus whisper.cpp's `LICENSE`. When `build.rs` sees `DOCS_RS=1` it skips compiling whisper.cpp and runs the normal bindgen step against those headers, so docs.rs gets exact bindings with nothing to maintain by hand. The docs.rs image includes libclang (`clang`, `libclang-dev` in [crates-build-env](https://github.com/rust-lang/crates-build-env)). -**Test the stub bindings work:** +Regular builds of the published crate ignore the packaged headers: `build.rs` only uses the bundled `whisper.cpp/` directory when it contains the full source tree (`CMakeLists.txt` and `src/whisper.cpp`), otherwise it downloads the pinned commit. -```bash -# Clean and rebuild with DOCS_RS simulation -export DOCS_RS=1 -cargo clean -p whisper-cpp-plus-sys -cargo check -p whisper-cpp-plus +**Simulate docs.rs against the packaged crate** (headers only, no network): -# Test docs generation -cargo doc -p whisper-cpp-plus --no-deps +```bash +cargo package -p whisper-cpp-plus-sys --no-verify +mkdir -p /tmp/wcp-docsrs && tar -xzf target/package/whisper-cpp-plus-sys-X.Y.Z.crate -C /tmp/wcp-docsrs +DOCS_RS=1 cargo doc --offline --no-deps \ + --manifest-path /tmp/wcp-docsrs/whisper-cpp-plus-sys-X.Y.Z/Cargo.toml \ + --target-dir /tmp/wcp-docsrs/target + +# The high-level crate, as docs.rs builds it (all features): +DOCS_RS=1 cargo doc --offline --no-deps -p whisper-cpp-plus --all-features --target-dir /tmp/wcp-docsrs/target ``` -If this fails, the stub bindings in `whisper-cpp-plus-sys/build.rs` (`generate_stub_bindings()`) need updating to include missing FFI symbols. - -Unset `DOCS_RS` before running normal tests again. +On Windows, GNU tar needs `--force-local` for paths with a drive letter. Use a separate `--target-dir` so the `DOCS_RS` build script run doesn't invalidate your normal build cache. ### 4. Run Tests @@ -146,24 +147,18 @@ After publishing, monitor the docs.rs build: 1. Check build queue: https://docs.rs/releases/queue 2. View build status: https://docs.rs/crate/whisper-cpp-plus/VERSION/builds -3. If build fails, check logs and fix stub bindings +3. If build fails, check the logs ### Common docs.rs Failures | Error | Cause | Fix | |-------|-------|-----| -| DNS resolution failed | Network access attempted | Ensure `DOCS_RS` check in build.rs | -| Cannot find function X | Missing stub binding | Add function to `generate_stub_bindings()` | -| Type mismatch | Stub signature wrong | Match stub to actual usage in high-level crate | -| Inner attribute not permitted | `#![allow(...)]` in included file | Remove inner attrs from stub bindings | - -## Stub Bindings Maintenance - -When adding new FFI functions to the high-level crate, also add stubs: +| DNS resolution failed | Network access attempted | Ensure the `DOCS_RS` early return in `build.rs` runs before any download or CMake step | +| `whisper.h not found` | Headers missing from the package | Check the `include` list in `whisper-cpp-plus-sys/Cargo.toml` and `cargo package -p whisper-cpp-plus-sys --list` | +| `'
.h' file not found` | A packaged header includes a file outside the packaged directories | Add the directory to the sys crate's `include` list | +| Unable to find libclang | docs.rs image changed | Check [crates-build-env](https://github.com/rust-lang/crates-build-env) and open an issue there | -1. Add function to `generate_stub_bindings()` in `whisper-cpp-plus-sys/build.rs` -2. Match the signature to how the high-level code calls it -3. Test with `DOCS_RS=1 cargo check -p whisper-cpp-plus` +New FFI functions need no docs.rs-specific work: bindings are generated from the same headers everywhere. ## Yanking Bad Releases diff --git a/whisper-cpp-plus-sys/Cargo.toml b/whisper-cpp-plus-sys/Cargo.toml index 80afb0e..91adee8 100644 --- a/whisper-cpp-plus-sys/Cargo.toml +++ b/whisper-cpp-plus-sys/Cargo.toml @@ -8,7 +8,17 @@ license.workspace = true repository.workspace = true description = "Low-level FFI bindings for whisper.cpp" readme = "README.md" -exclude = ["whisper.cpp"] +# Only whisper.cpp's public headers (and its license) are packaged, so docs.rs, which has no +# network access, can generate exact bindings. Regular builds download the full pinned source. +include = [ + "/build.rs", + "/cuda_detect.rs", + "/src/**", + "/README.md", + "/whisper.cpp/LICENSE", + "/whisper.cpp/include/*.h", + "/whisper.cpp/ggml/include/*.h", +] [dependencies] diff --git a/whisper-cpp-plus-sys/README.md b/whisper-cpp-plus-sys/README.md index c5f73f8..9e94bb0 100644 --- a/whisper-cpp-plus-sys/README.md +++ b/whisper-cpp-plus-sys/README.md @@ -1,6 +1,6 @@ # whisper-cpp-plus-sys -> **Pinned to whisper.cpp 1.9.4-dev** (fork: [`rmorse/whisper.cpp`](https://github.com/rmorse/whisper.cpp), branch: `stream-pcm`, commit [`de8fb5fd`](https://github.com/rmorse/whisper.cpp/commit/de8fb5fda8b25837a2ba0034c8c24223a6fd6c6c), based on upstream `ggml-org/whisper.cpp` `master` after `v1.9.3`) +> **Pinned to whisper.cpp post-v1.9.4** (fork: [`rmorse/whisper.cpp`](https://github.com/rmorse/whisper.cpp), branch: `stream-pcm`, commit [`de8fb5fd`](https://github.com/rmorse/whisper.cpp/commit/de8fb5fda8b25837a2ba0034c8c24223a6fd6c6c), based on upstream `ggml-org/whisper.cpp` `master` after the `v1.9.4` release) Low-level FFI bindings to [whisper.cpp](https://github.com/ggerganov/whisper.cpp) for Rust. diff --git a/whisper-cpp-plus-sys/build.rs b/whisper-cpp-plus-sys/build.rs index f1b3c78..d7e9de1 100644 --- a/whisper-cpp-plus-sys/build.rs +++ b/whisper-cpp-plus-sys/build.rs @@ -4,7 +4,7 @@ use std::path::{Path, PathBuf}; #[path = "cuda_detect.rs"] mod cuda_detect; -/// Pinned commit from rmorse/whisper.cpp (stream-pcm branch, whisper.cpp 1.9.4-dev, based on upstream master after v1.9.3) +/// Pinned commit from rmorse/whisper.cpp (stream-pcm branch, based on upstream master after the v1.9.4 release) const WHISPER_CPP_VERSION: &str = "de8fb5fda8b25837a2ba0034c8c24223a6fd6c6c"; const WHISPER_CPP_REPO: &str = "rmorse/whisper.cpp"; @@ -13,14 +13,19 @@ fn main() { println!("cargo:rerun-if-env-changed=WHISPER_PREBUILT_PATH"); println!("cargo:rerun-if-env-changed=WHISPER_CPP_SOURCE_DIR"); println!("cargo:rerun-if-env-changed=MACOSX_DEPLOYMENT_TARGET"); + println!("cargo:rerun-if-env-changed=DOCS_RS"); for var in &cuda_detect::CUDA_PATH_ENV_VARS { println!("cargo:rerun-if-env-changed={}", var); } - // docs.rs builds in a network-isolated container - skip compilation and generate stubs + // docs.rs builds in a network-isolated container: skip compiling whisper.cpp and generate + // bindings from the public headers packaged with the crate. if env::var("DOCS_RS").is_ok() { - println!("cargo:warning=docs.rs build detected, generating stub bindings only"); - generate_stub_bindings(); + println!( + "cargo:warning=docs.rs build detected, generating bindings from packaged headers only" + ); + let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap()); + generate_bindings(&manifest_dir.join("whisper.cpp")); return; } @@ -40,7 +45,9 @@ fn main() { link_platform_libs(&target_os); link_accelerator_libs(&target_os); build_quantize_wrapper(); - generate_bindings(); + + let out = PathBuf::from(env::var("OUT_DIR").unwrap()); + generate_bindings(&get_whisper_source(&out)); } // --------------------------------------------------------------------------- @@ -181,10 +188,12 @@ fn get_whisper_source(out_dir: &Path) -> PathBuf { ); } - // Check local submodule (inside sys crate for dev) + // Check local submodule (inside sys crate for dev). The published crate only ships the + // public headers under whisper.cpp/, so require the full source tree here. let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap()); let bundled_path = manifest_dir.join("whisper.cpp"); - if bundled_path.join("include/whisper.h").exists() { + if bundled_path.join("CMakeLists.txt").exists() && bundled_path.join("src/whisper.cpp").exists() + { copy_dir_filtered(&bundled_path, &whisper_root); return whisper_root; } @@ -499,316 +508,17 @@ fn link_accelerator_libs(_target_os: &str) { // Bindings // --------------------------------------------------------------------------- -/// Generate stub bindings for docs.rs (network-isolated, can't download whisper.cpp) -fn generate_stub_bindings() { +/// Generate bindings from whisper.cpp's public headers. `whisper_src` only needs the `include/` +/// and `ggml/include/` directories, so this also works on docs.rs with the packaged headers. +fn generate_bindings(whisper_src: &Path) { let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap()); - let stub_bindings = r#" -// Stub bindings for docs.rs documentation build. -// This crate requires whisper.cpp which cannot be built in docs.rs's sandbox. -// For actual usage, build locally or see the repository. - -pub type whisper_context = core::ffi::c_void; -pub type whisper_state = core::ffi::c_void; -pub type whisper_token = i32; -pub type whisper_pos = i64; - -#[repr(C)] -#[derive(Debug, Copy, Clone, Default)] -pub struct whisper_context_params { - pub use_gpu: bool, - pub flash_attn: bool, - pub gpu_device: core::ffi::c_int, - pub dtw_token_timestamps: bool, - pub dtw_aheads_preset: core::ffi::c_int, - pub dtw_n_top: core::ffi::c_int, - pub dtw_aheads: whisper_aheads, - pub dtw_mem_size: usize, -} - -#[repr(C)] -#[derive(Debug, Copy, Clone, Default)] -pub struct whisper_aheads { - pub n_heads: usize, - pub heads: *const whisper_ahead, -} - -#[repr(C)] -#[derive(Debug, Copy, Clone, Default)] -pub struct whisper_ahead { - pub n_text_layer: core::ffi::c_int, - pub n_head: core::ffi::c_int, -} - -#[repr(C)] -#[derive(Debug, Copy, Clone)] -pub struct whisper_full_params { - pub strategy: core::ffi::c_int, - pub n_threads: core::ffi::c_int, - pub n_max_text_ctx: core::ffi::c_int, - pub offset_ms: core::ffi::c_int, - pub duration_ms: core::ffi::c_int, - pub translate: bool, - pub no_context: bool, - pub no_timestamps: bool, - pub single_segment: bool, - pub print_special: bool, - pub print_progress: bool, - pub print_realtime: bool, - pub print_timestamps: bool, - pub token_timestamps: bool, - pub thold_pt: f32, - pub thold_ptsum: f32, - pub max_len: core::ffi::c_int, - pub split_on_word: bool, - pub max_tokens: core::ffi::c_int, - pub debug_mode: bool, - pub audio_ctx: core::ffi::c_int, - pub tdrz_enable: bool, - pub suppress_regex: *const core::ffi::c_char, - pub initial_prompt: *const core::ffi::c_char, - pub prompt_tokens: *const whisper_token, - pub prompt_n_tokens: core::ffi::c_int, - pub language: *const core::ffi::c_char, - pub detect_language: bool, - pub suppress_blank: bool, - pub suppress_nst: bool, - pub temperature: f32, - pub max_initial_ts: f32, - pub length_penalty: f32, - pub temperature_inc: f32, - pub entropy_thold: f32, - pub logprob_thold: f32, - pub no_speech_thold: f32, - pub greedy: whisper_full_params__bindgen_ty_1, - pub beam_search: whisper_full_params__bindgen_ty_2, - pub new_segment_callback: Option, - pub new_segment_callback_user_data: *mut core::ffi::c_void, - pub progress_callback: Option, - pub progress_callback_user_data: *mut core::ffi::c_void, - pub encoder_begin_callback: Option, - pub encoder_begin_callback_user_data: *mut core::ffi::c_void, - pub abort_callback: Option, - pub abort_callback_user_data: *mut core::ffi::c_void, - pub logits_filter_callback: Option, - pub logits_filter_callback_user_data: *mut core::ffi::c_void, - pub grammar_rules: *const *const core::ffi::c_void, - pub n_grammar_rules: usize, - pub i_start_rule: usize, - pub grammar_penalty: f32, -} - -#[repr(C)] -#[derive(Debug, Copy, Clone, Default)] -pub struct whisper_full_params__bindgen_ty_1 { - pub best_of: core::ffi::c_int, -} - -#[repr(C)] -#[derive(Debug, Copy, Clone, Default)] -pub struct whisper_full_params__bindgen_ty_2 { - pub beam_size: core::ffi::c_int, - pub patience: f32, -} - -pub const WHISPER_SAMPLE_RATE: u32 = 16000; -pub const WHISPER_N_FFT: u32 = 400; -pub const WHISPER_HOP_LENGTH: u32 = 160; -pub const WHISPER_CHUNK_SIZE: u32 = 30; - -// Sampling strategy enum -pub const whisper_sampling_strategy_WHISPER_SAMPLING_GREEDY: core::ffi::c_int = 0; -pub const whisper_sampling_strategy_WHISPER_SAMPLING_BEAM_SEARCH: core::ffi::c_int = 1; - -// Stub function declarations (not callable, just for docs) -extern "C" { - // Context initialization/cleanup - pub fn whisper_init_from_file_with_params( - path: *const core::ffi::c_char, - params: whisper_context_params, - ) -> *mut whisper_context; - pub fn whisper_init_from_buffer_with_params( - buffer: *const core::ffi::c_void, - buffer_size: usize, - params: whisper_context_params, - ) -> *mut whisper_context; - pub fn whisper_free(ctx: *mut whisper_context); - pub fn whisper_init_state(ctx: *mut whisper_context) -> *mut whisper_state; - pub fn whisper_free_state(state: *mut whisper_state); - pub fn whisper_ctx_init_openvino_encoder( - ctx: *mut whisper_context, - model_path: *const core::ffi::c_char, - device: *const core::ffi::c_char, - cache_dir: *const core::ffi::c_char, - ) -> core::ffi::c_int; - - // Context info - pub fn whisper_context_default_params() -> whisper_context_params; - pub fn whisper_n_vocab(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_n_audio_ctx(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_n_text_ctx(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_n_audio_state(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_n_text_state(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_n_text_head(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_n_text_layer(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_n_mels(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_n_len(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_n_len_from_state(state: *mut whisper_state) -> core::ffi::c_int; - pub fn whisper_is_multilingual(ctx: *mut whisper_context) -> core::ffi::c_int; - - // Language - pub fn whisper_lang_max_id() -> core::ffi::c_int; - pub fn whisper_lang_id(lang: *const core::ffi::c_char) -> core::ffi::c_int; - pub fn whisper_lang_str(id: core::ffi::c_int) -> *const core::ffi::c_char; - pub fn whisper_lang_str_full(id: core::ffi::c_int) -> *const core::ffi::c_char; - pub fn whisper_lang_auto_detect( - ctx: *mut whisper_context, - offset_ms: core::ffi::c_int, - n_threads: core::ffi::c_int, - lang_probs: *mut f32, - ) -> core::ffi::c_int; - pub fn whisper_lang_auto_detect_with_state( - ctx: *mut whisper_context, - state: *mut whisper_state, - offset_ms: core::ffi::c_int, - n_threads: core::ffi::c_int, - lang_probs: *mut f32, - ) -> core::ffi::c_int; - - // Transcription - pub fn whisper_full_default_params(strategy: core::ffi::c_int) -> whisper_full_params; - pub fn whisper_full( - ctx: *mut whisper_context, - params: whisper_full_params, - samples: *const f32, - n_samples: core::ffi::c_int, - ) -> core::ffi::c_int; - pub fn whisper_full_with_state( - ctx: *mut whisper_context, - state: *mut whisper_state, - params: whisper_full_params, - samples: *const f32, - n_samples: core::ffi::c_int, - ) -> core::ffi::c_int; - pub fn whisper_full_parallel( - ctx: *mut whisper_context, - params: whisper_full_params, - samples: *const f32, - n_samples: core::ffi::c_int, - n_processors: core::ffi::c_int, - ) -> core::ffi::c_int; - - // Segment results - pub fn whisper_full_lang_id(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_full_lang_id_from_state(state: *mut whisper_state) -> core::ffi::c_int; - pub fn whisper_full_n_segments(ctx: *mut whisper_context) -> core::ffi::c_int; - pub fn whisper_full_n_segments_from_state(state: *mut whisper_state) -> core::ffi::c_int; - pub fn whisper_full_get_segment_t0(ctx: *mut whisper_context, i_segment: core::ffi::c_int) -> i64; - pub fn whisper_full_get_segment_t0_from_state(state: *mut whisper_state, i_segment: core::ffi::c_int) -> i64; - pub fn whisper_full_get_segment_t1(ctx: *mut whisper_context, i_segment: core::ffi::c_int) -> i64; - pub fn whisper_full_get_segment_t1_from_state(state: *mut whisper_state, i_segment: core::ffi::c_int) -> i64; - pub fn whisper_full_get_segment_text(ctx: *mut whisper_context, i_segment: core::ffi::c_int) -> *const core::ffi::c_char; - pub fn whisper_full_get_segment_text_from_state(state: *mut whisper_state, i_segment: core::ffi::c_int) -> *const core::ffi::c_char; - pub fn whisper_full_get_segment_speaker_turn_next(ctx: *mut whisper_context, i_segment: core::ffi::c_int) -> bool; - pub fn whisper_full_get_segment_speaker_turn_next_from_state(state: *mut whisper_state, i_segment: core::ffi::c_int) -> bool; - pub fn whisper_full_get_segment_no_speech_prob(ctx: *mut whisper_context, i_segment: core::ffi::c_int) -> f32; - pub fn whisper_full_get_segment_no_speech_prob_from_state(state: *mut whisper_state, i_segment: core::ffi::c_int) -> f32; - - // Token results - pub fn whisper_full_n_tokens(ctx: *mut whisper_context, i_segment: core::ffi::c_int) -> core::ffi::c_int; - pub fn whisper_full_n_tokens_from_state(state: *mut whisper_state, i_segment: core::ffi::c_int) -> core::ffi::c_int; - pub fn whisper_full_get_token_text(ctx: *mut whisper_context, i_segment: core::ffi::c_int, i_token: core::ffi::c_int) -> *const core::ffi::c_char; - pub fn whisper_full_get_token_text_from_state(ctx: *mut whisper_context, state: *mut whisper_state, i_segment: core::ffi::c_int, i_token: core::ffi::c_int) -> *const core::ffi::c_char; - pub fn whisper_full_get_token_id(ctx: *mut whisper_context, i_segment: core::ffi::c_int, i_token: core::ffi::c_int) -> whisper_token; - pub fn whisper_full_get_token_id_from_state(state: *mut whisper_state, i_segment: core::ffi::c_int, i_token: core::ffi::c_int) -> whisper_token; - pub fn whisper_full_get_token_p(ctx: *mut whisper_context, i_segment: core::ffi::c_int, i_token: core::ffi::c_int) -> f32; - pub fn whisper_full_get_token_p_from_state(state: *mut whisper_state, i_segment: core::ffi::c_int, i_token: core::ffi::c_int) -> f32; - pub fn whisper_full_get_token_data(ctx: *mut whisper_context, i_segment: core::ffi::c_int, i_token: core::ffi::c_int) -> whisper_token_data; - pub fn whisper_full_get_token_data_from_state(state: *mut whisper_state, i_segment: core::ffi::c_int, i_token: core::ffi::c_int) -> whisper_token_data; - - // Token utilities - pub fn whisper_token_to_str(ctx: *mut whisper_context, token: whisper_token) -> *const core::ffi::c_char; - pub fn whisper_token_eot(ctx: *mut whisper_context) -> whisper_token; - pub fn whisper_token_sot(ctx: *mut whisper_context) -> whisper_token; - pub fn whisper_token_prev(ctx: *mut whisper_context) -> whisper_token; - pub fn whisper_token_solm(ctx: *mut whisper_context) -> whisper_token; - pub fn whisper_token_not(ctx: *mut whisper_context) -> whisper_token; - pub fn whisper_token_beg(ctx: *mut whisper_context) -> whisper_token; - pub fn whisper_token_lang(ctx: *mut whisper_context, lang_id: core::ffi::c_int) -> whisper_token; - pub fn whisper_token_translate(ctx: *mut whisper_context) -> whisper_token; - pub fn whisper_token_transcribe(ctx: *mut whisper_context) -> whisper_token; - - // Timing - pub fn whisper_print_timings(ctx: *mut whisper_context); - pub fn whisper_reset_timings(ctx: *mut whisper_context); - pub fn whisper_print_system_info() -> *const core::ffi::c_char; - - // VAD - pub fn whisper_vad_init(ctx: *mut whisper_context, model_path: *const core::ffi::c_char) -> *mut whisper_vad_context; - pub fn whisper_vad_init_from_buffer(ctx: *mut whisper_context, buffer: *const core::ffi::c_void, buffer_size: usize) -> *mut whisper_vad_context; - pub fn whisper_vad_init_with_params(model_path: *const core::ffi::c_char, params: whisper_vad_context_params) -> *mut whisper_vad_context; - pub fn whisper_vad_init_from_buffer_with_params(buffer: *const core::ffi::c_void, buffer_size: usize, params: whisper_vad_context_params) -> *mut whisper_vad_context; - pub fn whisper_vad_init_from_file_with_params(model_path: *const core::ffi::c_char, params: whisper_vad_context_params) -> *mut whisper_vad_context; - pub fn whisper_vad_free(vad_ctx: *mut whisper_vad_context); - pub fn whisper_vad_default_params() -> whisper_vad_params; - pub fn whisper_vad_default_context_params() -> whisper_vad_context_params; - pub fn whisper_vad_detect_speech(vad_ctx: *mut whisper_vad_context, samples: *const f32, n_samples: core::ffi::c_int) -> bool; - pub fn whisper_vad_n_probs(vad_ctx: *mut whisper_vad_context) -> core::ffi::c_int; - pub fn whisper_vad_probs(vad_ctx: *mut whisper_vad_context) -> *const f32; - pub fn whisper_vad_segments_from_probs(vad_ctx: *mut whisper_vad_context, params: whisper_vad_params) -> *mut whisper_vad_segments; - pub fn whisper_vad_segments_from_samples(vad_ctx: *mut whisper_vad_context, params: whisper_vad_params, samples: *const f32, n_samples: core::ffi::c_int) -> *mut whisper_vad_segments; - pub fn whisper_vad_segments_free(segments: *mut whisper_vad_segments); - pub fn whisper_vad_free_segments(segments: *mut whisper_vad_segments); - pub fn whisper_vad_segments_n_segments(segments: *mut whisper_vad_segments) -> core::ffi::c_int; - pub fn whisper_vad_segments_get_segment_t0(segments: *mut whisper_vad_segments, i_segment: core::ffi::c_int) -> f32; - pub fn whisper_vad_segments_get_segment_t1(segments: *mut whisper_vad_segments, i_segment: core::ffi::c_int) -> f32; -} - -// Additional type stubs -pub type whisper_vad_context = core::ffi::c_void; -pub type whisper_vad_segments = core::ffi::c_void; - -#[repr(C)] -#[derive(Debug, Copy, Clone, Default)] -pub struct whisper_vad_context_params { - pub use_gpu: bool, - pub gpu_device: core::ffi::c_int, - pub n_threads: core::ffi::c_int, -} - -#[repr(C)] -#[derive(Debug, Copy, Clone, Default)] -pub struct whisper_vad_params { - pub threshold: f32, - pub min_speech_duration_ms: core::ffi::c_int, - pub min_silence_duration_ms: core::ffi::c_int, - pub max_speech_duration_s: f32, - pub speech_pad_ms: core::ffi::c_int, - pub samples_overlap: f32, -} - -#[repr(C)] -#[derive(Debug, Copy, Clone, Default)] -pub struct whisper_token_data { - pub id: whisper_token, - pub tid: whisper_token, - pub p: f32, - pub plog: f32, - pub pt: f32, - pub ptsum: f32, - pub t0: i64, - pub t1: i64, - pub t_dtw: i64, - pub vlen: f32, -} -"#; - std::fs::write(out_dir.join("bindings.rs"), stub_bindings) - .expect("Failed to write stub bindings"); -} - -fn generate_bindings() { - let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap()); - let whisper_src = get_whisper_source(&out_dir); let header = whisper_src.join("include/whisper.h"); + if !header.exists() { + panic!( + "whisper.h not found at {}; whisper.cpp headers are missing", + header.display() + ); + } println!("cargo:rerun-if-changed={}", header.display()); diff --git a/whisper-cpp-plus/Cargo.toml b/whisper-cpp-plus/Cargo.toml index 0a44088..fc0bbfa 100644 --- a/whisper-cpp-plus/Cargo.toml +++ b/whisper-cpp-plus/Cargo.toml @@ -19,6 +19,7 @@ flate2 = "1.0" # Optional dependencies tokio = { version = "1.35", features = ["full"], optional = true } +log = { version = "0.4", optional = true } [dev-dependencies] tokio = { version = "1.35", features = ["full"] } @@ -33,6 +34,7 @@ cuda = ["whisper-cpp-plus-sys/cuda"] metal = ["whisper-cpp-plus-sys/metal"] openblas = ["whisper-cpp-plus-sys/openblas"] async = ["tokio"] +log = ["dep:log"] [[example]] name = "basic" diff --git a/whisper-cpp-plus/README.md b/whisper-cpp-plus/README.md index c128bc7..1103131 100644 --- a/whisper-cpp-plus/README.md +++ b/whisper-cpp-plus/README.md @@ -1,6 +1,6 @@ # whisper-cpp-plus -> **Pinned to whisper.cpp 1.9.4-dev** (fork: [`rmorse/whisper.cpp`](https://github.com/rmorse/whisper.cpp), branch: `stream-pcm`, commit [`de8fb5fd`](https://github.com/rmorse/whisper.cpp/commit/de8fb5fda8b25837a2ba0034c8c24223a6fd6c6c), based on upstream `ggml-org/whisper.cpp` `master` after `v1.9.3`) +> **Pinned to whisper.cpp post-v1.9.4** (fork: [`rmorse/whisper.cpp`](https://github.com/rmorse/whisper.cpp), branch: `stream-pcm`, commit [`de8fb5fd`](https://github.com/rmorse/whisper.cpp/commit/de8fb5fda8b25837a2ba0034c8c24223a6fd6c6c), based on upstream `ggml-org/whisper.cpp` `master` after the `v1.9.4` release) Safe Rust bindings for [whisper.cpp](https://github.com/ggerganov/whisper.cpp) with **real-time PCM streaming** and VAD support. @@ -98,6 +98,7 @@ while let Some(segments) = stream.process_step()? { | `metal` | Apple Metal acceleration (macOS) | | `openblas` | OpenBLAS acceleration (Linux) | | `async` | Async transcription API via tokio | +| `log` | Forward whisper.cpp log output to the `log` crate (`WhisperLog::use_log_crate`) | Enable in `Cargo.toml`: ```toml @@ -113,6 +114,7 @@ whisper-cpp-plus = { version = "0.1.5", features = ["cuda"] } - **VAD** — `WhisperVadProcessor` for Silero-based voice activity detection. Use this (or `WhisperStreamPcm::with_vad` / the enhanced VAD) rather than whisper.cpp's built-in `whisper_full` VAD, which does not run for the per-state transcription this crate uses ([ggml-org/whisper.cpp#3423](https://github.com/ggml-org/whisper.cpp/pull/3423)) - **Enhanced** — Temperature fallback + enhanced VAD aggregation for improved quality - **Quantization** — `WhisperQuantize` for model compression (feature = `quantization`) +- **Logging** — `WhisperLog` to redirect, silence, or forward whisper.cpp's log output (`log` crate integration with feature = `log`) ## Examples diff --git a/whisper-cpp-plus/src/context.rs b/whisper-cpp-plus/src/context.rs index f92592d..6854572 100644 --- a/whisper-cpp-plus/src/context.rs +++ b/whisper-cpp-plus/src/context.rs @@ -31,8 +31,11 @@ impl WhisperContext { let c_path = std::ffi::CString::new(path_str)?; + // All transcription runs on explicit `WhisperState`s, so don't allocate whisper.cpp's + // default state (KV caches and compute buffers) for the context. + crate::logging::ensure_installed(); let ptr = unsafe { - ffi::whisper_init_from_file_with_params( + ffi::whisper_init_from_file_with_params_no_state( c_path.as_ptr(), ffi::whisper_context_default_params(), ) @@ -48,8 +51,9 @@ impl WhisperContext { } pub fn new_from_buffer(buffer: &[u8]) -> Result { + crate::logging::ensure_installed(); let ptr = unsafe { - ffi::whisper_init_from_buffer_with_params( + ffi::whisper_init_from_buffer_with_params_no_state( buffer.as_ptr() as *mut std::os::raw::c_void, buffer.len(), ffi::whisper_context_default_params(), @@ -82,10 +86,6 @@ impl WhisperContext { pub fn n_text_ctx(&self) -> i32 { unsafe { ffi::whisper_n_text_ctx(self.ptr.0) } } - - pub fn n_len(&self) -> i32 { - unsafe { ffi::whisper_n_len(self.ptr.0) } - } } impl Clone for WhisperContext { diff --git a/whisper-cpp-plus/src/lib.rs b/whisper-cpp-plus/src/lib.rs index 4955853..08c96ce 100644 --- a/whisper-cpp-plus/src/lib.rs +++ b/whisper-cpp-plus/src/lib.rs @@ -49,6 +49,7 @@ mod context; mod error; +mod logging; mod params; mod state; mod stream; @@ -67,6 +68,7 @@ mod async_api; pub use context::WhisperContext; pub use error::{Result, WhisperError}; +pub use logging::{LogLevel, WhisperLog}; pub use params::{FullParams, SamplingStrategy, TranscriptionParams, TranscriptionParamsBuilder}; #[cfg(feature = "quantization")] pub use quantize::{QuantizationType, QuantizeError, WhisperQuantize}; @@ -89,6 +91,15 @@ pub use async_api::{AsyncWhisperStream, SharedAsyncStream}; // Re-export the sys crate for advanced users who need lower-level access pub use whisper_cpp_plus_sys; +fn result_from_segments(segments: Vec) -> TranscriptionResult { + let text = segments + .iter() + .map(|segment| segment.text.as_str()) + .collect::>() + .join(" "); + TranscriptionResult { text, segments } +} + impl WhisperContext { /// Transcribe audio using default parameters /// @@ -158,33 +169,123 @@ impl WhisperContext { ) -> Result { let mut state = WhisperState::new(self)?; state.full(params, audio)?; + Ok(result_from_segments(state.collect_segments()?)) + } - let n_segments = state.full_n_segments(); - let mut segments = Vec::with_capacity(n_segments as usize); - let mut full_text = String::new(); + /// Transcribe audio by splitting it into `n_processors` chunks that are transcribed + /// concurrently (`whisper_full_parallel`). + /// + /// The audio after `params.offset_ms` is split into `n_processors` equal chunks. Each chunk + /// is transcribed on its own [`WhisperState`] in a separate thread, and the segments are + /// merged in order with their times shifted onto the original timeline (a segment never + /// starts before the previous one ends). This follows whisper.cpp's `whisper_full_parallel`. + /// + /// Chunks are not overlapped, so a word that straddles a chunk boundary may be cut or + /// misrecognised. Each chunk uses `params.n_threads` threads, so up to + /// `n_processors * n_threads` threads run at once. With `n_processors == 1`, or audio too + /// short to split, this is a single transcription. + /// + /// # Arguments + /// * `params` - Full parameter configuration, applied to every chunk + /// * `audio` - Audio samples (must be 16kHz mono f32) + /// * `n_processors` - Number of chunks to transcribe concurrently (at least 1) + pub fn full_parallel( + &self, + params: FullParams, + audio: &[f32], + n_processors: usize, + ) -> Result { + if audio.is_empty() { + return Err(WhisperError::InvalidAudioFormat); + } + if n_processors == 0 { + return Err(WhisperError::InvalidParameter( + "n_processors must be at least 1".into(), + )); + } - for i in 0..n_segments { - let text = state.full_get_segment_text(i)?; - let (start_ms, end_ms) = state.full_get_segment_timestamps(i); - let speaker_turn_next = state.full_get_segment_speaker_turn_next(i); + let sample_rate = whisper_cpp_plus_sys::WHISPER_SAMPLE_RATE as usize; + let offset_ms = params.inner.offset_ms.max(0) as usize; + let offset_samples = (sample_rate * offset_ms / 1000).min(audio.len()); + let n_samples_per_processor = (audio.len() - offset_samples) / n_processors; - if i > 0 { - full_text.push(' '); + if n_processors == 1 || n_samples_per_processor == 0 { + return self.transcribe_with_full_params(audio, params); + } + + let transcribe_chunk = |params: FullParams, chunk: &[f32]| -> Result> { + let mut state = WhisperState::new(self)?; + state.full(params, chunk)?; + state.collect_segments() + }; + + // As in whisper.cpp: the first chunk also covers the `offset_ms` lead-in (so whisper + // skips it itself and reports times from the start of `audio`); later chunks start + // after it and are transcribed without an offset. + let chunk_results: Vec>> = std::thread::scope(|scope| { + let workers: Vec<_> = (1..n_processors) + .map(|i| { + let start = offset_samples + i * n_samples_per_processor; + let end = if i == n_processors - 1 { + audio.len() + } else { + start + n_samples_per_processor + }; + let chunk_params = params + .clone() + .offset_ms(0) + .print_progress(false) + .print_realtime(false); + let transcribe_chunk = &transcribe_chunk; + scope.spawn(move || transcribe_chunk(chunk_params, &audio[start..end])) + }) + .collect(); + + let first = transcribe_chunk( + params.clone().print_realtime(false), + &audio[..offset_samples + n_samples_per_processor], + ); + + std::iter::once(first) + .chain(workers.into_iter().map(|worker| { + worker + .join() + .unwrap_or_else(|panic| std::panic::resume_unwind(panic)) + })) + .collect() + }); + + let samples_to_ms = |samples: usize| samples as i64 * 1000 / sample_rate as i64; + + let mut segments: Vec = Vec::new(); + for (i, chunk_segments) in chunk_results.into_iter().enumerate() { + let chunk_offset_ms = if i == 0 { + 0 + } else { + samples_to_ms(offset_samples + i * n_samples_per_processor) + }; + let chunk_end_ms = if i == n_processors - 1 { + samples_to_ms(audio.len()) + } else { + samples_to_ms(offset_samples + (i + 1) * n_samples_per_processor) + }; + + for mut segment in chunk_segments? { + // whisper can report segment ends past the audio it was given; clamp to the + // chunk so an overrun can't push the next chunk's segments later (whisper.cpp's + // own merge doesn't do this). + segment.start_ms = (segment.start_ms + chunk_offset_ms).min(chunk_end_ms); + segment.end_ms = (segment.end_ms + chunk_offset_ms).min(chunk_end_ms); + // Keep segments from overlapping across chunk boundaries. + if let Some(previous) = segments.last() { + segment.start_ms = segment.start_ms.max(previous.end_ms); + } + segment.end_ms = segment.end_ms.max(segment.start_ms); + segments.push(segment); } - full_text.push_str(&text); - - segments.push(Segment { - start_ms, - end_ms, - text, - speaker_turn_next, - }); } - Ok(TranscriptionResult { - text: full_text, - segments, - }) + Ok(result_from_segments(segments)) } /// Create a new state for manual transcription control diff --git a/whisper-cpp-plus/src/logging.rs b/whisper-cpp-plus/src/logging.rs new file mode 100644 index 0000000..54ab583 --- /dev/null +++ b/whisper-cpp-plus/src/logging.rs @@ -0,0 +1,248 @@ +//! Control over whisper.cpp's log output. +//! +//! whisper.cpp (including its VAD and the ggml backends) writes log messages to stderr by +//! default. [`WhisperLog`] redirects that output to a Rust callback, silences it, or forwards it +//! to the [`log`](https://docs.rs/log) crate (feature `log`). +//! +//! whisper.cpp's log hook (`whisper_log_set`) is process-global state that whisper.cpp writes +//! and reads without synchronisation, so changing it while another thread is inside whisper.cpp +//! is a data race. The crate therefore installs its own hook exactly once, before its first call +//! into whisper.cpp, and never changes it again. [`WhisperLog`] only changes where that hook +//! sends messages, which is synchronised on the Rust side. + +use std::ffi::{c_char, c_void, CStr}; +use std::io::Write; +use std::panic::{catch_unwind, AssertUnwindSafe}; +use std::sync::atomic::{AtomicU8, Ordering}; +use std::sync::{Arc, Once, RwLock}; +use whisper_cpp_plus_sys as ffi; + +/// Severity of a whisper.cpp log message (`ggml_log_level`). +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum LogLevel { + Debug, + Info, + Warn, + Error, +} + +type LogCallback = Arc; + +// Where `log_trampoline` sends messages. +enum Sink { + // What whisper.cpp does when no log callback has been set: every message goes to stderr. + Stderr, + Discard, + Callback(LogCallback), +} + +// whisper.cpp only ever holds a pointer to `log_trampoline`, which looks the sink up here, so no +// Rust data is ever handed to C. +static SINK: RwLock = RwLock::new(Sink::Stderr); + +static INSTALL: Once = Once::new(); + +// Level of the last message, used for `GGML_LOG_LEVEL_CONT` ("continue previous message"). +static LAST_LEVEL: AtomicU8 = AtomicU8::new(LogLevel::Info as u8); + +/// Configures where whisper.cpp's log output goes. +/// +/// This covers whisper.cpp, its VAD, and the ggml backends it initialises. Output goes to +/// stderr until one of these functions is called. They are safe to call at any time, including +/// while other threads are transcribing. +/// +/// The crate installs its own hook with `whisper_log_set` before its first call into +/// whisper.cpp; a hook set earlier by calling `whisper_log_set` directly through +/// `whisper-cpp-plus-sys` is replaced at that point. +pub struct WhisperLog; + +impl WhisperLog { + /// Sends whisper.cpp log messages to `callback` instead of stderr. + /// + /// Messages are passed without their trailing newline; empty messages are skipped. The + /// callback may be called from any thread, including whisper.cpp's worker threads. A panic + /// inside the callback is caught and the message dropped, since unwinding into C is not + /// allowed. + pub fn set(callback: F) + where + F: Fn(LogLevel, &str) + Send + Sync + 'static, + { + set_sink(Sink::Callback(Arc::new(callback))); + } + + /// Discards all whisper.cpp log output. + pub fn disable() { + set_sink(Sink::Discard); + } + + /// Restores the default: every message goes to stderr unchanged, as it does when no log + /// callback has been set in whisper.cpp. + pub fn reset() { + set_sink(Sink::Stderr); + } + + /// Forwards whisper.cpp log messages to the [`log`](https://docs.rs/log) crate with target + /// `whisper_cpp`, mapping [`LogLevel`] to the matching `log::Level`. + #[cfg(feature = "log")] + pub fn use_log_crate() { + Self::set(|level, message| { + let level = match level { + LogLevel::Debug => log::Level::Debug, + LogLevel::Info => log::Level::Info, + LogLevel::Warn => log::Level::Warn, + LogLevel::Error => log::Level::Error, + }; + log::log!(target: "whisper_cpp", level, "{}", message); + }); + } +} + +/// Installs the crate's log hook in whisper.cpp, once per process. +/// +/// Every safe entry point that can call into whisper.cpp without going through an existing +/// context or VAD context (constructors, quantization) must call this first. `Once` orders the +/// `whisper_log_set` write before every later crate call into whisper.cpp, so the hook is never +/// written while the crate is inside whisper.cpp. +pub(crate) fn ensure_installed() { + INSTALL + .call_once(|| unsafe { ffi::whisper_log_set(Some(log_trampoline), std::ptr::null_mut()) }); +} + +fn set_sink(sink: Sink) { + ensure_installed(); + *SINK + .write() + .unwrap_or_else(|poisoned| poisoned.into_inner()) = sink; +} + +fn map_level(level: ffi::ggml_log_level) -> Option { + let mapped = match level { + ffi::ggml_log_level_GGML_LOG_LEVEL_DEBUG => LogLevel::Debug, + ffi::ggml_log_level_GGML_LOG_LEVEL_INFO => LogLevel::Info, + ffi::ggml_log_level_GGML_LOG_LEVEL_WARN => LogLevel::Warn, + ffi::ggml_log_level_GGML_LOG_LEVEL_ERROR => LogLevel::Error, + ffi::ggml_log_level_GGML_LOG_LEVEL_CONT => { + return Some(level_from_u8(LAST_LEVEL.load(Ordering::Relaxed))) + } + _ => return None, + }; + LAST_LEVEL.store(mapped as u8, Ordering::Relaxed); + Some(mapped) +} + +fn level_from_u8(value: u8) -> LogLevel { + match value { + v if v == LogLevel::Debug as u8 => LogLevel::Debug, + v if v == LogLevel::Warn as u8 => LogLevel::Warn, + v if v == LogLevel::Error as u8 => LogLevel::Error, + _ => LogLevel::Info, + } +} + +unsafe extern "C" fn log_trampoline( + level: ffi::ggml_log_level, + text: *const c_char, + _user_data: *mut c_void, +) { + let _ = catch_unwind(AssertUnwindSafe(|| { + if text.is_null() { + return; + } + let callback = match &*SINK.read().unwrap_or_else(|poisoned| poisoned.into_inner()) { + Sink::Stderr => { + let mut stderr = std::io::stderr().lock(); + let _ = stderr.write_all(CStr::from_ptr(text).to_bytes()); + let _ = stderr.flush(); + return; + } + Sink::Discard => return, + Sink::Callback(callback) => Arc::clone(callback), + }; + let Some(level) = map_level(level) else { + return; + }; + + let text = CStr::from_ptr(text).to_string_lossy(); + let message = text.trim_end_matches(&['\n', '\r'][..]); + if !message.is_empty() { + callback(level, message); + } + })); +} + +#[cfg(test)] +mod tests { + use super::*; + use std::ffi::CString; + use std::sync::Mutex; + + fn emit(level: ffi::ggml_log_level, text: &str) { + let text = CString::new(text).unwrap(); + unsafe { log_trampoline(level, text.as_ptr(), std::ptr::null_mut()) }; + } + + // Sets the sink without installing the trampoline in whisper.cpp, so other tests' log output + // doesn't reach it. + fn set_sink_for_test(sink: Sink) { + *SINK + .write() + .unwrap_or_else(|poisoned| poisoned.into_inner()) = sink; + } + + // One test so the shared SINK isn't raced by other tests in this module. + #[test] + fn trampoline_maps_levels_trims_and_contains_panics() { + let received: Arc>> = Arc::default(); + let sink = Arc::clone(&received); + set_sink_for_test(Sink::Callback(Arc::new(move |level, message: &str| { + sink.lock().unwrap().push((level, message.to_owned())); + }))); + + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_DEBUG, "debug line\n"); + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_INFO, "info line\n"); + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_WARN, "warn line\r\n"); + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_CONT, "continued"); + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_ERROR, "error line"); + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_INFO, "\n"); + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_NONE, "ignored\n"); + unsafe { + log_trampoline( + ffi::ggml_log_level_GGML_LOG_LEVEL_INFO, + std::ptr::null(), + std::ptr::null_mut(), + ) + }; + + // A panicking callback must not unwind into C. + set_sink_for_test(Sink::Callback(Arc::new(|_, _: &str| { + panic!("callback panicked") + }))); + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_INFO, "boom\n"); + + // Discard: messages are dropped. + set_sink_for_test(Sink::Discard); + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_INFO, "dropped\n"); + + // Stderr: written unchanged, nothing reaches the old callback. + set_sink_for_test(Sink::Stderr); + emit(ffi::ggml_log_level_GGML_LOG_LEVEL_INFO, "stderr line\n"); + + assert_eq!( + *received.lock().unwrap(), + vec![ + (LogLevel::Debug, "debug line".to_owned()), + (LogLevel::Info, "info line".to_owned()), + (LogLevel::Warn, "warn line".to_owned()), + (LogLevel::Warn, "continued".to_owned()), + (LogLevel::Error, "error line".to_owned()), + ] + ); + } + + #[test] + fn log_levels_are_ordered_by_severity() { + assert!(LogLevel::Debug < LogLevel::Info); + assert!(LogLevel::Info < LogLevel::Warn); + assert!(LogLevel::Warn < LogLevel::Error); + } +} diff --git a/whisper-cpp-plus/src/quantize.rs b/whisper-cpp-plus/src/quantize.rs index cbb5a1b..efb483e 100644 --- a/whisper-cpp-plus/src/quantize.rs +++ b/whisper-cpp-plus/src/quantize.rs @@ -263,6 +263,7 @@ impl WhisperQuantize { } // Perform quantization + crate::logging::ensure_installed(); let result = unsafe { ffi::whisper_model_quantize( input_cstr.as_ptr(), @@ -330,6 +331,7 @@ impl WhisperQuantize { let path_cstr = path_to_cstring(path)?; + crate::logging::ensure_installed(); let ftype = unsafe { ffi::whisper_model_get_ftype(path_cstr.as_ptr()) }; if ftype < 0 { diff --git a/whisper-cpp-plus/src/state.rs b/whisper-cpp-plus/src/state.rs index a95bf11..75851e2 100644 --- a/whisper-cpp-plus/src/state.rs +++ b/whisper-cpp-plus/src/state.rs @@ -58,40 +58,20 @@ impl WhisperState { Ok(()) } - pub fn full_parallel( - &mut self, - params: FullParams, - audio: &[f32], - n_processors: i32, - ) -> Result<()> { - if audio.is_empty() { - return Err(WhisperError::InvalidAudioFormat); - } - - if n_processors < 1 { - return Err(WhisperError::InvalidParameter( - "n_processors must be at least 1".into(), - )); - } - - let ret = unsafe { - ffi::whisper_full_parallel( - self._context.0, - params.as_raw(), - audio.as_ptr(), - audio.len() as i32, - n_processors, - ) - }; - - if ret != 0 { - return Err(WhisperError::TranscriptionError(format!( - "whisper_full_parallel returned {}", - ret - ))); - } - - Ok(()) + /// Collects the segments of the last transcription on this state. + pub(crate) fn collect_segments(&self) -> Result> { + (0..self.full_n_segments()) + .map(|i| { + let text = self.full_get_segment_text(i)?; + let (start_ms, end_ms) = self.full_get_segment_timestamps(i); + Ok(Segment { + start_ms, + end_ms, + text, + speaker_turn_next: self.full_get_segment_speaker_turn_next(i), + }) + }) + .collect() } pub fn full_n_segments(&self) -> i32 { @@ -102,6 +82,12 @@ impl WhisperState { unsafe { ffi::whisper_full_lang_id_from_state(self.ptr) } } + /// Returns the length, in mel frames, of the audio from the last transcription on this state + /// (`whisper_n_len_from_state`); 0 before any transcription. + pub fn n_len(&self) -> i32 { + unsafe { ffi::whisper_n_len_from_state(self.ptr) } + } + // The whisper.cpp result getters index their vectors without bounds checks, so every // wrapper validates indices before calling into C. diff --git a/whisper-cpp-plus/src/stream_pcm.rs b/whisper-cpp-plus/src/stream_pcm.rs index 63ddd40..3065763 100644 --- a/whisper-cpp-plus/src/stream_pcm.rs +++ b/whisper-cpp-plus/src/stream_pcm.rs @@ -49,6 +49,11 @@ pub enum PcmFormat { #[derive(Debug, Clone)] pub struct PcmReaderConfig { /// Ring buffer length in milliseconds (maps to `m_len_ms`). + /// + /// When the buffer is full, the oldest unread samples are dropped (see + /// [`PcmReader::dropped_samples`]). That suits live input, but a source that produces audio + /// faster than real time, such as a file or in-memory buffer, can overflow it and lose audio. + /// Size the buffer to hold the whole input in that case. pub buffer_len_ms: i32, /// Sample rate (must be 16000). pub sample_rate: i32, @@ -376,6 +381,10 @@ pub struct WhisperStreamPcmConfig { /// If true, don't carry prompt tokens across inference boundaries. pub no_context: bool, /// VAD probe chunk size in ms. + /// + /// With Silero VAD, audio is evaluated in whole 32 ms windows + /// ([`WhisperVadProcessor::WINDOW_SAMPLES`]); leftover samples carry into the next probe, + /// and a probe that doesn't complete a window keeps the previous speech/silence decision. pub vad_probe_ms: i32, /// Silence duration to end a segment in ms. pub vad_silence_ms: i32, @@ -435,6 +444,11 @@ pub struct WhisperStreamPcm { total_samples: i64, n_iter: i32, + // Silero streaming state: samples not yet fed to the VAD (less than one window) and the + // last speech/silence decision, reused when a probe doesn't complete a window. + vad_carry: Vec, + vad_last_silence: bool, + // VAD pre-computed vad_last_ms: i32, vad_pre_roll_samples: usize, @@ -469,10 +483,15 @@ impl WhisperStreamPcm { params: FullParams, config: &mut WhisperStreamPcmConfig, reader: PcmReader, - vad: Option, + mut vad: Option, ) -> Result { let state = WhisperState::new(ctx)?; + // The processor may have been used before; start this stream from a clean VAD state. + if let Some(vad) = vad.as_mut() { + vad.reset_state(); + } + // Normalize config (matches C++ main) if !config.use_vad { if config.step_ms <= 0 { @@ -530,6 +549,8 @@ impl WhisperStreamPcm { silence_samples: 0, total_samples: 0, n_iter: 0, + vad_carry: Vec::new(), + vad_last_silence: true, vad_last_ms, vad_pre_roll_samples, vad_silence_samples, @@ -678,17 +699,28 @@ impl WhisperStreamPcm { // Determine silence via Silero or simple VAD let silence = if let Some(ref mut vad) = self.vad { - if vad.detect_speech(&pcmf32_new) { - let probs = vad.get_probs(); - let avg = if probs.is_empty() { - 0.0 + // Silero is recurrent: carry its state across probes (reset only at stream start) + // so each probe is judged in context, and feed only whole windows so zero-padding + // never enters the carried state. Resetting per probe, or between utterances, + // would restart the model cold and misclassify speech onsets. + self.vad_carry.extend_from_slice(&pcmf32_new); + let window = WhisperVadProcessor::WINDOW_SAMPLES; + let whole = self.vad_carry.len() / window * window; + if whole > 0 { + self.vad_last_silence = if vad.detect_speech_no_reset(&self.vad_carry[..whole]) { + let probs = vad.get_probs(); + let avg = if probs.is_empty() { + 0.0 + } else { + probs.iter().sum::() / probs.len() as f32 + }; + avg < self.config.vad_thold } else { - probs.iter().sum::() / probs.len() as f32 + true // detect failed → treat as silence }; - avg < self.config.vad_thold - } else { - true // detect failed → treat as silence + self.vad_carry.drain(..whole); } + self.vad_last_silence } else { vad_simple( &pcmf32_new, diff --git a/whisper-cpp-plus/src/vad.rs b/whisper-cpp-plus/src/vad.rs index ea5cbab..b31a2b7 100644 --- a/whisper-cpp-plus/src/vad.rs +++ b/whisper-cpp-plus/src/vad.rs @@ -107,6 +107,13 @@ impl Drop for WhisperVadProcessor { } impl WhisperVadProcessor { + /// Number of samples Silero VAD evaluates per probability (512 samples = 32 ms at 16 kHz). + /// + /// whisper.cpp splits audio into windows of this size and zero-pads a trailing partial + /// window. When streaming with [`detect_speech_no_reset`](Self::detect_speech_no_reset), + /// feed whole windows so padding doesn't leak into the carried state. + pub const WINDOW_SAMPLES: usize = 512; + /// Create a new VAD processor from a model file pub fn new>(model_path: P) -> Result { Self::new_with_params(model_path, VadContextParams::default()) @@ -124,6 +131,7 @@ impl WhisperVadProcessor { let c_path = std::ffi::CString::new(path_str)?; + crate::logging::ensure_installed(); let ctx = unsafe { ffi::whisper_vad_init_from_file_with_params(c_path.as_ptr(), params.to_ffi()) }; @@ -137,7 +145,13 @@ impl WhisperVadProcessor { Ok(Self { ctx }) } - /// Detect speech in audio samples + /// Computes speech probabilities for `samples`, starting from a fresh model state. + /// + /// Returns `true` if the computation succeeded; it does not indicate whether speech was + /// found. Read the per-window probabilities with [`get_probs`](Self::get_probs). + /// + /// The model's recurrent state is reset first, so each call is evaluated independently. + /// For a continuous stream, use [`detect_speech_no_reset`](Self::detect_speech_no_reset). pub fn detect_speech(&mut self, samples: &[f32]) -> bool { if samples.is_empty() { return false; @@ -146,6 +160,31 @@ impl WhisperVadProcessor { unsafe { ffi::whisper_vad_detect_speech(self.ctx, samples.as_ptr(), samples.len() as i32) } } + /// Like [`detect_speech`](Self::detect_speech), but keeps the model's recurrent state from + /// previous calls, so consecutive chunks of a stream are evaluated in context. + /// + /// Feed multiples of [`WINDOW_SAMPLES`](Self::WINDOW_SAMPLES) and carry any remainder into + /// the next call; a partial window is zero-padded and that padding becomes part of the + /// state. Call [`reset_state`](Self::reset_state) between independent streams or utterances. + pub fn detect_speech_no_reset(&mut self, samples: &[f32]) -> bool { + if samples.is_empty() { + return false; + } + + unsafe { + ffi::whisper_vad_detect_speech_no_reset( + self.ctx, + samples.as_ptr(), + samples.len() as i32, + ) + } + } + + /// Resets the model's recurrent state, as at the start of a new stream. + pub fn reset_state(&mut self) { + unsafe { ffi::whisper_vad_reset_state(self.ctx) } + } + /// Get the number of probability values pub fn n_probs(&self) -> i32 { unsafe { ffi::whisper_vad_n_probs(self.ctx) } diff --git a/whisper-cpp-plus/tests/context_no_state.rs b/whisper-cpp-plus/tests/context_no_state.rs new file mode 100644 index 0000000..57c2752 --- /dev/null +++ b/whisper-cpp-plus/tests/context_no_state.rs @@ -0,0 +1,64 @@ +//! `WhisperContext` loads the model without whisper.cpp's default state; states are only +//! allocated by `WhisperState::new`. Observed through whisper.cpp's own allocation log messages, +//! which needs the process-global log hook, so this lives in its own test binary. + +mod common; + +use common::TestModels; +use std::sync::{Arc, Mutex}; +use whisper_cpp_plus::{FullParams, SamplingStrategy, WhisperContext, WhisperLog, WhisperState}; + +#[test] +fn test_context_allocates_no_default_state() { + let Some(model_path) = TestModels::tiny_en() else { + eprintln!("Skipping: model not found. Run `cargo xtask test-setup`"); + return; + }; + + let messages: Arc>> = Arc::default(); + let sink = Arc::clone(&messages); + WhisperLog::set(move |_, message| sink.lock().unwrap().push(message.to_owned())); + + let state_allocations = |messages: &Mutex>| { + messages + .lock() + .unwrap() + .iter() + .filter(|m| m.starts_with("whisper_init_state:")) + .count() + }; + + let ctx = WhisperContext::new(&model_path).expect("Failed to load model"); + assert!( + messages + .lock() + .unwrap() + .iter() + .any(|m| m.contains("loading model")), + "expected model loading messages" + ); + assert_eq!( + state_allocations(&messages), + 0, + "loading a context should not allocate a whisper state" + ); + + let mut state = WhisperState::new(&ctx).expect("Failed to create state"); + assert!( + state_allocations(&messages) > 0, + "creating a WhisperState should allocate one" + ); + + // The context-less path still works end to end, and n_len is per-state. + assert_eq!(state.n_len(), 0); + let audio = vec![0.0f32; 16000]; + state + .full( + FullParams::new(SamplingStrategy::Greedy { best_of: 1 }), + &audio, + ) + .expect("transcription failed"); + assert!(state.n_len() > 0, "n_len should be set after transcribing"); + + WhisperLog::reset(); +} diff --git a/whisper-cpp-plus/tests/integration.rs b/whisper-cpp-plus/tests/integration.rs index a1412cf..f9025a2 100644 --- a/whisper-cpp-plus/tests/integration.rs +++ b/whisper-cpp-plus/tests/integration.rs @@ -144,8 +144,7 @@ fn test_model_info() { assert!(ctx.n_audio_ctx() > 0, "n_audio_ctx should be positive"); assert!(ctx.n_text_ctx() > 0, "n_text_ctx should be positive"); - // Note: n_len() returns mel spectrogram length, which is 0 until audio is processed. - // It's a state property, not a model property - don't test it here. + // n_len (mel length of the last transcription) is per-state: see WhisperState::n_len. // tiny.en model is English-only assert!(!ctx.is_multilingual()); diff --git a/whisper-cpp-plus/tests/logging.rs b/whisper-cpp-plus/tests/logging.rs new file mode 100644 index 0000000..dc446dc --- /dev/null +++ b/whisper-cpp-plus/tests/logging.rs @@ -0,0 +1,90 @@ +//! Log routing is process-global, so these tests live in their own test binary and in a single +//! test function. + +mod common; + +use common::TestModels; +use std::sync::{Arc, Mutex}; +use whisper_cpp_plus::{LogLevel, WhisperContext, WhisperLog}; + +type Captured = Arc>>; + +fn capture_into(captured: &Captured) { + let sink = Arc::clone(captured); + WhisperLog::set(move |level, message| { + sink.lock().unwrap().push((level, message.to_owned())); + }); +} + +#[test] +fn test_log_routing() { + let Some(model_path) = TestModels::tiny_en() else { + eprintln!("Skipping: model not found. Run `cargo xtask test-setup`"); + return; + }; + + let captured: Captured = Arc::default(); + + // A callback receives whisper.cpp's messages from model loading. + capture_into(&captured); + drop(WhisperContext::new(&model_path).expect("Failed to load model")); + { + let messages = captured.lock().unwrap(); + assert!( + messages + .iter() + .any(|(level, message)| *level == LogLevel::Info + && message.contains("loading model")), + "expected an info 'loading model' message, got {:?}", + *messages + ); + assert!( + messages.iter().all(|(_, message)| !message.ends_with('\n')), + "messages should have their trailing newline trimmed" + ); + } + + // disable() silences everything, and the previous callback is no longer called. + let before = captured.lock().unwrap().len(); + WhisperLog::disable(); + drop(WhisperContext::new(&model_path).expect("Failed to load model")); + assert_eq!(captured.lock().unwrap().len(), before); + + // A callback can be set again after disabling. + capture_into(&captured); + drop(WhisperContext::new(&model_path).expect("Failed to load model")); + assert!(captured.lock().unwrap().len() > before); + + // reset() restores whisper.cpp's default stderr output and detaches the callback. + let before = captured.lock().unwrap().len(); + WhisperLog::reset(); + drop(WhisperContext::new(&model_path).expect("Failed to load model")); + assert_eq!(captured.lock().unwrap().len(), before); + + // Changing the routing while another thread is inside whisper.cpp is allowed: it only + // touches Rust-side state, never whisper.cpp's global log hook. (A data race here would + // need a thread sanitizer to detect; this checks the calls are accepted and routing still + // works afterwards.) + let stop = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let toggler = { + let stop = Arc::clone(&stop); + std::thread::spawn(move || { + while !stop.load(std::sync::atomic::Ordering::Relaxed) { + WhisperLog::set(|_, _| {}); + WhisperLog::disable(); + WhisperLog::reset(); + } + }) + }; + for _ in 0..3 { + drop(WhisperContext::new(&model_path).expect("Failed to load model")); + } + stop.store(true, std::sync::atomic::Ordering::Relaxed); + toggler.join().expect("toggler thread panicked"); + + capture_into(&captured); + let before = captured.lock().unwrap().len(); + drop(WhisperContext::new(&model_path).expect("Failed to load model")); + assert!(captured.lock().unwrap().len() > before); + WhisperLog::reset(); +} diff --git a/whisper-cpp-plus/tests/logging_log_crate.rs b/whisper-cpp-plus/tests/logging_log_crate.rs new file mode 100644 index 0000000..39dcbd7 --- /dev/null +++ b/whisper-cpp-plus/tests/logging_log_crate.rs @@ -0,0 +1,59 @@ +//! `WhisperLog::use_log_crate()` (feature `log`). Installs a global `log` logger, so it lives in +//! its own test binary. +#![cfg(feature = "log")] + +mod common; + +use common::TestModels; +use std::sync::Mutex; +use whisper_cpp_plus::{WhisperContext, WhisperLog}; + +struct CapturingLogger { + records: Mutex>, +} + +impl log::Log for CapturingLogger { + fn enabled(&self, _metadata: &log::Metadata) -> bool { + true + } + + fn log(&self, record: &log::Record) { + self.records.lock().unwrap().push(( + record.level(), + record.target().to_owned(), + record.args().to_string(), + )); + } + + fn flush(&self) {} +} + +static LOGGER: CapturingLogger = CapturingLogger { + records: Mutex::new(Vec::new()), +}; + +#[test] +fn test_use_log_crate() { + let Some(model_path) = TestModels::tiny_en() else { + eprintln!("Skipping: model not found. Run `cargo xtask test-setup`"); + return; + }; + + log::set_logger(&LOGGER).expect("logger already set"); + log::set_max_level(log::LevelFilter::Trace); + + WhisperLog::use_log_crate(); + drop(WhisperContext::new(&model_path).expect("Failed to load model")); + WhisperLog::reset(); + + let records = LOGGER.records.lock().unwrap(); + assert!( + records.iter().any(|(level, target, message)| { + *level == log::Level::Info + && target == "whisper_cpp" + && message.contains("loading model") + }), + "expected an info 'loading model' record with target whisper_cpp, got {:?}", + *records + ); +} diff --git a/whisper-cpp-plus/tests/real_audio.rs b/whisper-cpp-plus/tests/real_audio.rs index 6d9712d..ed92fbe 100644 --- a/whisper-cpp-plus/tests/real_audio.rs +++ b/whisper-cpp-plus/tests/real_audio.rs @@ -241,6 +241,157 @@ fn test_state_getters_reject_out_of_range_indices() { })); } +/// Loads the model and jfk.wav, or returns `None` (skip) if either is missing. +fn jfk_fixture() -> Option<(WhisperContext, Vec)> { + let Some(model_path) = find_whisper_model() else { + eprintln!( + "Skipping: model not found. Set WHISPER_TEST_MODEL_DIR or run `cargo xtask test-setup`" + ); + return None; + }; + let Some(audio_path) = find_jfk_audio() else { + eprintln!("Skipping: JFK audio not found. Set WHISPER_TEST_AUDIO_DIR or run `cargo xtask test-setup`"); + return None; + }; + let audio = load_wav_file(&audio_path).expect("Failed to load JFK audio"); + let ctx = WhisperContext::new(&model_path).expect("Failed to load model"); + Some((ctx, audio)) +} + +fn greedy_params() -> FullParams { + FullParams::new(SamplingStrategy::Greedy { best_of: 1 }) +} + +fn assert_segments_ordered(result: &whisper_cpp_plus::TranscriptionResult) { + for (i, segment) in result.segments.iter().enumerate() { + assert!( + segment.start_ms <= segment.end_ms, + "segment {i} ends before it starts: {segment:?}" + ); + if i > 0 { + let previous = &result.segments[i - 1]; + assert!( + segment.start_ms >= previous.end_ms, + "segment {i} overlaps the previous one: {previous:?} then {segment:?}" + ); + } + } +} + +#[test] +fn test_full_parallel_merges_chunks_on_original_timeline() { + let Some((ctx, audio)) = jfk_fixture() else { + return; + }; + let audio_ms = audio.len() as i64 * 1000 / 16000; + let split_ms = audio_ms / 2; + + let result = ctx + .full_parallel(greedy_params(), &audio, 2) + .expect("full_parallel failed"); + println!("full_parallel(2): {:?}", result.segments); + + assert!(!result.segments.is_empty()); + assert_segments_ordered(&result); + + // The second chunk's segments must be shifted past the split point. + assert!( + result.segments.iter().any(|s| s.start_ms >= split_ms), + "no segment starts after the {split_ms} ms split: {:?}", + result.segments + ); + let last = result.segments.last().unwrap(); + assert!( + last.end_ms > audio_ms * 3 / 4 && last.end_ms <= audio_ms, + "last segment ends at {} ms, expected close to (and not past) {audio_ms} ms", + last.end_ms + ); + // Segments from the first chunk are clamped to it, so they can't overrun the split. + for segment in result.segments.iter().filter(|s| s.start_ms < split_ms) { + assert!( + segment.end_ms <= split_ms, + "first-chunk segment overruns the {split_ms} ms split: {segment:?}" + ); + } + + let text = result.text.to_lowercase(); + assert!( + text.contains("americans"), + "missing first-chunk text: {text}" + ); + assert!( + text.contains("your country"), + "missing second-chunk text: {text}" + ); +} + +#[test] +fn test_full_parallel_respects_offset() { + let Some((ctx, audio)) = jfk_fixture() else { + return; + }; + let audio_ms = audio.len() as i64 * 1000 / 16000; + let offset_ms = 2000; + // Chunks split the audio after the offset: the second starts halfway through the rest. + let split_ms = offset_ms + (audio_ms - offset_ms) / 2; + + let result = ctx + .full_parallel(greedy_params().offset_ms(offset_ms as i32), &audio, 2) + .expect("full_parallel failed"); + println!( + "full_parallel(2, offset {offset_ms} ms): {:?}", + result.segments + ); + + assert!(!result.segments.is_empty()); + assert_segments_ordered(&result); + assert!( + result.segments[0].start_ms >= offset_ms - 100, + "first segment starts before the offset: {:?}", + result.segments[0] + ); + assert!( + result.segments.iter().any(|s| s.start_ms >= split_ms), + "no segment starts after the {split_ms} ms split: {:?}", + result.segments + ); +} + +#[test] +fn test_full_parallel_single_chunk_matches_full() { + let Some((ctx, audio)) = jfk_fixture() else { + return; + }; + + let single = ctx + .transcribe_with_full_params(&audio, greedy_params()) + .expect("transcription failed"); + let parallel = ctx + .full_parallel(greedy_params(), &audio, 1) + .expect("full_parallel failed"); + assert_eq!(parallel.text, single.text); + assert_eq!(parallel.segments.len(), single.segments.len()); + + // Audio too short to split into chunks falls back to a single transcription. + let short = &audio[..3]; + let fallback = ctx + .full_parallel(greedy_params(), short, 4) + .expect("full_parallel on short audio failed"); + let expected = ctx + .transcribe_with_full_params(short, greedy_params()) + .expect("transcription failed"); + assert_eq!(fallback.text, expected.text); +} + +#[test] +fn test_full_parallel_rejects_invalid_input() { + let Some((ctx, audio)) = jfk_fixture() else { + return; + }; + assert!(ctx.full_parallel(greedy_params(), &audio, 0).is_err()); + assert!(ctx.full_parallel(greedy_params(), &[], 2).is_err()); +} + #[test] fn test_audio_duration_handling() { let Some(model_path) = find_whisper_model() else { diff --git a/whisper-cpp-plus/tests/stream_pcm_integration.rs b/whisper-cpp-plus/tests/stream_pcm_integration.rs index b1b60b3..65f8f8b 100644 --- a/whisper-cpp-plus/tests/stream_pcm_integration.rs +++ b/whisper-cpp-plus/tests/stream_pcm_integration.rs @@ -46,8 +46,23 @@ fn wav_to_raw_pcm(path: &Path, format: PcmFormat) -> (Vec, usize) { (bytes, n_samples) } +/// Reader buffer length that holds the whole clip, plus a margin. +/// +/// These tests feed an in-memory file far faster than real time. With a buffer shorter than the +/// clip, `PcmReader` overflows and (correctly, for live input) drops the oldest samples, which +/// silently cut the first second of jfk.wav. +fn buffer_len_ms_for(n_samples: usize) -> i32 { + (n_samples * 1000 / 16000) as i32 + 1000 +} + fn check_jfk_keywords(text: &str) { let lower = text.to_lowercase(); + // The opening words go missing if the start of the clip is dropped. + assert!( + lower.contains("and so"), + "Transcript is missing the opening \"And so\": {}", + text + ); let keywords = ["ask", "not", "what", "country", "you"]; let found: Vec<&&str> = keywords.iter().filter(|k| lower.contains(**k)).collect(); println!("Transcript: {}", text); @@ -78,7 +93,7 @@ fn test_stream_pcm_fixed_step_f32() { return; }; - let (raw_bytes, _) = wav_to_raw_pcm(&jfk_path, PcmFormat::F32); + let (raw_bytes, n_samples) = wav_to_raw_pcm(&jfk_path, PcmFormat::F32); let ctx = WhisperContext::new(&model_path).unwrap(); let params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 }) @@ -89,7 +104,7 @@ fn test_stream_pcm_fixed_step_f32() { let reader = PcmReader::new( Box::new(std::io::Cursor::new(raw_bytes)), PcmReaderConfig { - buffer_len_ms: 10000, + buffer_len_ms: buffer_len_ms_for(n_samples), sample_rate: 16000, format: PcmFormat::F32, }, @@ -134,7 +149,7 @@ fn test_stream_pcm_fixed_step_s16() { return; }; - let (raw_bytes, _) = wav_to_raw_pcm(&jfk_path, PcmFormat::S16); + let (raw_bytes, n_samples) = wav_to_raw_pcm(&jfk_path, PcmFormat::S16); let ctx = WhisperContext::new(&model_path).unwrap(); let params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 }) @@ -145,7 +160,7 @@ fn test_stream_pcm_fixed_step_s16() { let reader = PcmReader::new( Box::new(std::io::Cursor::new(raw_bytes)), PcmReaderConfig { - buffer_len_ms: 10000, + buffer_len_ms: buffer_len_ms_for(n_samples), sample_rate: 16000, format: PcmFormat::S16, }, @@ -190,7 +205,7 @@ fn test_stream_pcm_vad_simple() { return; }; - let (raw_bytes, _) = wav_to_raw_pcm(&jfk_path, PcmFormat::F32); + let (raw_bytes, n_samples) = wav_to_raw_pcm(&jfk_path, PcmFormat::F32); let ctx = WhisperContext::new(&model_path).unwrap(); let params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 }).language("en"); @@ -198,7 +213,7 @@ fn test_stream_pcm_vad_simple() { let reader = PcmReader::new( Box::new(std::io::Cursor::new(raw_bytes)), PcmReaderConfig { - buffer_len_ms: 10000, + buffer_len_ms: buffer_len_ms_for(n_samples), sample_rate: 16000, format: PcmFormat::F32, }, @@ -256,7 +271,7 @@ fn test_stream_pcm_vad_silero() { return; }; - let (raw_bytes, _) = wav_to_raw_pcm(&jfk_path, PcmFormat::F32); + let (raw_bytes, n_samples) = wav_to_raw_pcm(&jfk_path, PcmFormat::F32); let ctx = WhisperContext::new(&model_path).unwrap(); let vad = whisper_cpp_plus::WhisperVadProcessor::new(&vad_model_path).unwrap(); @@ -265,7 +280,7 @@ fn test_stream_pcm_vad_silero() { let reader = PcmReader::new( Box::new(std::io::Cursor::new(raw_bytes)), PcmReaderConfig { - buffer_len_ms: 10000, + buffer_len_ms: buffer_len_ms_for(n_samples), sample_rate: 16000, format: PcmFormat::F32, }, @@ -288,14 +303,13 @@ fn test_stream_pcm_vad_silero() { stream .run(|segments, start_ms, end_ms| { segment_count += 1; + let text: String = segments.iter().map(|seg| seg.text.as_str()).collect(); println!( - "Silero VAD segment {}: {}ms-{}ms", - segment_count, start_ms, end_ms + "Silero VAD segment {}: {}ms-{}ms:{}", + segment_count, start_ms, end_ms, text ); - for seg in segments { - all_text.push_str(&seg.text); - all_text.push(' '); - } + all_text.push_str(&text); + all_text.push(' '); }) .expect("WhisperStreamPcm::run with Silero VAD failed"); diff --git a/whisper-cpp-plus/tests/vad_integration.rs b/whisper-cpp-plus/tests/vad_integration.rs index d6958c5..3991c5c 100644 --- a/whisper-cpp-plus/tests/vad_integration.rs +++ b/whisper-cpp-plus/tests/vad_integration.rs @@ -460,3 +460,109 @@ mod rand { }) } } + +/// Average of the full-pass window probabilities whose window starts inside `[start, end)`. +fn reference_probe_avg(full: &[f32], window: usize, start: usize, end: usize) -> f32 { + // Round up without usize::div_ceil (MSRV 1.70). + let first = (start + window - 1) / window; + let last = ((end + window - 1) / window).min(full.len()); + let probs = &full[first.min(last)..last]; + if probs.is_empty() { + 0.0 + } else { + probs.iter().sum::() / probs.len() as f32 + } +} + +#[test] +fn test_streaming_vad_matches_full_pass() { + let Some(vad_path) = find_vad_model() else { + eprintln!("Skipping: VAD model not found. Run `cargo xtask test-setup`"); + return; + }; + let Some(audio_path) = find_jfk_audio() else { + eprintln!("Skipping: JFK audio not found. Run `cargo xtask test-setup`"); + return; + }; + + let window = WhisperVadProcessor::WINDOW_SAMPLES; + let audio = load_wav_16khz_mono(&audio_path).expect("Failed to load audio"); + let audio = &audio[..audio.len() / window * window]; + + // WhisperStreamPcm defaults: 200 ms probes, 0.6 speech threshold. + let probe = 3200; + let thold = 0.6; + + let mut vad = WhisperVadProcessor::new(&vad_path).expect("Failed to load VAD model"); + + // Reference: one pass over the whole file. + assert!(vad.detect_speech(audio)); + let full = vad.get_probs(); + assert_eq!(full.len(), audio.len() / window); + + // Streaming without resets, feeding whole windows and carrying the remainder. + vad.reset_state(); + let mut streamed = Vec::new(); + let mut streamed_probe_avgs = Vec::new(); + let mut carry = Vec::new(); + for chunk in audio.chunks(probe) { + carry.extend_from_slice(chunk); + let whole = carry.len() / window * window; + if whole == 0 { + streamed_probe_avgs.push(None); + continue; + } + assert!(vad.detect_speech_no_reset(&carry[..whole])); + let probs = vad.get_probs(); + streamed_probe_avgs.push(Some(probs.iter().sum::() / probs.len() as f32)); + streamed.extend(probs); + carry.drain(..whole); + } + + // Same LSTM input sequence as the full pass, so the probabilities must match. + assert_eq!(streamed.len(), full.len()); + let max_diff = streamed + .iter() + .zip(&full) + .map(|(a, b)| (a - b).abs()) + .fold(0.0f32, f32::max); + assert!( + max_diff < 1e-4, + "streamed probabilities differ from the full pass by up to {max_diff}" + ); + + // Per-probe decisions: previous reset-per-probe behaviour vs streaming, against the + // full-pass reference. Printed for comparison; only streaming is asserted. + let mut reset_mismatches = 0; + let mut stream_mismatches = 0; + let mut reset_abs_err = 0.0f32; + let mut stream_abs_err = 0.0f32; + let n_probes = audio.chunks(probe).count(); + for (i, chunk) in audio.chunks(probe).enumerate() { + let start = i * probe; + let reference = reference_probe_avg(&full, window, start, start + chunk.len()); + + assert!(vad.detect_speech(chunk)); + let probs = vad.get_probs(); + let reset_avg = probs.iter().sum::() / probs.len() as f32; + reset_abs_err += (reset_avg - reference).abs(); + if (reset_avg < thold) != (reference < thold) { + reset_mismatches += 1; + } + + if let Some(stream_avg) = streamed_probe_avgs[i] { + stream_abs_err += (stream_avg - reference).abs(); + if (stream_avg < thold) != (reference < thold) { + stream_mismatches += 1; + } + } + } + + println!( + "{n_probes} probes of {probe} samples: reset-per-probe mean abs error {:.3}, \ + {reset_mismatches} speech/silence decisions differ from the full pass; \ + streaming mean abs error {:.3}, {stream_mismatches} differ", + reset_abs_err / n_probes as f32, + stream_abs_err / n_probes as f32, + ); +}