Context
Workshop voice transcription is GPU-only by design: gpu_transcription_available() hides the mic unless the build carries a GPU whisper backend, because a CPU pass on the large models stalls a take for tens of seconds. The wired-up backend today is CUDA (promptforge-ws --features cuda, the default). Metal is the natural macOS counterpart and is being added as an opt-in feature chain (whisper-rs/metal).
whisper-rs also exposes a coreml feature (whisper-rs-sys/coreml), which routes the whisper encoder through Core ML so it can run on the Apple Neural Engine instead of (or alongside) the GPU. On Apple Silicon this is reported to be substantially faster than Metal for the encoder, which dominates whisper inference time. That could matter for the workshop's interim loop, which transcribes a sliding window every interval_ms while a take is recording.
What to investigate
- Real-world speedup vs Metal on M-series hardware for the models the workshop actually ships:
ggml-large-v3-turbo (interim) and ggml-large-v3 (final pass), at the 5 to 15 second window sizes the interim loop uses. Is the win large enough to justify the added complexity?
- Model provisioning. Core ML needs a converted encoder model (
*-encoder.mlmodelc) alongside each GGML file. Does the upstream ggerganov/whisper.cpp Hugging Face repo publish these for the models we use, and in what packaging (zip archives)? How does that fit the gateway-cache download flow in [workshop.voice] (interim_source / final_source), which today assumes one file per model?
- First-load latency. Core ML compiles/specializes the model on first load on a given machine, which can take minutes for large models. Where does that cost land in our startup and provisioning flow, and does it need a status-bar note like the model downloads have?
- Fallback behavior. whisper.cpp falls back to the non-CoreML path when the
.mlmodelc is missing. Should coreml builds degrade to plain Metal silently, or surface the difference?
- Feature wiring. Presumably mirrors the CUDA/Metal chain:
promptforge-ws/coreml -> promptforge-gateway/workshop-coreml -> promptforge-ws-server/coreml -> whisper-rs/coreml, plus the gpu_transcription_available() gate.
Outcome
A recommendation: adopt CoreML as the preferred macOS voice backend, keep it as an opt-in alongside Metal, or drop it as not worth the provisioning complexity.
Context
Workshop voice transcription is GPU-only by design:
gpu_transcription_available()hides the mic unless the build carries a GPU whisper backend, because a CPU pass on the large models stalls a take for tens of seconds. The wired-up backend today is CUDA (promptforge-ws --features cuda, the default). Metal is the natural macOS counterpart and is being added as an opt-in feature chain (whisper-rs/metal).whisper-rs also exposes a
coremlfeature (whisper-rs-sys/coreml), which routes the whisper encoder through Core ML so it can run on the Apple Neural Engine instead of (or alongside) the GPU. On Apple Silicon this is reported to be substantially faster than Metal for the encoder, which dominates whisper inference time. That could matter for the workshop's interim loop, which transcribes a sliding window everyinterval_mswhile a take is recording.What to investigate
ggml-large-v3-turbo(interim) andggml-large-v3(final pass), at the 5 to 15 second window sizes the interim loop uses. Is the win large enough to justify the added complexity?*-encoder.mlmodelc) alongside each GGML file. Does the upstreamggerganov/whisper.cppHugging Face repo publish these for the models we use, and in what packaging (zip archives)? How does that fit the gateway-cache download flow in[workshop.voice](interim_source/final_source), which today assumes one file per model?.mlmodelcis missing. Shouldcoremlbuilds degrade to plain Metal silently, or surface the difference?promptforge-ws/coreml->promptforge-gateway/workshop-coreml->promptforge-ws-server/coreml->whisper-rs/coreml, plus thegpu_transcription_available()gate.Outcome
A recommendation: adopt CoreML as the preferred macOS voice backend, keep it as an opt-in alongside Metal, or drop it as not worth the provisioning complexity.