Skip to content

fix(download): include required root-level files without a metadata extension - #918

Merged
Alex-Wengg merged 1 commit into
mainfrom
fix/modelhub-root-required-file
Sep 14, 2026
Merged

fix(download): include required root-level files without a metadata extension#918
Alex-Wengg merged 1 commit into
mainfrom
fix/modelhub-root-required-file

Conversation

@Alex-Wengg

Copy link
Copy Markdown
Member

Summary

ModelHub.repoIncludeRule builds one pattern per required model with a trailing / (bundle directories). For repos without a subPath, a required plain file was admitted only by prefix match or by the .json / .txt allowance, so a required root-level file such as tokenizer.model never matched tokenizer.model/ and the post-download verify pass threw modelNotFound (every run re-listed the repo and failed again).

The rule now also accepts a pattern that equals the file path plus /. No change for subPath repos or for bundle directories.

Test

🤖 Generated with Claude Code

https://claude.ai/code/session_018UHoFANi4DcvU6TzyTPnHH

…xtension

ModelHub.repoIncludeRule builds one pattern per required model with a
trailing "/" (bundle directories). For repos without a subPath, a required
plain file was only admitted by prefix match or by the .json/.txt allowance,
so a required root file such as `tokenizer.model` never matched
"tokenizer.model/" and the post-download verify pass threw modelNotFound.
Accept a pattern that equals the file path plus "/". Regression test added.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UHoFANi4DcvU6TzyTPnHH
Alex-Wengg added a commit that referenced this pull request Sep 13, 2026
… 48 kHz stereo

Swift host for the CoreML conversion of OpenMOSS-Team/MOSS-TTS-Nano-100M
(FluidInference/moss-tts-nano-coreml): Prefill -> (Frame -> CodecStep -> Step)
per 80 ms frame, sampling inside the Frame graph from host uniforms.

- MossTtsNanoTokenizer: SentencePiece BPE with nmt_nfkc normalization, dummy
  prefix and byte fallback; id-exact against the upstream processor on 8 vectors
  (English, Chinese, Japanese, Russian, emoji, whitespace edge cases). The
  PocketTTS SentencePieceTokenizer is unigram and cannot serve this model.
- MossTtsNanoTextChunker: upstream sentence/clause/token-budget splitter and
  inter-chunk pauses. MossTtsNanoPromptBuilder: voice-clone row layout from the
  repo's config.json (template ids, special tokens).
- MossTtsNanoManager: preset voices (en_2, zh_1), cloneVoice(audioURL:) via the
  fp32 codec encoder, synthesize / synthesizeStreaming (AsyncThrowingStream of
  stereo frames), seedable sampling.
- Prefill/Step pinned to CPU+GPU (ANECCompile fails); Frame/CodecStep honour
  the requested compute units.
- CLI: --backend moss-tts-nano with --voice/--clone-voice/--save-voice/
  --voice-file/--seed/--greedy/--tokens-only; stereo WAV writer.
- SentencePieceProto: parse the piece `type` field.
- Docs: Documentation/TTS/MossTtsNano.md, Documentation/Models.md rows.

Depends on the ModelHub root-file include fix (#918): tokenizer.model is a
required root-level file.

Measured (M5 Pro, release CLI, en_2 voice, 9-10 s utterances): RTFx 3.1-3.8x,
first audio 0.22-0.40 s; zh_1 3 s utterance 2.5x. Parakeet WER 0 % on the
27-word phrase, one spurious trailing word on the short one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UHoFANi4DcvU6TzyTPnHH
@github-actions

Copy link
Copy Markdown

PocketTTS Smoke Test ✅

Check Result
Build
Model download
Model load
Synthesis pipeline
Output WAV ✅ (172.5 KB)

Runtime: 0m18s

Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon.

@github-actions

Copy link
Copy Markdown

Parakeet EOU Benchmark Results ✅

Status: Benchmark passed
Chunk Size: 320ms
Files Tested: 100/100

Performance Metrics

Metric Value Description
WER (Avg) 7.03% Average Word Error Rate
WER (Med) 4.17% Median Word Error Rate
RTFx 10.53x Real-time factor (higher = faster)
Total Audio 470.6s Total audio duration processed
Total Time 45.9s Total processing time

Streaming Metrics

Metric Value Description
Avg Chunk Time 0.046s Average chunk processing time
Max Chunk Time 0.092s Maximum chunk processing time
EOU Detections 0 Total End-of-Utterance detections

Test runtime: 1m10s • 09/13/2026, 08:02 PM EST

RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O

@github-actions

Copy link
Copy Markdown

VAD Benchmark Results

Performance Comparison

Dataset Accuracy Precision Recall F1-Score RTFx Files
MUSAN 94.0% 89.3% 100.0% 94.3% 738.5x faster 50
VOiCES 94.0% 89.3% 100.0% 94.3% 753.1x faster 50

Dataset Details

  • MUSAN: Music, Speech, and Noise dataset - standard VAD evaluation
  • VOiCES: Voices Obscured in Complex Environmental Settings - tests robustness in real-world conditions

✅: Average F1-Score above 70%

@github-actions

Copy link
Copy Markdown

ASR Benchmark Results ✅

Status: All benchmarks passed

Parakeet v3 (multilingual)

Dataset WER Avg WER Med RTFx Status
test-clean 0.57% 0.00% 6.06x
test-other 1.19% 0.00% 3.42x

Parakeet v2 (English-optimized)

Dataset WER Avg WER Med RTFx Status
test-clean 0.80% 0.00% 6.06x
test-other 1.00% 0.00% 3.54x

Streaming (v3)

Metric Value Description
WER 0.00% Word Error Rate in streaming mode
RTFx 0.64x Streaming real-time factor
Avg Chunk Time 1.413s Average time to process each chunk
Max Chunk Time 1.502s Maximum chunk processing time
First Token 1.679s Latency to first transcription token
Total Chunks 31 Number of chunks processed

Streaming (v2)

Metric Value Description
WER 0.00% Word Error Rate in streaming mode
RTFx 0.70x Streaming real-time factor
Avg Chunk Time 1.299s Average time to process each chunk
Max Chunk Time 1.424s Maximum chunk processing time
First Token 1.305s Latency to first transcription token
Total Chunks 31 Number of chunks processed

Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming

25 files per dataset • Test runtime: 6m31s • 09/13/2026, 08:10 PM EST

RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)

Expected RTFx Performance on Physical M1 Hardware:

• M1 Mac: ~28x (clean), ~25x (other)
• CI shows ~0.5-3x due to virtualization limitations

Testing methodology follows HuggingFace Open ASR Leaderboard

@github-actions

Copy link
Copy Markdown

Supertonic3 Smoke Test ✅

Check Result
Build
Model download (incl. VectorEstimatorVariants/ int4 buckets)
Model load
Synthesis pipeline (--ve-variant int4)
Output WAV ✅ (364.7 KB)

Runtime: 0m25s

Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf.

@github-actions

Copy link
Copy Markdown

Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

Metric Value Target Status
DER 30.3% <35%
Miss Rate 28.2% - -
False Alarm 0.9% - -
Speaker Error 1.2% - -
RTFx 19.5x >1.0x
Speakers 4/4 - -

Sortformer High-Latency • ES2004a • Runtime: 2m 36s • 2026-09-14T00:18:22.082Z

@github-actions

Copy link
Copy Markdown

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Optimal clustering with Hungarian algorithm for maximum accuracy

Metric Value Target Status Description
DER 10.4% <20% Diarization Error Rate (lower is better)
RTFx 11.13x >1.0x Real-Time Factor (higher is faster)

Offline VBx Pipeline Timing Breakdown

Time spent in each stage of batch diarization

Stage Time (s) % Description
Model Download 20.002 21.2 Fetching diarization models
Model Compile 8.572 9.1 CoreML compilation
Audio Load 0.062 0.1 Loading audio file
Segmentation 24.665 26.2 VAD + speech detection
Embedding 94.009 99.7 Speaker embedding extraction
Clustering (VBx) 0.128 0.1 Hungarian algorithm + VBx clustering
Total 94.297 100 Full VBx pipeline

Speaker Diarization Research Comparison

Offline VBx achieves competitive accuracy with batch processing

Method DER Mode Description
FluidAudio (Offline) 10.4% VBx Batch On-device CoreML with optimal clustering
FluidAudio (Streaming) 17.7% Chunk-based First-occurrence speaker mapping
Research baseline 18-30% Various Standard dataset performance

Pipeline Details:

  • Mode: Offline VBx with Hungarian algorithm for optimal speaker-to-cluster assignment
  • Segmentation: VAD-based voice activity detection
  • Embeddings: WeSpeaker-compatible speaker embeddings
  • Clustering: PowerSet with VBx refinement
  • Accuracy: Higher than streaming due to optimal post-hoc mapping

🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 118.8s processing • Test runtime: 2m 10s • 09/13/2026, 08:20 PM EST

@github-actions

Copy link
Copy Markdown

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Evaluating "who spoke when" detection accuracy

Metric Value Target Status Description
DER 15.1% <30% Diarization Error Rate (lower is better)
JER 24.9% <25% Jaccard Error Rate
RTFx 26.11x >1.0x Real-Time Factor (higher is faster)

Diarization Pipeline Timing Breakdown

Time spent in each stage of speaker diarization

Stage Time (s) % Description
Model Download 11.216 27.9 Fetching diarization models
Model Compile 4.807 12.0 CoreML compilation
Audio Load 0.060 0.1 Loading audio file
Segmentation 12.053 30.0 Detecting speech regions
Embedding 20.088 50.0 Extracting speaker voices
Clustering 8.035 20.0 Grouping same speakers
Total 40.192 100 Full pipeline

Speaker Diarization Research Comparison

Research baselines typically achieve 18-30% DER on standard datasets

Method DER Notes
FluidAudio 15.1% On-device CoreML
Research baseline 18-30% Standard dataset performance

Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:

  • M2 MacBook Air (2022): Runs at 150 RTFx real-time
  • Performance scales with Apple Neural Engine capabilities

🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 40.2s diarization time • Test runtime: 2m 57s • 09/13/2026, 08:27 PM EST

@Alex-Wengg
Alex-Wengg merged commit 71242fa into main Sep 14, 2026
14 checks passed
@Alex-Wengg
Alex-Wengg deleted the fix/modelhub-root-required-file branch September 14, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant