Repository navigation
sherpa: P2P lexicon generation for espeak-orphaned models (#66) - #67
Merged
Merged
Conversation
…ntion findings Phase 2 investigation results (SHERPA2_TODO.md): SOLVED: gruut→vocab symbol mapping. All 40 gruut English IPA symbols map identity or decompose (diphthongs → component vowels, affricates → component consonants) into piper's 130-token vocabulary. 124k-entry lexicon generated with 100% coverage, zero skips. UNSOLVED: phonological conventions. Each model family was trained on a different phonemizer's output: - kokoro-en: misaki convention (uppercase O/A/I as internal symbols; the multi-lang release's bundled lexicon-us-en.txt WORKS — proven) - piper/matcha-en: espeak convention (gruut symbol-mapped lexicon produces audio but garbled — stress placement and unstressed vowel reduction differ between gruut and espeak) src/sherpa_lexicon.rs (feature-gated, 6 unit tests): - PhonemeMap::for_english(): builds gruut→target symbol mapping from the model's tokens.txt, verifying every identity/decomposition symbol is in-vocab - generate_lexicon_txt(): gruut lexicon format → sherpa lexicon.txt - Next: per-family convention mapping (kokoro: use bundled lexicon; piper: gruut→espeak phonological conversion)
src/sherpa_convention.rs (7 tests): rule-based gruut→model phonological convention conversion — symbol substitution (5 diphthong shorthands + 2 affricate ligatures for misaki; component decomposition for espeak), function-word destressing (~50 words), rhotic splitting (ɚ → ɜ ɹ / ə ɹ), unstressed vowel reduction (ɛ/ʌ → ə in unstressed syllables). 124k entries generated with 100% symbol coverage, zero vocab violations. HARD FINDING (documented in SHERPA2_TODO.md): the rules handle the symbol layer and systematic phonology correctly — ~90% of common words produce output identical to the model's bundled lexicon. But ~10% word-specific differences (stress level: 'go' ˈ vs ˌ, weak form vowel: 'the' ə vs i, strong form: 'a' ə vs A, allophones: flap ɾ and barred-i ᵻ) garble the Whisper transcription despite correct symbols. REVISED APPROACH: - kokoro-en/kitten: use the proven kokoro multi-lang bundled lexicon (Apache 2.0, ships as en-misaki voicegarden bundle — no conversion) - piper/matcha-en: CMUdict is a better source than gruut (BSD, ARPABET maps directly to espeak IPA, same phonemizer family as piper training) - convention conversion code kept for non-English languages without alternative sources
sherpa-onnx 2.x removed the espeak-ng phonemizer; piper, matcha-en, kokoro-en and kitten models synthesize silence without a lexicon. Rules-based convention conversion alone garbles speech: ~10% of common words differ between phonemizer conventions in word-specific ways (stress level, weak forms, allophones) that no rule set captures. Layered lexicon generation, validated against kokoro-en's own bundled lexicon (ground truth): - sherpa_p2p.rs: P2pTable — aligned-pairs TSV (gruut phoneme sequence → target convention sequence) loaded into an exact-lookup HashMap. Pairs are generated offline by aligning the gruut lexicon against the target family's own reference lexicon (prepare_p2p_data.py). - sherpa_convention.rs: convert_lexicon_file now layers table (exact, word-specific) → rules (symbol substitution + phonological rules) → vocabulary filter; reports per-layer counts. - sherpaonnx_engine.rs: ensure_generated_lexicon at model load — orphaned models with no lexicon get one generated when a gruut lexicon + aligned pairs are locally available. Convention auto-detected from tokens.txt (misaki uppercase diphthongs A I O W Y). Pairs are required; rules-only generation is refused (it garbles). - examples/lexicon-convert: offline generation CLI with pairs support. - examples/kokoro-ab: A/B wav capture for listening comparison. Validation (kokoro-en-v0_19): 133,637 entries from the gruut lexicon, 0 vocabulary skips (53,213 via table / 80,424 via rules). All six previously-garbling words now match ground truth exactly (go, the, a, converted, hello, world). 88.8% of overlap words byte-identical; the residual differs only in sub-phonemic allophones (ə/ᵊ, t/ɾ) that do not garble speech. Also fixes a test-isolation bug (two sherpa_lexicon tests shared a fixed temp path and raced).
Blockers from review (6/10): - B1: lexicon.txt is now written to lexicon.txt.tmp and renamed — a crash mid-generation can't leave a truncated file that poisons every later load. Generation that produces 0 entries deletes the temp and errors instead of persisting an empty lexicon (the has-lexicon check would then short-circuit forever). Mid-file I/O and UTF-8 errors abort conversion instead of silently truncating (read_line loops replace lines().map_while(Result::ok) in convert_lexicon_file, read_tokens_txt, P2pTable::parse). - B2: generation is gated to orphaned ENGLISH families via should_auto_generate_lexicon (kokoro/kitten/matcha model types + English id markers; vits only for vits-piper-en). Non-English and non-orphaned models are never touched. Serious findings: - empty/unparseable tokens.txt vocabulary is now an error, not a silent espeak-convention default - tokens.txt tab separator handled (comment said TAB-or-SPACE; code only split on space) - convention detection deduplicated into detect_convention (was implemented twice, could drift) - SHERPA_*_LEXICON env vars pointing at missing paths now warn instead of silently falling through - P2pTable malformed lines counted + reported (not silently dropped) Nits: dead WEAK_FORMS const removed (documented instead), stale vowel-reduction doc fixed, unreachable pop/push branch in split_rhotics removed, DESTRESS_WORDS deduplicated, lexicon filename predicate shared, P2pTable parse unified over BufRead, LexiconGenStats struct replaces the (usize, usize, usize) tuple with a rules() helper, examples validate args / write errors to stderr / exit non-zero, fixed-path temp test moved to tempdir. New tests: zero-entry refusal + cleanup, empty-vocab error, tab separators, unstressed-vowel-reduction firing, the family gate (positive + negative), malformed pairs counting. Validation re-run: real-data output byte-identical to the pre-refactor lexicon (133,637 entries, 53,213 table / 80,424 rules); engine auto-generation exercised end-to-end on kokoro-en-v0_19 (no lexicon → discovery → atomic write → audio).
Loop-2 (6/10, do-not-merge) found the loop-1 gate fix was itself wrong:
- piper gate used a fabricated prefix: registry ids are
piper-en_US-amy-low (no vits-), so starts_with("vits-piper-en")
excluded ALL 132+ English piper models. The vits arm now uses the
existing is_piper_or_github_model prefixes (piper-, coqui-, icefall-,
melo-, vctk-, ljs-) — verified against sherpa-onnx-models 1.1.0.
- bilingual <lang>_en compounds (kokoro-zh_en-*, matcha-fa_en-*,
icefall-zh_en-zh-en, melo-zh_en-*) passed the gate via their 'en'
marker — an English-only lexicon in a bilingual model is the same
poisoning class as loop-1 B1. New is_bilingual_en_compound rejects
two-letter-code+_en compounds; 'northern_english' style ids are
correctly NOT matched (boundary check).
- English id markers tightened: bare 'en_' substring matched
'thorsten_emotional' — now en_us/en_gb/en-us/en-gb/-en-/-en suffix.
- 'pairs required' invariant now enforced: empty pairs table errors
(rules-only generation garbles); P2pTable::parse propagates I/O
errors instead of silently truncating (same class as loop-1 B1).
- new full-registry test runs the gate over all 1780 real models:
no bilingual passes, every gated model is English-market, the
English piper fleet (≥100) is gated in. Synthetic-id tests replaced
with real registry ids.
- nits: engine does the cheap has-lexicon check before asset
discovery; collect_lexicons reuses the shared filename predicate;
example exits non-zero on pairs-load failure.
Validation: 476 tests, clippy -D warnings clean, fmt clean; real-data
output byte-identical (133,637 entries, 53,213 table / 80,424 rules).
- gate: inflect-en-* (tarball-verified orphan: espeak-ng-data, no lexicon) and willwade-patched-en_* (piper re-exports, same layout) now included — 158 orphaned English models total vs 150 before - full-registry test now pins per-family gated counts (piper 132, willwade 6, coqui 3, icefall 3, inflect 2, melo/vctk/ljs 1, kokoro 2, kitten nano 4 / micro 1 / mini 2). A count drop fails loudly — the invariant-only version was blind to false negatives (the exact loop-2/loop-3 bug class) - generate_lexicon_txt: read_line loop with error propagation, matching convert_lexicon_file (a mid-file I/O error no longer silently truncates a 'successful' generation) - lexicon.txt.<pid>.tmp: concurrent engine loads no longer share a temp path (speak() is &self; rename stays same-dir atomic) 476 tests, clippy -D warnings clean, fmt clean; real-data output byte-identical.
The pid suffix didn't cover the scenario its comment named: two threads in one process share a pid, and generation runs inside speak() (&self) before the instance mutex. Per-call atomic counter in the temp name now; the stale fixed-name assertion in the zero-entry test replaced with a .tmp sweep.
sherpa-onnx 2.x has two lexicon formats with very different pipelines: - old format (space-separated) → Lexicon frontend → word2ids lookup feeding RAW token ids to the model: no BOS (^), no EOS ($), no pad interleaving. Kokoro's own impl handles this (ParseLexiconFile + its own wrapping), but VITS/piper models — trained on piper1's ^ phoneme _ ... _ $ streams — garble badly: 'Hello world, this is a test' transcribed as 'Low words as to the verbless one', audio duration roughly half the espeak reference. - new format (word||phonemes) → ParseLexiconFile → PiperPhonemesToIdsVits wrapping (bos, pad after every phoneme, eos) + sentence splitting + longest-phrase matching + punctuation and word-boundary handling 'like espeak does' (#4020, #4075). Lexicon generation now emits the new format for all families. Validation (piper-en_US-amy-low, whisper-small round-trip vs the 1.13.8 espeak reference captured from the pre-#65 build): - 'Hello world, this is a test of the converted lexicon.' → transcribed word-for-word IDENTICAL to the reference - stress sentences (fox/riverbank, 77 seashells/February, technology) → all perfect - token stream debug-verified: [1, 0, ph, 0, ..., 2] - kokoro (misaki pairs) unaffected by the format switch Also: examples/kokoro-ab reads the model's real sample rate from its .onnx.json (amy-low is 16 kHz, not 24) and writes a PCM16 WAV header matching the callback's i16 output (was float-tagged). examples/sherpa-debug: direct sherpa OfflineTts debug harness for lexicon-path investigation.
The runtime guards (is_sherpa2_orphan / skip_if_sherpa2_orphan) and the #66 reason strings are gone: under sherpa-onnx 2.x the English piper and matcha voices now speak via the auto-generated lexicon (gruut lexicon + P2P pairs from the voicegarden eng-US bundle, release v0.3.1). Tests stay #[ignore] like every other live test needing downloaded models; the CI runs them with --ignored. - default VITS test model switched piper-nl_BE-rdh-low → piper-en_US-amy-low: the lexicon path is English-only (the gate refuses non-English models), and the Dutch voice has no rescue path - CI downloads the voicegarden eng-US bundle (lexicon.txt + both P2P pair tables) into ~/.voicegarden/lexicons/eng-US/, cached with the models - asset discovery covers all three cache layouts (eng-US/, en/, expanded/eng-US/) and finds the pairs in any of them — the voicegarden release ships the pairs inside every English bundle Local validation (models + assets on disk, --ignored --test-threads=1): 20/20 live tests pass — the engine auto-generates lexicon.txt at load (133,637 entries; 125,985 table / 7,652 rules; 0 skipped) and piper + matcha synthesize correct audio (whisper round-trip validated in the voicegarden release notes).
sherpa-onnx-sys skips its lib download when target/sherpa-onnx-prebuilt exists; a partially-restored rust-cache then fails at link time on every rerun. A completeness check deletes the cache when the static lib is missing, forcing a clean re-download.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes the misaki half of #66: kokoro-en and kitten models (and the piper/matcha-en machinery, pending espeak pairs) get a working phonetic front-end under sherpa-onnx 2.x, which removed the bundled espeak-ng phonemizer.
sherpa_p2p:P2pTable— an aligned-pairs TSV (gruut phoneme sequence → target-convention sequence) loaded into an exact-lookup HashMap. Pairs are generated offline by aligning the gruut lexicon against the target family's own reference lexicon (prepare_p2p_data.py, in floravox). One table per convention, not per model.sherpa_convention:convert_lexicon_filelayers table (exact, word-specific) → rules (symbol substitution + phonological rules) → vocabulary filter, reporting per-layer counts.sherpaonnx_engine: at model load, orphaned English models with no lexicon get one generated automatically when a gruut lexicon + aligned pairs are locally available (SHERPA_GRUUT_LEXICON/SHERPA_P2P_PAIRSenv vars, model dir, or the voicegarden cache). Convention is detected from the model's owntokens.txt(misaki uppercase diphthongsA I O W Y). Generation is atomic (temp + rename, per-call unique temp names), refuses zero-entry output, and treats an empty pairs table as an error — rules-only generation garbles and is never attempted.willwade-patched-*,inflect-*, kokoro-en ×2, kitten ×7, matcha/icefall-en, coqui/melo/vctk/ljs-en). Bilingualzh_en/fa_encompounds and non-English models are never touched. A full-registry test pins per-family counts so prefix drift fails loudly.Why rules alone weren't enough
~10% of common words differ between phonemizer conventions in word-specific ways no rule set captures — stress level (
go ɡˈOvsɡˌO), weak forms (the ðəvsði), strong forms (a əvsA), allophones (converted ...tɪdvs...ɾᵻd). The pairs table encodes these as data learned from each family's own bundled lexicon.Validation (kokoro-en-v0_19, against the model's own bundled lexicon as ground truth)
go,the,a,converted,hello,worldexamples/kokoro-abcaptures A/B wavs)Review loops
4 loop reviews (6 → 6 → 7 → 8/10, merge). Loop 1: atomic write, zero-entry poisoning, family gate. Loop 2: gate was wrong against real registry ids (piper has no
vits-prefix; bilinguals passed); pairs hardening. Loop 3:inflect/willwade-patchedcoverage, false-negative-sensitive registry test,generate_lexicon_txttruncation. Loop 4: thread-unique temp names.Remaining (tracked in SHERPA2_TODO.md)
p2p-misaki.tsvas a voicegarden-lexicons release asset so auto-generation works out of the boxp2p-espeak.tsvfor piper/matcha-en (needs an espeak-convention reference lexicon; CMUdict is BSD)PhonemeTransducerhybrid (separate PR, already committed onfeat/p2p-transducerthere)