Nonsense Responses

[maroc-dev]

Hello,

I have been getting nonsense responses from the model after the initial prompt.

I have the client portion on Ubuntu 24
I have the server portion on Ubuntu 22

Thanks

[cleverpal-ai]

Same here, some mooing and hardly distinguishable wording

[twilligon]

TBH as a base model the results are very prompt-dependent. It's easy to push the model out of distribution and it takes some experimentation to avoid this. But completions for the default "Bob" prompt should at least sound English-like; anything else seems like an inference or audio driver bug.

[AbrahamSanders]

From my experiments using the inference client (not the notebook) the model seems to be very sensitive to the token temperature. Lowering it below 0.8 leads it to degenerative repetition, where the model just repeats itself with silence or background noise tokens. Raising it beyond 0.9 leads to incoherent random speech sounds.

My intuition is that other sampling strategies, perhaps top_p or min_p sampling might yield better results. I'll do some experiments when I have time.

[humblenginr]

For me when running the inference client, the model never seems to comprehend the user's input. It recognizes interruptions and stops, but never really understands what the user is saying.

#34 (@AbrahamSanders any suggestions or thoughts?)

[AbrahamSanders]

@humblenginr I've noticed the same. I've been able to get it to respond conversationally for about 5 seconds beyond the initial prompt audio, and then it degrades to repetitive generation of silence or background noise. I think one of the following strategies is likely to help:
(a) higher temp (0.9-1.0) with min_p sampling to prevent sampling "nonsense" tokens at the tail of the distribution, or
(b) lower temp (0.6-0.7) with some flavor of repetition penalty to prevent degeneration.

I've been swamped but hope to try something soon

[humblenginr]

Thanks @AbrahamSanders . Do let me know if you find something!

[AbrahamSanders]

@humblenginr Ok, so I've got some interesting results using the notebook so far. I implemented the top_p, min_p, and repetition_penalty generation parameters using LogitsProcessors from the HuggingFace Transformers library. It's been able to get around the degeneration issue for the most part, although it's still quite sensitive to hyperparameters. Next I'll see how this works with the inference client.

Bob prompt (split channels)

bob_prompt.mp4

Bob completions

bob1.mp4

bob2.mp4

bob3.mp4

Phone call prompt (split channels, from the CallFriend corpus)

phone_prompt.mp4

Phone call completions

phone1.mp4

phone2.mp4

[humblenginr]

Thanks for the update, @AbrahamSanders! The results sound better. Any chance you could share the notebook you used? Would love to take a closer look. Also curious how it performs with the inference client—keep me posted!

[AbrahamSanders]

Thanks for the update, @AbrahamSanders! The results sound better. Any chance you could share the notebook you used? Would love to take a closer look. Also curious how it performs with the inference client—keep me posted!

Sure - I'll commit it to my fork and issue a PR soon. Just need to finish testing a few more things.

[AbrahamSanders]

@humblenginr I've committed these changes to a branch on my fork - have a look: https://github.com/AbrahamSanders/hertz-dev/blob/support-sampling-params/inference.ipynb

You can see the supported sample params and their defaults here: https://github.com/AbrahamSanders/hertz-dev/blob/6344f280684680d32a0e8a48bcbf7d2fa48f88e0/model.py#L226

The SampleParams defaults are set to preserve the original repo behavior, but the notebook shows how to customize it to use the top_k, top_p, and repetition_penalty params to get better results. min_p and typical_p are also available for experimentation.

Despite the improvement over the default behavior, degeneration still remains a significant issue especially beyond the 30 second mark. The live inference client amplifies this issue, with degeneration typically happening in the 15-30 second range (previously it was immediate). I've on rare occasions been able to hold a conversation of 45 seconds before the model either goes silent or starts generating waves of feedback / noise.

Some general observations:

• You can set the token_temp as high as 2.0 without loss of coherence as long as you counteract it with a top_k around 25 or equivalently appropriate top_p or min_p values. Sometimes this can increase the diversity of the generated speech.

• Setting repetition_penalty is a must in all cases. When disabled (1.0) degeneration happens immediately. Range of 1.4 - 1.6 seems to be a sweet spot. For fun, you can set it to a value below 1.0 and the speakers will immediately start stuttering comically 🤣

• Decreasing token_temp, top_k, top_p, or increasing min_p all have the effect of making speech more coherent at the cost of quicker degeneration into silence or noise. I've found good ranges to be:

  • token_temp: 0.7 - 0.9. Higher values are possible if combined with top_k, top_p, or min_p.
  • top_k: 50 - 80.
  • top_p: 0.95 - 0.99
  • min_p: 0.001 - 0.02 - this one is very sensitive to small changes!
  • typical_p: Doesn't appear to work well at any value

• At more constrained temperatures / sampling settings, you will observe the speakers start to stutter and speak with a "low confidence" voice - a lot of filled pauses "umm" and "uhhh" and slower overall speech. At less constrained values, you will observe more fluid speech and turn-taking and less degeneration into silence or noise, but this comes at a cost of coherence - words will be poorly formed and not fit in contextually.

In conclusion, I think a more sophisticated sampling approach is needed that can avoid degenerative repetition without unnecessarily truncating the next token distribution (top_k, top_p, min_p) or decreasing its natural entropy (token_temp). Perhaps something based on contrastive search https://arxiv.org/pdf/2202.06417. Also it would help to understand what types of shifts occur in the logit distribution when the model starts to degenerate - perhaps this is a detectable pattern that could be avoided with a custom logits processor.

@devanshrpandey @nicholascc @calculating @eltociear I would love to hear your thoughts on this!