Skip to content

fix(text): interpolate token value in _normalize_token error message - #769

Open
erensh27 wants to merge 1 commit into
google-deepmind:mainfrom
erensh27:fix/f-string-in-normalize-token-error
Open

fix(text): interpolate token value in _normalize_token error message#769
erensh27 wants to merge 1 commit into
google-deepmind:mainfrom
erensh27:fix/f-string-in-normalize-token-error

Conversation

@erensh27

@erensh27 erensh27 commented Aug 9, 2026

Copy link
Copy Markdown

Fix for #658: the error message in _normalize_token() used a plain string literal with {token!r}, so the placeholder was printed literally instead of the offending token value.

# before
raise ValueError(
    'Invalid token: {token!r}. `stop_token`s and `forbidden_token`s must'
    ' map to single token ids in the vocab.'
)
# after
raise ValueError(
    f'Invalid token: {token!r}. `stop_token`s and `forbidden_token`s must'
    ' map to single token ids in the vocab.'
)

Now the exception reports the actual token (e.g. 'hello world') instead of the raw placeholder, making stop_tokens / forbidden_tokens misconfigurations debuggable.

Fixes #658

The message used a non-f-string literal with {token!r}, so the placeholder
was printed literally instead of the offending token. This made debugging
stop/forbidden token misconfigurations impossible.
@google-cla

google-cla Bot commented Aug 9, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@erensh27

Copy link
Copy Markdown
Author

CLA check done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Missing f-prefix in error message in _normalize_token() makes stop_tokens debugging impossible

1 participant