Repository navigation

fix(tokenizer): prevent empty-symbol hangs with hosted regression coverage - #3855

Merged

LauraGPT merged 2 commits into

mainfrom

codex/pr3852-hosted-regressions-20261107

Nov 7, 2026

Merged

fix(tokenizer): prevent empty-symbol hangs with hosted regression coverage#3855
LauraGPT merged 2 commits into
mainfrom
codex/pr3852-hosted-regressions-20261107

Commits on Nov 7, 2026
- fix(tokenizer): ignore empty non-linguistic symbols to avoid a hang
  MohammadHijjawi97
  committed
  5156f09
- test(tokenizer): run empty-symbol regressions in NumPy CI
  LauraGPT
  authored
  a6516ae