Skip to content

fix(xpacks/llm): stop TokenCountSplitter dropping characters at punctuation cuts - #279

Open
linhongyu510 wants to merge 2 commits into
pathwaycom:mainfrom
linhongyu510:fix/tokencount-splitter-drops-chars
Open

linhongyu510 wants to merge 2 commits into
pathwaycom:mainfrom
linhongyu510:fix/tokencount-splitter-drops-chars

fix(xpacks/llm): locate token split boundaries by byte length

df560d3
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

5 warnings and 1 notice
🦀 cargo fmt
succeeded Oct 5, 2026 in 29s