[ANDROIDHASSAN]

Problem

sentence_postprocess collects a BPE word in word_item until a token without @@ closes it. When the token list ends on a @@ piece, nothing closes it and the piece is dropped:

>>> sentence_postprocess(["hello", "wor@@"])
('hello', ['hello'])
>>> sentence_postprocess(["你", "好", "wor@@"])
('你好', ['你', '好'])

Streaming Paraformer hits this every time a word spans two chunks: paraformer_streaming/model.py and funasr_onnx/paraformer_online_bin.py postprocess each chunk on its own, so the first part of that word disappears from the transcript.

Fix

After the loop, a non-empty word_item is added as a word, with the [begin, end] already tracked for it when timestamps are given. The same 6 lines go into all three copies of postprocess_utils.py (funasr, funasr_onnx, funasr_torch). Token lists that end on a complete word are unchanged.

Tests

Added to tests/test_sentence_postprocess_timestamps.py, over all three copies: ["hello", "wor@@"], ["你", "好", "wor@@"], and the timestamped ["hello", "wor@@", "l@@"] → ["hello", "worl"], [[0, 100], [100, 300]].

• Without the fix: 9 failed subtests (3 cases × 3 copies), each one on its own.
• With the fix, the text-postprocessing CI set (test_postprocess_hotwords.py, test_rich_transcription_postprocess.py, test_sentence_postprocess_timestamps.py): 26 passed, 6 skipped (the fuzzy hotword tests, pypinyin/rapidfuzz not installed here; same 6 skipped on main).

Scope

A @@ piece followed by a Chinese character in mixed text (["hel@@", "你"]) is still mishandled the way it is today; that is a different case and not changed here.

[LauraGPT]

The trailing-only examples reproduce on main, and all 32 selected text-postprocessing tests pass with this patch. However, the unconditional flush also emits an older buffered piece after a later Chinese token, so the mixed-text case called out as unchanged in the PR is changed by these lines.

At 84c7b76b5622caf803f7838be6e908769b22f417, in all three copies:

sentence_postprocess(["hel@@", "你"])
# ('你hel', ['你', 'hel'])
sentence_postprocess(["hel@@", "你"], [[0, 100], [100, 200]])
# ('你 hel', [[0, 100], [100, 100]], ['你', 'hel'])

Main already drops that partial piece and has a timestamp bug for the Chinese token, but this patch newly appends the piece out of order and attaches a zero-length timestamp to it. ["hel@@", "你", "wor@@"] also produces 你helwor, combining pieces separated by Chinese text.

Please address the stale buffer at mixed-language boundaries before merging: preserve token order and each retained piece's own span, or keep a strictly scoped trailing-only fix from flushing stale pieces across a Chinese boundary. Add timed and untimed regressions for these boundary cases in all three implementations alongside the existing trailing-only cases. The separate streaming cross-chunk word-joining semantics should not be claimed solved by a stateless text flush.

Validation here is real pure-Python text/timestamp execution (baseline 2 failing tests, candidate 32 passing selected tests plus 96 boundary observations), not GPU inference or ASR quality. No production files were changed during review.