Skip to content

fix: Added New Word Count Method#9185

Open
CooperWang0912 wants to merge 2 commits into
AstrBotDevs:masterfrom
CooperWang0912:fix/Multilingual-Word-Count
Open

fix: Added New Word Count Method#9185
CooperWang0912 wants to merge 2 commits into
AstrBotDevs:masterfrom
CooperWang0912:fix/Multilingual-Word-Count

Conversation

@CooperWang0912

@CooperWang0912 CooperWang0912 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Issue / 问题

  • The current segmented reply word count threshold does not take into account different languages, bloating the word count for languages such as English
  • The original word_cnt method uses ASCII numbers for separation, which breaks for any language that doesn't use the English alphabet and emojis
  • 目前的分段回复默认使用字符识别决定字数阈值,使得像英语之类的其他语言字数多于实际输出
  • 已有的word_cnt函数基于ASCII码进行分类,导致除了使用英语字母表之外的语言以及emoji都会被按照字符进行字数统计

Fixes #9184

Modifications / 改动点

  • Added new word_cnt method that supports Chinese, Japanese, and all the languages that use spaces to separate words
  • 新增了word_cnt函数,现支持包括中文,日文,以及其他使用空格作为分割的语言字数统计
  • This is NOT a breaking change. / 这不是一个破坏性变更。

Screenshots or Test Results / 运行截图或测试结果

image

Checklist / 检查清单

  • 😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
    / 如果 PR 中有新加入的功能,已经通过 Issue / 邮件等方式和作者讨论过。

  • 👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
    / 我的更改经过了良好的测试,并已在上方提供了“验证步骤”和“运行截图”

  • 🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in requirements.txt and pyproject.toml.
    / 我确保没有引入新依赖库,或者引入了新依赖库的同时将其添加到 requirements.txtpyproject.toml 文件相应位置。

  • 😮 My changes do not introduce malicious code.
    / 我的更改没有引入恶意代码。

@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. area:core The bug / feature is about astrbot's core, backend labels Jul 8, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The _word_cnt helper is declared async but does only synchronous CPU work; making it a regular function would simplify usage and avoid unnecessary await overhead in tight loops.
  • In the TTS and other loops you compute word_count = await _word_cnt(comp.text) before checking isinstance(comp, Plain); consider moving the word count call inside the Plain branch to skip work for non-Plain components.
  • Since _word_cnt is called repeatedly, consider precompiling the no_space_pattern regex at module level to reduce regex compilation overhead on hot paths.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The `_word_cnt` helper is declared `async` but does only synchronous CPU work; making it a regular function would simplify usage and avoid unnecessary `await` overhead in tight loops.
- In the TTS and other loops you compute `word_count = await _word_cnt(comp.text)` before checking `isinstance(comp, Plain)`; consider moving the word count call inside the `Plain` branch to skip work for non-`Plain` components.
- Since `_word_cnt` is called repeatedly, consider precompiling the `no_space_pattern` regex at module level to reduce regex compilation overhead on hot paths.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a helper function _word_cnt to count words differently depending on the language (character-based for Chinese/Japanese, space-separated for other languages) and integrates it into various threshold checks in the pipeline. The review feedback highlights a critical bug where accessing comp.text before verifying its type as Plain could lead to an AttributeError crash. Additionally, the reviewer suggests changing _word_cnt from an asynchronous to a synchronous function since it is CPU-bound and contains no asynchronous I/O, which subsequently allows removing the unnecessary await keywords across all call sites.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread astrbot/core/pipeline/result_decorate/stage.py Outdated
Comment thread astrbot/core/pipeline/result_decorate/stage.py Outdated
Comment thread astrbot/core/pipeline/result_decorate/stage.py Outdated
Comment thread astrbot/core/pipeline/result_decorate/stage.py Outdated
Comment thread astrbot/core/pipeline/result_decorate/stage.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core The bug / feature is about astrbot's core, backend size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Segmented Reply Threshold Error

1 participant