[IlyaKhalafi]

This PR adds optional BM25 tokenizers. With this, Chinese text can use a Jieba adapter for documents, queries and parallel embedding.

Refs #610.

13 focused tests passed, along with pre-commit, mypy and the repository Pyright check. The adapter was also checked with Jieba 0.42.1. An existing worker-shutdown test fails on unchanged main too. The full model and GPU test matrix was not run.

Checks

• Followed the contribution guide.
• Checked for existing PRs.
• Added and ran focused tests.
• Installed and ran pre-commit hooks.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 54489438-bf9f-46e8-afb4-996506bdc5ec

📥 Commits

Reviewing files that changed from the base of the PR and between f48af32 and 0e10764.

📒 Files selected for processing (3)

• README.md
• fastembed/sparse/bm25.py
• tests/test_bm25_tokenizer.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📝 Walkthrough

Bm25 now accepts an optional tokenizer and uses it for documents, queries, and token counts. When no tokenizer is supplied, it uses SimpleTokenizer. Parallel embedding forwards the custom tokenizer. Tests cover tokenization behavior, worker embedding, empty inputs, and the default tokenizer. The README adds a Chinese BM25 example using Jieba and describes tokenizer setup for parallel workers.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to 0e107

No actionable issue remains in the supplied review. The change is mergeable after normal checks.

Pre-merge checks · 4 · 1

Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 2 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title clearly and concisely describes the main change: adding optional tokenizers for BM25.
Description check		The description directly explains the optional BM25 tokenizers, Chinese Jieba support, parallel embedding, tests, and validation results.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 2 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}