fix: validate BM25 average document length

[qq1069730908]

BM25 currently accepts avg_len=0, negative values and NaN. Embedding a nonempty document can then raise ZeroDivisionError or produce negative/NaN sparse weights.

Validate that avg_len is finite and positive during initialization. Add offline regressions through both Bm25 and SparseTextEmbedding, including signed zero, validation before downloads, valid fractional/default lengths, empty documents, and real parallel processing with a non-default average length.

Validation:

• python -m pytest tests/test_bm25_parameters.py tests/test_common.py tests/test_parallel_processor.py -q — 27 passed.
• pre-commit run --files fastembed/sparse/bm25.py tests/test_bm25_parameters.py — passed.
• git diff --check — passed.

All Submissions:

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: cdfdc92f-8a92-4ab1-847f-7dfe4029257b

📥 Commits

Reviewing files that changed from the base of the PR and between a460268 and 7e9daba.

📒 Files selected for processing: 2

• fastembed/sparse/bm25.py
• tests/test_bm25_parameters.py

🚧 Files skipped from review as they are similar to previous changes: 2

• fastembed/sparse/bm25.py
• tests/test_bm25_parameters.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📝 Walkthrough

Bm25 now raises ValueError when avg_len is non-finite or less than or equal to zero. The constructor documentation describes this constraint. Tests cover invalid values for both model classes, positive values, empty input, and matching serial and parallel embedding results.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 7e9da

No actionable issue has been established that would prevent merging after normal checks.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly and concisely describes the main change: validating BM25 average document length.
Description check		The description directly explains the BM25 validation change, affected invalid values, added regression coverage, and reported validation results.
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}