Skip to content

feat(vectorisers): Add FastEmbedVectoriser implementation of VectoriserBase and add fast embed dependency group - #224

Open
Tom-Owen-ONS wants to merge 4 commits into
mainfrom
222-make-huggingface_light-vectoriser-dependency-group
Open

feat(vectorisers): Add FastEmbedVectoriser implementation of VectoriserBase and add fast embed dependency group#224
Tom-Owen-ONS wants to merge 4 commits into
mainfrom
222-make-huggingface_light-vectoriser-dependency-group

Conversation

@Tom-Owen-ONS

Copy link
Copy Markdown
Contributor

✨ Summary

Add a new FastEmbedVectoriser to ClassifAI as a lightweight local embedding backend that avoids torch and transformers at runtime, alongside the new fastembed optional dependency group, tests, and documentation updates.

I have run a manual check in general_workflow_demo.ipynb with FastEmbedVectoriser and it runs end to end.

I haven't done a performance benchmark, however happy to do so with guidance on how this is done for classifai.

I have not included the model caching or normalisation logic in survey-assist-embed-core to remain consistent with other implementations of VectoriserBase.

📜 Changes Introduced

  • Added FastEmbedVectoriser as a new VectoriserBase implementation using fastembed.TextEmbedding with ClassifAI-style error handling.
  • Added the fastembed optional dependency group and included it in the all extra.
  • Updated vectoriser exports and overview documentation so FastEmbed appears in the public vectoriser surface and generated API docs.
  • Updated installation and usage documentation to explain the lighter classifai[fastembed] install path.
  • Added unit tests covering single-string input, batch input, load failures, transform failures, and empty input handling.
  • Feature implementation (feat:) / bug fix (fix:) / refactoring (chore:) / documentation (docs:) / testing (test:)
  • Updates to tests and/or documentation
  • Terraform changes (if applicable)

✅ Checklist

  • Code passes linting with Ruff
  • Security checks pass using Bandit
  • API and Unit tests are written and pass using pytest
  • Terraform files (if applicable) follow best practices and have been validated (terraform fmt & terraform validate)
  • DocStrings follow Google-style and are added as per Pylint recommendations
  • Documentation has been updated if needed

🔍 How to Test

  1. Sync the repo environment, including optional extras:
    uv sync --all-extras
  2. Run the test suite:
    uv run pytest -q
  3. Smoke test the new backend directly:
    uv run --extra fastembed python -c "from classifai.vectorisers import FastEmbedVectoriser; vectoriser = FastEmbedVectoriser(model_name='sentence-transformers/all-MiniLM-L6-v2'); print(vectoriser.transform('hello world').shape)"
  4. Open DEMO/general_workflow_demo.ipynb and instantiate FastEmbedVectoriser there. This has been manually validated on this branch.
  5. Review documentation updates.

@Tom-Owen-ONS
Tom-Owen-ONS requested a review from jamie-ons August 21, 2026 15:15
@Tom-Owen-ONS
Tom-Owen-ONS requested a review from a team as a code owner August 21, 2026 15:15
@Tom-Owen-ONS Tom-Owen-ONS linked an issue Aug 21, 2026 that may be closed by this pull request
@github-actions github-actions Bot added the enhancement New feature or request label Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make huggingface_light Vectoriser & dependency group

1 participant