sift

Feature engineering & text classification as a service — Python as the orchestration bridge. sift turns a record into a dictionary of features by composing small featurizers: deterministic lexical stats, domain keyword dictionaries, and AI featurizers that bridge to a Hugging Face model (downloaded at runtime — weights never vendored). It composes them in a pipeline, emits a pandas-ready feature matrix, and can orchestrate other microservices (sync and async) to enrich features. Everything is injectable, so the whole thing tests offline with no model download.

---

Why this shape

Python isn't the fastest runtime, but it's the best bridge — for data structures, feature engineering, and gluing AI models and services together. sift leans into exactly that:

• Feature engineering, composable. Each Featurizer maps a record → a flat dict[str, float]. Add or drop one without touching callers.
  • LexicalFeaturizer — deterministic, framework-free text stats.
  • KeywordFeaturizer — encode domain knowledge as a dictionary of feature groups (finance terms, risk terms, …) → presence/count features.
  • SentimentFeaturizer — an AI bridge: wraps a Hugging Face classifier into numeric features.
• A pipeline that yields a matrix. FeaturePipeline merges featurizers and to_columns() returns a columnar view you can drop straight into pandas.DataFrame(...) or a model.
• Microservice orchestration. RemoteFeaturizer enriches features from another service; gather_remote_features fans a record out to many services concurrently with asyncio + httpx.
• No weight redistribution. The HF model is pulled from the Hub at runtime and cached under HF_HOME; the repo and image ship code, not licensed weights.
• Injectable everywhere. Featurizers, the classifier, and the httpx client are all injected — the test suite runs on fakes and an httpx mock transport (no torch, no network).

flowchart LR
  R[record: text + fields] --> P[FeaturePipeline]
  subgraph P[FeaturePipeline]
    L[lexical] & K[keyword dict] & S[HF sentiment] & X[remote svc]
  end
  P --> F[feature dict / matrix]

Loading

API

Endpoint	Description
POST /features {"text": "..."}	Merged feature dict from all featurizers.
POST /features/batch {"records":[...]}	Per-record features + a columnar (pandas-ready) matrix.
GET /featurizers	Active featurizers.
POST /classify {"text":"..."}	Raw labels/scores from the AI featurizer.
GET /healthz	Liveness + featurizers.

docker compose up --build
curl -s localhost:8000/features -H 'content-type: application/json' \
     -d '{"text":"Record revenue and strong guidance; no lawsuit risk."}'
# {"char_count":51,...,"kw_finance_present":1.0,"kw_risk_present":0.0,"sentiment_positive":0.98,...}

Develop & test

make venv    # test deps only (fastapi, httpx, pytest) — no torch
make test    # 11 passing tests: featurizers, pipeline, matrix, remote + async fan-out

Layout

Module	Responsibility
sift/features.py	Featurizer protocol + lexical / keyword / sentiment / remote.
sift/pipeline.py	Compose + merge; to_columns → pandas-ready matrix.
sift/orchestrate.py	Async fan-out to feature microservices.
sift/model.py	Lazy Hugging Face pipeline (runtime download, no vendoring).
sift/app.py	FastAPI wiring; create_app(pipeline=...) for injection.

---

_{The high-level AI / data-science layer of a small polyglot portfolio — Python here; Go for networking, TypeScript for test/QA, Rust for the from-scratch numerical core (loads the same .safetensors weights and runs the forward pass by hand). — Nicholas Martins · github.com/nickmartins-lambda}