whatdo

A Typesafe-compatible API backed by the Laya non-autoregressive System 1 decision engine.

whatdo implements the Jev API wire contract (POST /v1/systemone) over the local Laya engine, so the official typesafe-sdk-python works against a self-hosted deployment unchanged — point it at your server with TYPESAFE_BASE_URL and your existing client.system_one(...) code runs locally.

What it does

A caller submits a State (the content to evaluate) and a set of Questions, and receives typed Answers in a single forward pass. Three decision primitives are supported:

• Noul — a calibrated boolean (probability 0–1)
• Choice — select one of a defined option set (returns the choice, its distribution, and a confidence)
• Score — rate on an ordered rubric of ≥2 levels (returns a probability-weighted value and a confidence)

Key characteristics

• Drop-in Jev API — POST /v1/systemone plus GET /v1/models, /healthz, /readyz.
• Auth or no-auth — bearer-token auth against configured API keys, or open for trusted networks.
• Configurable inference — one served model per deployment, a thread-based worker pool of model copies fed by a bounded queue, and a retryable 529 on overload.
• Optional OpenTelemetry — logs, traces, and metrics via OTLP, shipped as the whatdo[otel] extra (no OTEL libs required for the base install).
• Production packaging — an ACCEL-parametrized multi-stage Dockerfile (CPU + CUDA now; ROCm + Jetson planned), managed with uv and a committed uv.lock; dependencies are declared inline in pyproject.toml (PEP 621 extras + PEP 735 groups).
• Configuration via Pydantic settings (WHATDO_-prefixed environment variables).

Installation

whatdo requires Python >= 3.12. Every install includes the Laya engine and PyTorch; the choice below is only which PyTorch build you get.

From PyPI

Create a virtual environment first (recommended):

python3 -m venv .venv
source .venv/bin/activate

Nvidia CUDA

pip install whatdo

CPU:

pip install whatdo --extra-index-url https://download.pytorch.org/whl/cpu

macOS (uses the Apple GPU when available, otherwise the CPU):

pip install whatdo

From the GitHub source

Clone the repository:

git clone https://github.com/androiddrew/whatdo.git
cd whatdo

With uv (creates .venv and installs the exact locked versions):

uv sync --extra cuda    # CUDA (Linux with an NVIDIA GPU)
uv sync --extra cpu     # CPU (Linux without a GPU, or macOS)
source .venv/bin/activate

With pip (installs the exact locked versions from the requirements-*.txt files, then whatdo itself from the checkout):

CUDA:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements-cuda.txt
pip install --no-deps .

CPU (Linux without a GPU, or macOS):

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
pip install --no-deps .

The requirements files pin every dependency but not whatdo itself; pip install --no-deps . installs whatdo from the checkout without changing those pinned versions.

Starting the Server

whatdo serve

The server listens on http://0.0.0.0:8000 and runs the Laya engine. The first start downloads the model checkpoint from Hugging Face, which takes a minute or two; GET /readyz returns 200 once it's loaded.

Common options:

whatdo serve --port 9000                  # listen on another port
whatdo serve --workers 2 --device cpu     # two model copies serving in parallel, on the CPU
whatdo serve --log-level debug --text-logs  # readable debug logs in a terminal
whatdo serve --help                       # every option, with its WHATDO_* environment variable

Every option can also be set with its WHATDO_* environment variable; flags take precedence. On Apple silicon the GPU (MPS) supports only one worker, so use --device cpu for more.

Example Curl Requests

curl -X POST http://localhost:8000/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jev-latest",
    "state": "Hi, I have been trying to connect my Stripe account for 3 days and the integration keeps failing. This is completely blocking our checkout flow!",
    "questions": {
      "urgency_expressed": {
        "type": "noul",
        "instructions": "Does this message express high urgency or a blocked workflow?"
      }
    }
  }' | jq

curl -X POST http://localhost:8000/v1/systemone \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
  "model": "jev-latest",
  "state": "Help! I was charged twice for my subscription this month and I need a refund immediately before my account overdrafts.",
  "questions": {
    "urgency": {
      "type": "noul",
      "instructions": "Does this message express urgency or request immediate action?"
    },
    "routing_department": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": ["charges", "invoices", "refunds"],
        "technical": ["bugs", "outages", "errors"],
        "general": null
      }
    },
    "severity": {
      "type": "score",
      "instructions": "How severe is the issue?",
      "criteria": [
        "Cosmetic or informational",
        "Workaround exists, non-critical",
        "Blocking issue, requires immediate intervention"
      ]
    }
  }
}' | jq

Design

The domain glossary lives in CONTEXT.md, and the architecture decisions in docs/adr/.

Development

Developer workflow is driven by a Makefile (make setup, lock, lint, fmt, typecheck, test, test-otel, test-slow, build-cpu, build-cuda, docs, load-test, run) using uv for environments and dependency locking. make setup runs uv sync --extra cpu (base deps incl. Laya + the CPU torch build, the dev group, and the editable package, from uv.lock; use make setup ACCEL=cuda on a GPU box); make lock refreshes uv.lock and re-exports the pinned requirements*.txt. Tests run against a deterministic FakeEngine in CI (no GPU); the full end-to-end suite drives the real SDK against real Laya checkpoints on GPU-equipped machines. Load tests use k6.

Container images

One multi-stage Dockerfile is parametrized by an ACCEL build arg that selects the base image and torch wheel index (ADR-0001). cpu and cuda are implemented; rocm/jetson are documented, unbuilt slots. The builder installs the pinned deps into a uv-managed venv; the final stage copies only that venv (no dev dependencies) and runs as a non-root user.

make build-cpu     # trim CPU image (torch+cpu, no CUDA wheels), serves the real Laya engine
make build-cuda    # CUDA image (builds on CPU-only hosts; running inference needs a GPU)

docker run -p 8000:8000 whatdo:cpu-dev   # then POST /v1/systemone

To build against a fork of laya (e.g. to test a patch) instead of the pinned release, pass a git+…@ref spec — it installs over the pinned dependency stack:

make build-cpu LAYA_FORK="git+https://github.com/you/laya@my-branch"
# or directly:
docker build --build-arg ACCEL=cpu \
  --build-arg LAYA_FORK="git+https://github.com/you/laya@my-branch" -t whatdo:cpu .

License

Apache-2.0 — see LICENSE. Author: Drew Bednar.