Run the models in this process, through the same calls as a NIM

[Manushpm8]

The client could only reach a model over HTTP: a NIM, or a Databricks serving endpoint. Someone who already has structured-data-models loaded in a notebook had to call its tensor API directly, and then use a different API again once the model was hosted. SDMClient.in_process() runs the model in the caller's own Python process behind the same calls.

import sdm
from sdm_client import KumoRelational, SDMClient

with SDMClient.in_process(sdm.models.KumoRelational()) as client:
    result = client.predict(request, model=KumoRelational())

With no model passed, each one is loaded from its published weights on Hugging Face the first time it is asked for, on CUDA when available, else the CPU.

How it works

It is a third transport beside the HTTP one and the Databricks one. The adapters send exactly the payload they would send a NIM and read back what a NIM answers, so predictions, fit() sessions, rankings and forecasts work unchanged: nothing above the transport knows where the model runs.

The transport reads that payload the way the SDM NIM does:

• Wire tables are decoded by their declared dtypes. That covers base-10 integers past 2**53, timestamps normalised to UTC, and numeric join keys read as int64, with a fractional key refused rather than truncated onto another row.
• Each relational neighbourhood is scoped by its instance_id, the way the model's own sampler scopes each example.
• The context is fitted with the NIM's seed (42), one estimator and, on a GPU, its autocast precision.
• Answers come back with the NIM's ids, class labels and quantile levels. A refusal is the same NimRequestError, with the NIM's status and code.

Verified against the NIM itself

I served the current SDM NIM code locally on the CPU, with the same published weights and structured-data-models 0.2.1, and sent each request both ways. Every prediction, probability and quantile was identical (largest difference 0.0):

	Cases
kumo-tabular	binary (int labels), multiclass (string labels), regression with quantiles, bool target, composite key, int64 key past 2**53, a session
kumo-relational	count and sum regressions with quantiles, binary, multiclass, a bool column, use_prediction_time, custom quantile levels, mean statistic, float foreign keys with nulls, int64 keys past 2**53, string keys
through the client	RANK TOP 3, FORECAST 3 TIMEFRAMES, sessions scored twice
real data	Adult, Wine, Titanic, Bank Marketing, Students (kumo-tabular); Northwind counts, sums, binary and integer regression (kumo-relational), one-shot and in a session

Every refusal matched on status, code and message:

Request	Answer
scores, embeddings, explanation, or probabilities on a regression	422 UNSUPPORTED_OUTPUT
bare LIST_DISTINCT, ranking	422 UNSUPPORTED_TASK
non-empty inference_config, unknown inference option, use_prediction_time on kumo-tabular	400 INVALID_REQUEST
an inference option of the wrong type, such as a run_mode that is not a string	422 VALIDATION_FAILED
a name or label over 1,024 characters, or a table naming a column twice, which the contract refuses	422 VALIDATION_FAILED
text, multicategorical or sequence column; a tabular timestamp feature	422 UNSUPPORTED_DTYPE
a table the model cannot hold, such as a context of only its key and target	422 VALIDATION_FAILED
a relational context with no rows, such as one a holdout of every entity empties	422 VALIDATION_FAILED
a binary context with one class, or one missing a class its task declares, which a holdout can produce	422 TASK_SCHEMA_MISMATCH
any validation refusal while creating a session, or scoring against one	422 SESSION_CREATE_VALIDATION_FAILED, 422 SESSION_PREDICT_SCHEMA_MISMATCH
unknown session	404 SESSION_NOT_FOUND
model id outside the contract	422 VALIDATION_FAILED

Inference options are typed and coerced as the NIM's request model does them, so use_prediction_time: 'no' asks for nothing on both. A sweep of 32 values across the four inference fields, on both models, in one call and in a session, matched the NIM on status, code and message.

And on a GPU. On an NVIDIA L4, I served the same NIM code with its GPU profile (CUDA, bf16 autocast) and ran in-process on the same GPU, with no model passed and so the default device. In-process loaded onto cuda:0 under bf16 autocast, gave identical answers when asked twice, and matched the NIM with a largest difference of 0.0 across all 37 compared answers: tabular and relational, sessions, and every real dataset above. Every refusal probe also matched on status, code and message.

qa/probe_in_process.py repeats the real-data comparison against any NIM. When this process runs on the same device as the NIM, CPU or GPU, it needs no tolerance. With this process on a CPU and the NIM on a GPU, set SDM_IN_PROCESS_TOLERANCE.

Tests

test_a_model_runs_in_this_process.py uses small random-weight models, so it needs no network and no GPU:

• A tabular answer equals what the model returns when called directly through its own tensor API, for int labels, string labels and regression. This is exact equality, not a tolerance.
• A session answers what one call answers, for regression, binary and multiclass.
• Rankings and forecasts are answered in process.
• Each refusal carries the NIM's status and code.
• A context the caller fitted on their own model survives.
• A model is loaded once, on first use, on the default device. A load that fails is 503 MODEL_NOT_READY, with the original error as its cause.
• Configuration mistakes are refused when the client is opened, and a model left in training mode is named.
• A refusal inside a session carries the session's code, and a model loaded for one task alone refuses the other with UNSUPPORTED_TASK before fitting.
• Without the extra, in_process() raises MissingExtraError naming in-process. This is checked in a fresh interpreter with structured-data-models, PyTorch and PyArrow all absent, and also by hand on a clean install of the built wheel.

Two deliberate breaks were each caught: changing the seed, and fitting on the caller's model instead of a copy. CI runs the module in the engine job, which now installs CPU PyTorch and the in-process extra. The other jobs skip it.

Locally: 2707 passed across the four pure-Python packages and the root suite, and 2568 passed in the engine suite. ruff, mypy over all five packages, hygiene and the notices check are clean. I also installed the built wheel with [in-process] into a clean venv and ran the quickstart from outside the checkout, using the published kumo-tabular weights.

Worth a look

• CI installs the wheels it built. structured-data-models-client 1.0.0 reached PyPI while this was in review. After that, --find-links dist with ==1.0.0 resolved the published release instead of the wheel CI had just built, so installed-wheel started testing the release and failed on the new method. installed-wheel, sbom and the build dry-run now name each wheel file, and installed-wheel asserts that all four distributions came from dist/. This affects every pull request against the client from now on, not only this one.
• The hygiene rule changes. It refused the bare name structured-data-models as an internal build a reader cannot look up. That package is now public on GitHub and PyPI, and it is what the new extra installs, so the rule now covers only the internal build name it was written for.
• Text, multicategorical and sequence columns are refused, as the current NIM refuses them. An older deployment answered without them. The client's existing warning at materialize() still says those columns are dropped. It is now wrong against both a current NIM and this transport, and is left for a follow-up.
• The extra needs Python 3.11, because structured-data-models does. It carries a python_version >= '3.11' marker, because without one the workspace lock cannot resolve. As a result, on 3.10 the extra installs nothing, and in_process() says it needs Python 3.11 rather than asking for an extra that was already installed. This was checked on a real 3.10 interpreter.
• uv.lock grows. It is what the open source scan inventories, and it resolves every extra, so it gains structured-data-models, PyTorch, and PyTorch's CUDA wheels for Linux. Nothing already in it moves.
• In-process copies the NIM's request handling, so it has to follow it. It matches the current SDM NIM release. A pending change to the NIM's validation, which refuses a feature that carries no information, will need mirroring here when it lands. The parity probe is how to tell when the two have drifted.
• A deployment's protective limits do not apply in process. These are the limits a NIM sets per deployment through TFM_MAX_*, such as string-cell bytes, distinct values per column and body size. They protect a shared server, and two NIMs may set them differently. The limits this client checks before sending still apply. docs/reference/limits.md says so.
• A payload written by hand is not held to every rule a NIM applies. In-process makes every check that a request built by the client can reach. The client already refuses, before sending, the requests that would reach the NIM's remaining checks, such as a relational target the model cannot represent. A payload written by hand and passed straight to the transport, such as a task with no kind, can be answered differently, for example with a 500 where a NIM answers 422.
• A passed model is used as given. Each context is fitted on a shallow copy that shares its weights, so nothing fitted on the original is disturbed.

Follow-ups, not in this change:

• A predictive query longer than 1,024 characters fails against every NIM today. The relational adapter sends the query text as metadata.source_query, and the contract caps that field at 1,024 characters, so the NIM answers 422 VALIDATION_FAILED. In-process now refuses it the same way. The fix belongs in what the client sends, for example by not sending the text past the cap.
• A table named instance_table. The client lets a user name a table instance_table, which is the wire's own name for the scored rows. The NIM refuses such a request with 422 INVALID_SCHEMA, and so does in-process, so it should be refused when the request is gathered.

CPU against GPU. On a CPU, in-process computes at full precision and draws its random numbers on the CPU. A GPU-hosted NIM does neither, so the two can differ by more than rounding: on the L4, a class probability moved by 0.16 and a regression by 0.41. The quickstart said "the last digits", which understated it, and now gives the measured figure. On the same device, the two are identical.

🤖 Generated with Claude Code