Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 16 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,31 +95,32 @@ Every question has an `id`, unique within the list.

`input.config` for `evaluate`, or the argument to `createSystemOneAdapter`:

| Field | Type | Default | Description |
| ----------- | ---------------- | ---------------------- | -------------------------------------------------------------------------------- |
| `endpoint` | `EndpointConfig` | `{ kind: "official" }` | `official`, `gateway`, or `custom` with a `url`. Each takes an optional `model`. |
| `timeoutMs` | `number` | `1500` | Bound for the whole round trip. |
| `apiKey` | `string` | from the environment | Overrides the environment key. |
| Field | Type | Default | Description |
| ----------- | ---------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `endpoint` | `EndpointConfig` | `{ kind: "official" }` | `official`, `gateway`, or `custom` with a `url`. Each takes an optional `model`. `custom` may set `auth: "none"`. |
| `timeoutMs` | `number` | `1500` | Bound for the whole round trip. |
| `apiKey` | `string` | from the environment | Overrides the environment key. |

### Endpoints and keys

| `kind` | URL | Default model | Key from the environment |
| ---------- | ---------------------------------------------------- | ----------------- | ---------------------------------------------- |
| `official` | `https://api.typesafe.ai/v1/systemone` | `jev-latest` | `TYPESAFE_API_KEY` |
| `gateway` | `https://ai-gateway.vercel.sh/typesafe/v1/systemone` | `typesafe-ai/jev` | `AI_GATEWAY_API_KEY`, then `VERCEL_OIDC_TOKEN` |
| `custom` | `url` | `typesafe-ai/jev` | `SYSTEM_ONE_API_KEY` |
| `custom` | `url` | `typesafe-ai/jev` | `SYSTEM_ONE_API_KEY`, unless `auth: "none"` |

`SYSTEM_ONE_API_KEY` is also a supported alias on every endpoint, read after the endpoint's own key.
A custom endpoint with `auth: "none"` does not require a key and does not send `Authorization`. Official and gateway always require a key. Any other `auth` value is invalid input and throws.

### Fallback reasons

| `reason` | When |
| ------------- | ------------------------------------------------------------- |
| `no-key` | No key in `apiKey` or the environment. No request is sent. |
| `timeout` | The round trip, including the body read, exceeds `timeoutMs`. |
| `network` | The request fails before a response. |
| `http-error` | Non-2xx response. `httpStatus` holds the status. |
| `parse-error` | The body is not JSON or fails validation. |
| `reason` | When |
| ------------- | -------------------------------------------------------------------------------------------------------------- |
| `no-key` | No key in `apiKey` or the environment, and the endpoint is not custom with `auth: "none"`. No request is sent. |
| `timeout` | The round trip, including the body read, exceeds `timeoutMs`. |
| `network` | The request fails before a response. |
| `http-error` | Non-2xx response. `httpStatus` holds the status. |
| `parse-error` | The body is not JSON or fails validation. |

`detail` carries the underlying message.

Expand Down Expand Up @@ -208,7 +209,7 @@ The hub offering's `quirks` are validated and all optional:
}
```

Without `endpoint`, the adapter posts to the catalog provider's `baseURL` + `/systemone`, the host the offering's credential was issued for. A `model` quirk (top level or on `endpoint`) is the model sent on the wire, so one catalog model can be served by providers that name it differently. `questions` and `state` are per-call defaults; `providerOptions.systemOne` on a call wins field by field. Unknown keys throw. The adapter sends the bearer sentinel and the harness injects the offering's credential.
Without `endpoint`, the adapter posts to the catalog provider's `baseURL` + `/systemone`, the host the offering's credential was issued for. A `model` quirk (top level or on `endpoint`) is the model sent on the wire, so one catalog model can be served by providers that name it differently. `questions` and `state` are per-call defaults; `providerOptions.systemOne` on a call wins field by field. Unknown keys throw. The adapter sends the bearer sentinel and the harness injects the offering's credential, except a custom endpoint with `auth: "none"`, which omits `Authorization` so the host does not attach a key.

| Provider | Provider `baseURL` | `model` quirk |
| ----------------- | ------------------------------------------ | ----------------------------- |
Expand All @@ -221,7 +222,7 @@ Without `endpoint`, the adapter posts to the catalog provider's `baseURL` + `/sy

- `createSystemOneAdapterFactory` without an `endpoint` quirk now posts to the catalog provider's `baseURL` + `/systemone` instead of the official TypeSafe URL. Set the provider `baseURL` to the API root (for example `https://api.typesafe.ai/v1`), or keep the old behavior with `{"endpoint":{"kind":"official"}}`. The harness sends the offering's credential to whatever URL the adapter returns, so only set an explicit `endpoint` on a host that credential belongs to.
- A `model` quirk now overrides the model the harness passes (the catalog canonical name).
- `createSystemOneAdapter` and `evaluate` are unchanged.
- `createSystemOneAdapter` and `evaluate` accept `auth: "none"` on a custom endpoint and omit `Authorization`. Any other `auth` value throws `SystemOneError` before a request is built.

## Upgrading from 0.1

Expand Down
49 changes: 49 additions & 0 deletions src/adapter.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
createSystemOneAdapter,
createSystemOneAdapterFactory,
} from "./adapter";
import { SystemOneError } from "./errors";

describe("adapter", () => {
function turns(): ConversationTurn[] {
Expand Down Expand Up @@ -122,6 +123,54 @@ describe("adapter factory", () => {
{ role: "user", timestamp: 0, content: [{ type: "text", text: "Hi" }] },
];

test("buildRequest omits Authorization for custom auth none and keeps the sentinel otherwise", () => {
const turns = (): ConversationTurn[] => [
{ role: "user", timestamp: 0, content: [{ type: "text", text: "Hi" }] },
];
const keyless = createSystemOneAdapter({
endpoint: {
kind: "custom",
url: "https://local.test/eval",
auth: "none",
},
}).buildRequest(turns(), "jev-latest", {});
expect(keyless.headers["authorization"]).toBeUndefined();
expect(Object.keys(keyless.headers)).not.toContain("authorization");

const official = createSystemOneAdapter({
endpoint: { kind: "official" },
}).buildRequest(turns(), "jev-latest", {});
expect(official.headers["authorization"]).toBe(BEARER_CREDENTIAL_SENTINEL);

const gateway = createSystemOneAdapter({
endpoint: { kind: "gateway" },
}).buildRequest(turns(), "jev-latest", {});
expect(gateway.headers["authorization"]).toBe(BEARER_CREDENTIAL_SENTINEL);

const keyedCustom = createSystemOneAdapter({
endpoint: { kind: "custom", url: "https://h.test/e" },
}).buildRequest(turns(), "jev-latest", {});
expect(keyedCustom.headers["authorization"]).toBe(
BEARER_CREDENTIAL_SENTINEL,
);
});

test("invalid auth throws before a request is built", () => {
let thrown: unknown;
try {
createSystemOneAdapter({
endpoint: {
kind: "custom",
url: "https://h.test/e",
auth: "bearer",
},
} as unknown as Parameters<typeof createSystemOneAdapter>[0]);
} catch (cause) {
thrown = cause;
}
expect(thrown).toBeInstanceOf(SystemOneError);
});

test("no quirks routes to the provider baseURL with sentinel auth", () => {
const request = createSystemOneAdapterFactory(source).buildRequest(
turns(),
Expand Down
21 changes: 17 additions & 4 deletions src/adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,16 @@ import type {
} from "@intx/types/runtime";

import { resolveEndpoint } from "./config.js";
import { SystemOneError } from "./errors.js";
import {
Decision,
toDecision,
toWireQuestions,
WireResponseBody,
EndpointConfig,
EvaluateConfig,
QuestionList,
State,
type EvaluateConfig,
type WireUsage,
} from "./schemas.js";

Expand Down Expand Up @@ -158,7 +159,9 @@ function parseDecisions(
* host's single copy of `@intx/inference` (peer dependency, never vendored).
* The optional `config` pins the endpoint (official by default, gateway, or
* a custom URL) and the per-call model default; the inference call's own
* `model` argument wins for the request body.
* `model` argument wins for the request body. Official and gateway requests
* carry `authorization: BEARER_CREDENTIAL_SENTINEL` so the host injects a
* key. A custom endpoint with `auth: "none"` omits that header.
*/
function createSystemOneProviderAdapter(
config?: EvaluateConfig,
Expand Down Expand Up @@ -193,8 +196,10 @@ function createSystemOneProviderAdapter(
const headers: Record<string, string> = {
"content-type": "application/json",
accept: "application/json",
authorization: BEARER_CREDENTIAL_SENTINEL,
};
if (endpoint.auth !== "none") {
headers["authorization"] = BEARER_CREDENTIAL_SENTINEL;
}
return {
url: endpoint.url,
headers,
Expand Down Expand Up @@ -264,7 +269,15 @@ function createSystemOneProviderAdapter(
export function createSystemOneAdapter(
config?: EvaluateConfig,
): ProviderAdapter {
return createSystemOneProviderAdapter(config);
if (config === undefined) return createSystemOneProviderAdapter();
const parsed = EvaluateConfig(config);
if (parsed instanceof type.errors) {
throw new SystemOneError(
"parse-error",
`createSystemOneAdapter: invalid config: ${parsed.summary}`,
);
}
return createSystemOneProviderAdapter(parsed);
}

// Deployment quirks a hub offering carries for this provider. `questions`
Expand Down
6 changes: 6 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ export type ResolvedEndpoint = {
url: string;
backend: "corbits-system-one" | "gateway" | "custom";
model: string;
/** `none` only when a custom endpoint explicitly opts out of a key. */
auth: "key" | "none";
};

/**
Expand All @@ -79,6 +81,8 @@ export type ResolvedEndpoint = {
* distinguishes caller-routed traffic from the official endpoint —
* `'gateway'` is reserved for the proxy quirks path. The model always
* resolves here: a `custom` target without one sends `typesafe-ai/jev`.
* `auth` is `"none"` only when the caller set `auth: "none"` on a custom
* endpoint; every other target still requires a key.
*/
export function resolveEndpoint(endpoint?: EndpointConfig): ResolvedEndpoint {
if (endpoint?.kind === "custom") {
Expand All @@ -92,6 +96,7 @@ export function resolveEndpoint(endpoint?: EndpointConfig): ResolvedEndpoint {
url: endpoint.url,
backend: "custom",
model: endpoint.model ?? GATEWAY_QUIRKS.model,
auth: endpoint.auth === "none" ? "none" : "key",
};
}
const kind = endpoint?.kind ?? "official";
Expand All @@ -108,5 +113,6 @@ export function resolveEndpoint(endpoint?: EndpointConfig): ResolvedEndpoint {
url,
backend: kind === "gateway" ? "gateway" : "corbits-system-one",
model: endpoint?.model ?? quirks.model,
auth: "key",
};
}
107 changes: 107 additions & 0 deletions src/evaluate.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,113 @@ async function bearerFor(
return seen.authorization;
}

describe("custom auth none", () => {
const keyless: EndpointConfig = {
kind: "custom",
url: "https://local.test/eval",
auth: "none",
};

test("skips no-key and sends no Authorization, even with a key present", async () => {
process.env["SYSTEM_ONE_API_KEY"] = "legacy";
let authorization: string | null = "unset";
let called = false;
const result = await evaluate(
{
state: {},
questions,
config: { endpoint: keyless, apiKey: "should-not-send" },
},
{
deps: {
fetch: (_input, init) => {
called = true;
authorization = new Headers(init?.headers).get("authorization");
return Promise.resolve(
new Response(JSON.stringify({ answers: answers() }), {
status: 200,
headers: { "content-type": "application/json" },
}),
);
},
scheduler: createDefaultScheduler(),
},
},
);
expect(called).toBe(true);
expect(authorization).toBeNull();
if (result.fallback !== false) {
throw new Error(`expected decisions, got ${JSON.stringify(result)}`);
}
expect(result.decisions).toHaveLength(3);
});

test("official and gateway still fail closed with no-key and do not fetch", async () => {
const deps = {
fetch: () => Promise.reject(new Error("fetch must not be called")),
scheduler: createDefaultScheduler(),
};
const official = await evaluate(
{ state: {}, questions, config: { endpoint: { kind: "official" } } },
{ deps },
);
const gateway = await evaluate(
{ state: {}, questions, config: { endpoint: { kind: "gateway" } } },
{ deps },
);
const custom = await evaluate(
{
state: {},
questions,
config: { endpoint: { kind: "custom", url: "https://h.test/e" } },
},
{ deps },
);
expect(official).toMatchObject({ fallback: true, reason: "no-key" });
expect(gateway).toMatchObject({ fallback: true, reason: "no-key" });
expect(custom).toMatchObject({ fallback: true, reason: "no-key" });
});

test("auth other than none is invalid input and throws", async () => {
let thrown: unknown;
try {
await evaluate({
state: {},
questions,
config: {
endpoint: {
kind: "custom",
url: "https://h.test/e",
auth: "bearer",
} as unknown as EndpointConfig,
},
});
} catch (cause) {
thrown = cause;
}
expect(thrown).toBeInstanceOf(SystemOneError);
});

test("official rejects an auth field", async () => {
let thrown: unknown;
try {
await evaluate({
state: {},
questions,
config: {
endpoint: {
kind: "official",
auth: "none",
} as unknown as EndpointConfig,
},
});
} catch (cause) {
thrown = cause;
}
expect(thrown).toBeInstanceOf(SystemOneError);
});
});

describe("credential precedence", () => {
test("official prefers TYPESAFE_API_KEY, then SYSTEM_ONE_API_KEY", async () => {
process.env["SYSTEM_ONE_API_KEY"] = "legacy";
Expand Down
13 changes: 10 additions & 3 deletions src/evaluate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,9 @@ export type EvaluateOptions = {
* `TYPESAFE_API_KEY` for official, `AI_GATEWAY_API_KEY` then
* `VERCEL_OIDC_TOKEN` for gateway — then the `SYSTEM_ONE_API_KEY` alias,
* then a fail-closed `'no-key'` fallback with zero fetch calls.
* A custom endpoint with `auth: "none"` skips that check and sends no
* `Authorization` header, even if a key is present. Official and gateway
* always require a key; a custom endpoint without `auth: "none"` does too.
* A `timeoutMs` that is not a finite number >= 0 falls back to
* `DEFAULT_TIMEOUT_MS`. Transport failures map to matching fallback reasons
* (`'timeout'`, `'network'`, `'http-error'`, or `'parse-error'` for a 2xx
Expand Down Expand Up @@ -282,7 +285,11 @@ export async function evaluate(
const rawApiKey = parsed.config?.apiKey;
let apiKey: string | undefined;
let keySources: string;
if (rawApiKey !== undefined && rawApiKey !== "") {
const keyless = endpoint.auth === "none";
if (keyless) {
apiKey = undefined;
keySources = "";
} else if (rawApiKey !== undefined && rawApiKey !== "") {
apiKey = rawApiKey;
keySources = "config.apiKey";
} else if (endpoint.backend === "corbits-system-one") {
Expand Down Expand Up @@ -332,7 +339,7 @@ export async function evaluate(
};
emit(startEvent);

if (apiKey === undefined || apiKey === "") {
if (!keyless && (apiKey === undefined || apiKey === "")) {
return fallback("no-key", {
detail: `no API key supplied; set config.apiKey or ${keySources}`,
});
Expand All @@ -356,7 +363,7 @@ export async function evaluate(
const response = await postEvaluate(
endpoint.url,
wire,
{ timeoutMs, apiKey },
apiKey === undefined ? { timeoutMs } : { timeoutMs, apiKey },
deps,
);
data = response.data;
Expand Down
3 changes: 3 additions & 0 deletions src/schemas.ts
Original file line number Diff line number Diff line change
Expand Up @@ -138,10 +138,13 @@ export type QuestionList = typeof QuestionList.infer;
// `gateway` the proxy endpoint, `custom` a caller-supplied URL. `url` only
// exists on `custom` — a `url` on another kind would be silently ignored,
// so the schema rejects it instead of accepting misleading config.
// `auth: "none"` exists only on `custom`: it is the explicit opt-out from
// the fail-closed key requirement. Official and gateway cannot carry it.
export const EndpointConfig = type({
kind: "'custom'",
url: "string",
"model?": "string",
"auth?": "'none'",
"+": "reject",
}).or(
type({
Expand Down
Loading