Skip to content

Commit d778ed7

Browse files
Bill LeoutsakosBill Leoutsakos
Bill Leoutsakos
authored and
Bill Leoutsakos
committed
feat(typesafe): add Jev decision integration
1 parent 43c9f6c commit d778ed7

21 files changed

Lines changed: 1341 additions & 5 deletions

File tree

‎apps/docs/components/ui/icon-mapping.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -648,6 +648,7 @@ export const blockTypeToIconMap: Record<string, IconComponent> = {
648648
twilio_sms: TwilioIcon,
649649
twilio_voice: TwilioIcon,
650650
typeform: TypeformIcon,
651+
typesafe: ApiIcon,
651652
upstash: UpstashIcon,
652653
uptimerobot: UptimeRobotIcon,
653654
vanta: VantaIcon,

‎apps/docs/content/docs/integrations/meta.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,7 @@
272272
"twilio_sms",
273273
"twilio_voice",
274274
"typeform",
275+
"typesafe",
275276
"upstash",
276277
"uptimerobot",
277278
"vanta",

‎apps/docs/content/docs/integrations/typesafe.mdx‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
---
2+
title: TypeSafe (Jev)
3+
description: Classify, score, and evaluate input with Jev
4+
---
5+
6+
import { BlockInfoCard } from "@/components/ui/block-info-card"
7+
8+
<BlockInfoCard
9+
type="typesafe"
10+
color="#343A40"
11+
/>
12+
13+
{/* MANUAL-CONTENT-START:intro */}
14+
## Setup and input
15+
16+
Create an API key in the [TypeSafe console](https://console.typesafe.ai/) and enter it in the block’s API Key field, or reference a Sim environment variable. The same four operations are available in the Agent block’s tool picker. The key and model selection are configured by the user.
17+
18+
Input / State accepts ordinary text, a JSON object or array, or an upstream output reference. JSON objects and arrays entered as text are decoded; other text is preserved. Criteria and Questions are JSON fields and reject invalid JSON. Resolved objects and arrays can be supplied directly. Jev evaluates the supplied content; fetch URLs or convert non-text files to text before calling it.
19+
20+
The three convenience operations require a Question. Choice uses a map of 1–255 named options, Score uses an ordered array of 2–10 rubric levels, and Noul accepts optional true/false descriptions. The [structured input format](https://docs.typesafe.ai/primitives/advanced) also supports objects, arrays, and null descriptions. Evaluate accepts optional or null instructions, and preserves the supplied question IDs.
21+
22+
## Independent questions
23+
24+
Evaluate Questions sends all named questions against the shared state in one request. Each question is independent: it cannot read another question’s answer. Use sequential blocks for dependent decisions, such as choosing a parent category before classifying its children. Question IDs identify outputs; describe the actual task in instructions and criteria.
25+
26+
For example, evaluate a ticket with:
27+
28+
```json
29+
{
30+
"team": {
31+
"type": "choice",
32+
"instructions": "Which team should respond?",
33+
"criteria": { "billing": "Invoices and payments", "technical": "Product problems" }
34+
},
35+
"urgency": {
36+
"type": "score",
37+
"instructions": "How urgent is this ticket?",
38+
"criteria": ["Routine", "Time sensitive", "Service unavailable"]
39+
},
40+
"needs_reply": {
41+
"type": "noul",
42+
"instructions": "Does the customer need a reply?"
43+
}
44+
}
45+
```
46+
47+
Read the resulting `answers.team.choice`, `answers.urgency.score`, and `answers.needs_reply.noul`. Convenience operations expose their answer fields directly, such as `choice` or `score`, plus the resolved `model` and token `usage`.
48+
49+
## Interpreting results and routing
50+
51+
A [Score](https://docs.typesafe.ai/primitives/score) is an expected value on a rubric indexed from zero. It can fall between levels: preserve a value such as 1.4 rather than rounding it. The legend maps indices to the original descriptions, including structured values.
52+
53+
Choice and Score return a probability distribution and [confidence](https://docs.typesafe.ai/confidence). A probability describes an outcome; confidence summarizes the distribution’s certainty. Confidence does not guarantee correctness. Low confidence remains a successful result for downstream handling.
54+
55+
[Noul](https://docs.typesafe.ai/primitives/noul) returns only a yes probability between zero and one, including both endpoints. It has no boolean, confidence, explanation, or built-in threshold.
56+
57+
Use a Condition block to compare the relevant probability, score, or confidence with thresholds chosen for your application. For example, split citation support into supported, unsupported, and review bands. Calibrate those bands on representative inputs; the integration does not impose a universal cutoff. See TypeSafe’s [routing guidance](https://docs.typesafe.ai/patterns/intent-routing).
58+
59+
## Models, limits, and errors
60+
61+
The advanced Model combobox defaults to `jev-latest`, suggests `jev-preview` and `jev-1.13.0`, and accepts custom IDs. Aliases can move; a versioned ID pins behavior. Every response includes the resolved model and `usage.input_tokens` / `usage.output_tokens` for the whole request.
62+
63+
The [current model documentation](https://docs.typesafe.ai/models) lists a 64k-token budget for state plus all questions, and a 32k-token budget for state plus the longest question. These limits can change with the model. Sim neither estimates tokens nor silently truncates content. Shorten the state or split independent question batches when the API rejects an oversized request.
64+
65+
Requests use Sim’s shared executor, with two retries, an initial 500 ms delay, a 5-second delay cap, and its existing Retry-After and cancellation handling. This includes throttling (429) and overload (529). Invalid credentials (401) and validation failures (400 or 422) surface as errors. Missing answer fields or mismatched answer types also fail clearly. See the [HTTP API reference](https://docs.typesafe.ai/api).
66+
{/* MANUAL-CONTENT-END */}
67+
68+
69+
## Usage Instructions
70+
71+
Use TypeSafe Jev to choose an option, assign a fractional score, estimate a yes/no probability, or evaluate independent named questions in one request. Supply text or structured state, define criteria, and route results through downstream Condition blocks. All four operations are also available as Agent tools.
72+
73+
74+
75+
## Actions
76+
77+
### TypeSafe Choose an Option
78+
79+
Choose one named option with TypeSafe Jev and return its probability distribution.
80+
81+
#### Input
82+
83+
| Parameter | Type | Required | Description |
84+
| --------- | ---- | -------- | ----------- |
85+
| `state` | string | Yes | Shared input to evaluate: plain text or a JSON-encoded object or array. Resolved workflow objects and arrays are also supported. |
86+
| `instructions` | string | Yes | Question or evaluation instructions. Provide text or a JSON-encoded object or array. |
87+
| `criteria` | json | Yes | Object mapping 1–255 option names to text, structured, or null descriptions. Example: \{"billing":"Invoices and payments","technical":"Product problems"\} |
88+
| `apiKey` | string | Yes | TypeSafe API key |
89+
| `model` | string | No | Model ID or alias, such as jev-latest, jev-preview, or jev-1.13.0. Custom IDs are supported. |
90+
91+
#### Output
92+
93+
| Parameter | Type | Description |
94+
| --------- | ---- | ----------- |
95+
| `choice` | string | Selected option name |
96+
| `probabilities` | json | Dynamic map of option names to probabilities from 0 to 1, including zero values |
97+
| `confidence` | number | Distribution-based confidence from 0 to 1; not a guarantee of correctness |
98+
| `model` | string | Resolved model ID returned by TypeSafe |
99+
| `usage` | object | Token usage for the entire request |
100+
| ↳ `input_tokens` | number | Number of input tokens used |
101+
| ↳ `output_tokens` | number | Number of output tokens used |
102+
103+
### TypeSafe Evaluate Questions
104+
105+
Evaluate independent named Choice, Score, and Noul questions against shared state in one TypeSafe Jev request.
106+
107+
#### Input
108+
109+
| Parameter | Type | Required | Description |
110+
| --------- | ---- | -------- | ----------- |
111+
| `state` | string | Yes | Shared input to evaluate: plain text or a JSON-encoded object or array. Resolved workflow objects and arrays are also supported. |
112+
| `questions` | json | Yes | Nonempty object of named questions, each with type choice, score, or noul. Instructions are optional text, structured values, or null. Choice requires a 1–255 option criteria object; Score requires a 2–10 item criteria array; Noul accepts optional true/false criteria. Example: \{"supported":\{"type":"noul","instructions":"Does the source support the claim?"\}\} |
113+
| `apiKey` | string | Yes | TypeSafe API key |
114+
| `model` | string | No | Model ID or alias, such as jev-latest, jev-preview, or jev-1.13.0. Custom IDs are supported. |
115+
116+
#### Output
117+
118+
| Parameter | Type | Description |
119+
| --------- | ---- | ----------- |
120+
| `answers` | json | Dynamic map keyed by your question IDs. Choice answers contain type, choice, probabilities, confidence; Score answers contain type, score, legend, probabilities, confidence; Noul answers contain type and noul. |
121+
| `model` | string | Resolved model ID returned by TypeSafe |
122+
| `usage` | object | Token usage for the entire request |
123+
| ↳ `input_tokens` | number | Number of input tokens used |
124+
| ↳ `output_tokens` | number | Number of output tokens used |
125+
126+
### TypeSafe Estimate Yes/No Probability
127+
128+
Estimate the probability of a yes answer using TypeSafe Jev. Returns a number, not a boolean.
129+
130+
#### Input
131+
132+
| Parameter | Type | Required | Description |
133+
| --------- | ---- | -------- | ----------- |
134+
| `state` | string | Yes | Shared input to evaluate: plain text or a JSON-encoded object or array. Resolved workflow objects and arrays are also supported. |
135+
| `instructions` | string | Yes | Question or evaluation instructions. Provide text or a JSON-encoded object or array. |
136+
| `criteria` | json | No | Optional descriptions for true and false outcomes, as text, structured values, or null. Example: \{"true":"The source supports the claim","false":"The source does not support the claim"\} |
137+
| `apiKey` | string | Yes | TypeSafe API key |
138+
| `model` | string | No | Model ID or alias, such as jev-latest, jev-preview, or jev-1.13.0. Custom IDs are supported. |
139+
140+
#### Output
141+
142+
| Parameter | Type | Description |
143+
| --------- | ---- | ----------- |
144+
| `noul` | number | Probability of yes from 0 to 1, including exact zero and one. Apply any routing threshold downstream. |
145+
| `model` | string | Resolved model ID returned by TypeSafe |
146+
| `usage` | object | Token usage for the entire request |
147+
| ↳ `input_tokens` | number | Number of input tokens used |
148+
| ↳ `output_tokens` | number | Number of output tokens used |
149+
150+
### TypeSafe Assign a Score
151+
152+
Score input against an ordered rubric using TypeSafe Jev, preserving fractional scores.
153+
154+
#### Input
155+
156+
| Parameter | Type | Required | Description |
157+
| --------- | ---- | -------- | ----------- |
158+
| `state` | string | Yes | Shared input to evaluate: plain text or a JSON-encoded object or array. Resolved workflow objects and arrays are also supported. |
159+
| `instructions` | string | Yes | Question or evaluation instructions. Provide text or a JSON-encoded object or array. |
160+
| `criteria` | array | Yes | Ordered array of 2–10 rubric descriptions, indexed from zero. Descriptions may be text, structured, or null. Example: \["Routine","Time sensitive","Service unavailable"\] |
161+
| `apiKey` | string | Yes | TypeSafe API key |
162+
| `model` | string | No | Model ID or alias, such as jev-latest, jev-preview, or jev-1.13.0. Custom IDs are supported. |
163+
164+
#### Output
165+
166+
| Parameter | Type | Description |
167+
| --------- | ---- | ----------- |
168+
| `score` | number | Expected score on the zero-indexed rubric; may be fractional |
169+
| `legend` | json | Dynamic map of score indices to the original text, structured, or null rubric descriptions |
170+
| `probabilities` | json | Dynamic map of score indices to probabilities from 0 to 1 |
171+
| `confidence` | number | Distribution-based confidence from 0 to 1; not a guarantee of correctness |
172+
| `model` | string | Resolved model ID returned by TypeSafe |
173+
| `usage` | object | Token usage for the entire request |
174+
| ↳ `input_tokens` | number | Number of input tokens used |
175+
| ↳ `output_tokens` | number | Number of output tokens used |
176+
177+

0 commit comments

Comments
 (0)