Jev integration

The first judge provider is @jeval/provider-jev, built on @typesafe-ai/sdk 0.6.0. The integration was verified on 2026-09-19 against the installed type declarations and the pages at docs.typesafe.ai (JavaScript SDK, choice primitive, confidence, models, API reference). If TypeSafe changes its API, the adapter's tests will tell you.

What is sent

One POST /v1/systemone per case with state (the judge state built from the case) and a questions map of choice questions, one per semantic check part. The model is configurable (jev-latest by default; the docs also list jev-1.13.0 and jev-preview), and the versioned model id the server actually used is stored on every case result.

import { JevProvider } from "@jeval/provider-jev";
const provider = new JevProvider({ model: "jev-1.13.0", timeoutMs: 20_000 }); // apiKey defaults to TYPESAFE_API_KEY

Via Vercel AI Gateway

Vercel's AI Gateway proxies TypeSafe's native API, so you can run Jeval without a TypeSafe account and be billed by Vercel instead. Point the same adapter at the gateway; nothing else changes. Vercel requires a card on file before it serves requests, and the model id returned is the gateway's (typesafe-ai/jev) rather than the versioned Jev id.

.env

TYPESAFE_API_KEY=<AI Gateway API key or VERCEL_OIDC_TOKEN>
TYPESAFE_BASE_URL=https://ai-gateway.vercel.sh/typesafe
TYPESAFE_DEFAULT_MODEL=typesafe-ai/jev

Probabilities and confidence

Each answer carries the selected option, a full probability distribution over the options, and a confidence number. Per TypeSafe's documentation, confidence is a statistic computed from the shape of that distribution: it says how peaked the answer is, not whether the answer is right. Jeval decides using the probabilities and your thresholds, stores confidence alongside them, and never presents it as accuracy.

Usage, cost and limits

Token usage is returned per request and recorded once per case, so a request carrying five questions is not counted five times. Cost is an estimate: the CLI multiplies measured input tokens by the rate in your config and records that rate and its date. The documented rate on the verification date was listed on TypeSafe's models page; check it before relying on it. Documented limits: 64k tokens per request, 32k for state plus the longest question, 255 options per question. Rate limits are enforced server-side and are adjusted dynamically; Jeval retries 429 and 5xx responses with backoff and honours Retry-After.

Errors

Authentication and validation errors fail fast. Rate limits, timeouts, connection failures and server errors (including 529 overloaded) are transient and retried up to maxAttempts. Any response that lacks an answer, offers different options than asked, or whose probabilities do not sum to one is reported as an error, never turned into a decision.

Fixture mode

FixtureProvider returns hand-authored or deterministic fallback answers, reports zero latency and marks every response simulated. It exists for development and tests. Jeval never substitutes it for a failed real provider.