Open source · bring your own TypeSafe key

Know when your AI gets it wrong.

Open-source evaluations for AI outputs and agents. Define your checks, run them with Jev, and inspect failures and uncertain results.

Get started Join the Cloud waitlist Try the demo

Small checks, structured answers

Every rubric asks Jev one narrow multiple-choice question about the state you supply: acceptable, unacceptable, or not enough evidence. Jeval turns the returned probabilities into Pass, Fail or Review using thresholds you control, and composes them with exact checks in code, such as whether the booking tool actually returned confirmed.

01Define cases from your app’s inputs, outputs, tool events and references, plus rubrics as JSON or code.
02Run one case after a response is generated, or a JSONL dataset with bounded concurrency, retries and a CI gate.
03Review failures and uncertain results in a local HTML report, and compare runs to catch regressions.

examples/sdk-example.ts

import { evaluateCase, FixtureProvider, builtinRubric } from "@jeval/core";
import { JevProvider } from "@jeval/provider-jev";

// Real judgments need TYPESAFE_API_KEY and send the case to TypeSafe.
// Without a key this example uses the simulated fixture provider.
const provider = process.env.TYPESAFE_API_KEY
  ? new JevProvider({ model: process.env.TYPESAFE_DEFAULT_MODEL ?? "jev-latest" })
  : new FixtureProvider();

const result = await evaluateCase(
  {
    id: "booking-42",
    input: "Book me a cleaning for Tuesday at 10am.",
    output: "Done! Your cleaning is booked for Tuesday at 10:00.",
    policy: "Only confirm a booking after the book_appointment tool succeeds.",
    toolEvents: [{ id: "t1", name: "book_appointment", status: "failure", error: "slot unavailable" }],
  },
  [builtinRubric("booking-claim"), builtinRubric("policy-compliance")],
  { provider },
);

for (const check of result.checks) {
  console.log(check.rubricId, check.status, "—", check.reason);
}
console.log(result.request?.simulated ? "simulated run" : `model ${result.request?.model}`, result.request?.usage);

See a check decide

Open the full demo

Interactive example

Illustrative demo: simulated results

1. Pick a scenario

Booking confirmed, tool succeededThe assistant says the appointment is booked and the booking tool actually returned a confirmed result.Booking confirmed, but the tool failedThe assistant tells the patient the slot is booked even though the booking tool returned an error.Factual claim, reference material missingThe assistant states how long a whitening treatment lasts, but the supplied reference material only covers opening hours.

2. Select checks

booking-claimA claimed booking is backed by a successful book_appointment tool result.policy-complianceThe response follows the supplied assistant policy.claim-supportFactual claims are supported by the supplied reference material.no-unverified-promisesThe response does not promise outcomes, discounts or refunds.

3. Run

Live mode is off on this deployment. Run real evaluations locally with the CLI and your own TypeSafe key.

Built for the failures that matter

Support assistants

Did the reply follow the refund policy? Did it promise something the policy forbids?

Booking and action agents

Did the assistant confirm an action that the tool log says failed, or never ran?

Retrieval applications

Is the stated price, date or duration actually in the retrieved passages, or unsupported?

Escalation rules

When the policy required a hand-off to a human, was the escalation tool actually called?

Two ways to use Jeval

Available now

Jeval open source

MIT licensed. No Jeval account. Bring your own TypeSafe API key; runs and reports stay on your machine.

TypeScript SDK for single cases and datasets
CLI: init, run, report, compare, benchmark
Five starter rubric kinds plus custom criteria
Local HTML reports and CI exit codes
Deterministic fixture mode for offline tests

Quickstart

Planned · waitlist

Jeval Cloud

A managed service we are designing. None of this exists yet; the waitlist tells us what to build first.

Managed runs without your own provider key
Saved history and run-over-run comparisons
Shared dashboards for a team
Alerts on new failures
Review queues for uncertain results

Join the waitlist

Questions

Is Jeval free?: The framework is MIT-licensed and free. Real evaluations call TypeSafe's Jev with your own API key, and TypeSafe may bill that usage. Jeval estimates cost from a rate you configure and labels it as an estimate.
Do I need a Jeval account?: No. The open-source SDK and CLI work from a clone with your own provider key. Jeval Cloud is a planned managed service and is not available yet.
What do I need for real runs?: A TypeSafe API key. Without one you can still run everything in fixture mode, which uses deterministic simulated answers and is labelled as such everywhere.
What does 'review' mean?: The judge's probabilities did not reach your pass or fail threshold, or the evidence needed for the check was missing. Review results need a person; they never count as passes, and a strict CI run treats them as incomplete.
Does running locally keep my data local?: Reports and datasets stay on your machine, but a live run sends the selected fields of each case (input, output, policy, references, tool events) to TypeSafe. Expected labels and metadata are never sent. Fixture mode sends nothing.
Is Jeval made by TypeSafe?: No. Jeval is an independent open-source project that uses TypeSafe's Jev as its first judge provider. It is not affiliated with or endorsed by TypeSafe.
Is the judge immune to prompt injection?: No. Evaluated text is marked as untrusted in every question, and the examples include adversarial cases, but this is a mitigation with observed limits, not immunity. See the limitations page.

Jeval Cloud waitlist

Managed runs, saved history and team dashboards are planned. Leave your email to hear about early access.