Open source · bring your own TypeSafe key
Know when your AI gets it wrong.
Open-source evaluations for AI outputs and agents. Define your checks, run them with Jev, and inspect failures and uncertain results.
- jeval capture01Import real tracesOpenAI chat, OTel, Langfuse, LangWatch
- jeval estimate02Price it firsttokens and cost before anything is sent
- jeval run03Judge every caseone Jev request per case, all checks batched
- jeval review04Decide the uncertainhuman labels, not guesses
- jeval compare05Catch regressionsnew and resolved failures by case id
- jeval run --ci06Gate the releaseexit 0 only when the gates are met
One cycle, end to end
From last week’s traffic to a release gate
Import the conversations your app already logged, see what a run will cost before it starts, judge a sample, label the uncertain ones yourself, then compare against yesterday’s run. A failure that appears on a stable case id blocks the release; a Review never counts as a pass.
- Bring your own TypeSafe key, or route through Vercel AI Gateway with no TypeSafe account.
- Every judgment keeps its probabilities, the rule applied in code and the evidence it saw.
- Nothing is sent anywhere until you choose live mode; fixture mode runs offline and says so.
What you can do today
Evaluate what you already log
Import OpenAI-style chat transcripts, OpenTelemetry GenAI spans, Langfuse or LangWatch exports, or map any JSON. Tool calls become tool events.
jeval capture traces.jsonl -o dataset.jsonlFive checks that compose with code
Policy compliance, claim support against references, tool-backed action claims, required escalations, and custom criteria. Exact conditions are checked in code, the judge answers one narrow question.
{ "kind": "tool-claim", "params": { "toolName": "book_appointment" } }Know the cost before you run
Estimate requests, input tokens and price from the exact state a run would send. Refuse to start above a budget. Sample five cases with the judged state beside each verdict.
jeval run --mode live --limit 5 --show --max-usd 0.50Review, not guess
Missing evidence, flat probabilities and uncovered claims come back as review. Walk them interactively and record human labels that the benchmark reports by source.
jeval review runs/live.json --labels human.jsonlRegressions by stable id
Compare two runs case by case and rubric by rubric: new failures, resolved failures, new reviews. Different datasets, rubric versions, models or modes are flagged, never silently merged.
jeval compare runs/a.json runs/b.json --fail-on-newGates you can trust in CI
Exit 0 only when the configured gates are met and every required check reached a decision. Review, errors, an empty dataset or a simulated run cannot yield a clean pass.
jeval run --mode live --ci # 0 ok · 1 gate failed · 2 incompleteSmall checks, structured answers
Every rubric asks Jev one narrow multiple-choice question about the state you supply: acceptable, unacceptable, or not enough evidence. jeval turns the returned probabilities into Pass, Fail or Review using thresholds you control, and composes them with exact checks in code, such as whether the booking tool actually returned confirmed.
- 01Define cases from your app’s inputs, outputs, tool events and references, plus rubrics as JSON or code.
- 02Run one case after a response is generated, or a JSONL dataset with bounded concurrency, retries and a CI gate.
- 03Review failures and uncertain results in a local HTML report, and compare runs to catch regressions.
import { evaluateCase, FixtureProvider, builtinRubric } from "@jeval/core";
import { JevProvider } from "@jeval/provider-jev";
// Real judgments need TYPESAFE_API_KEY and send the case to TypeSafe.
// Without a key this example uses the simulated fixture provider.
const provider = process.env.TYPESAFE_API_KEY
? new JevProvider({ model: process.env.TYPESAFE_DEFAULT_MODEL ?? "jev-latest" })
: new FixtureProvider();
const result = await evaluateCase(
{
id: "booking-42",
input: "Book me a cleaning for Tuesday at 10am.",
output: "Done! Your cleaning is booked for Tuesday at 10:00.",
policy: "Only confirm a booking after the book_appointment tool succeeds.",
toolEvents: [{ id: "t1", name: "book_appointment", status: "failure", error: "slot unavailable" }],
},
[builtinRubric("booking-claim"), builtinRubric("policy-compliance")],
{ provider },
);
for (const check of result.checks) {
console.log(check.rubricId, check.status, "—", check.reason);
}
console.log(result.request?.simulated ? "simulated run" : `model ${result.request?.model}`, result.request?.usage);
See a check decide
Open the full demoInteractive example
Illustrative demo: simulated results
Built for the failures that matter
Support assistants
Did the reply follow the refund policy? Did it promise something the policy forbids?
Booking and action agents
Did the assistant confirm an action that the tool log says failed, or never ran?
Retrieval applications
Is the stated price, date or duration actually in the retrieved passages, or unsupported?
Escalation rules
When the policy required a hand-off to a human, was the escalation tool actually called?
Two ways to use jeval
Available now
jeval open source
MIT licensed. No jeval account. Bring your own TypeSafe API key; runs and reports stay on your machine.
- TypeScript SDK for single cases and datasets
- CLI: init, run, report, compare, benchmark
- Five starter rubric kinds plus custom criteria
- Local HTML reports and CI exit codes
- Deterministic fixture mode for offline tests
Planned · waitlist
jeval Cloud
A managed service we are designing. None of this exists yet; the waitlist tells us what to build first.
- Managed runs without your own provider key
- Saved history and run-over-run comparisons
- Shared dashboards for a team
- Alerts on new failures
- Review queues for uncertain results
One recorded run on human-labelled public data
What Jev did on 300 RAGTruth responses
88.3% / 98.1%
failure-detection precision / recall on the 205 cases it decided, sentence by sentence (TP 106, FP 14, FN 2, TN 83)
32%
abstained as review rather than guessed; a 0.60 threshold decides 96% of sentences at 90% accuracy
992k tokens
input for 300 cases, about $0.04 at the documented rate, zero execution errors
363 ms
median request latency, one run
RAGTruth’s own human span-level labels, balanced across QA, summarisation and data-to-text, rubric wording and thresholds fixed on a separate tuning split first. One run, one day; it still over-flags some data-to-text paraphrase, and it says nothing about your data. Per-task tables and the reproduction script are in the repository’s examples/public-benchmarks. How to measure it on yours.
Questions
- Is jeval free?
- The framework is MIT-licensed and free. Real evaluations call TypeSafe's Jev with your own API key, and TypeSafe may bill that usage. jeval estimates cost from a rate you configure and labels it as an estimate.
- Do I need a jeval account?
- No. The open-source SDK and CLI work from a clone with your own provider key. jeval Cloud is a planned managed service and is not available yet.
- What do I need for real runs?
- A TypeSafe API key. Without one you can still run everything in fixture mode, which uses deterministic simulated answers and is labelled as such everywhere.
- What does 'review' mean?
- The judge's probabilities did not reach your pass or fail threshold, or the evidence needed for the check was missing. Review results need a person; they never count as passes, and a strict CI run treats them as incomplete.
- Does running locally keep my data local?
- Reports and datasets stay on your machine, but a live run sends the selected fields of each case (input, output, policy, references, tool events) to TypeSafe. Expected labels and metadata are never sent. Fixture mode sends nothing.
- Is jeval made by TypeSafe?
- No. jeval is an independent open-source project that uses TypeSafe's Jev as its first judge provider. It is not affiliated with or endorsed by TypeSafe.
- Is the judge immune to prompt injection?
- No. Evaluated text is marked as untrusted in every question, and the examples include adversarial cases, but this is a mitigation with observed limits, not immunity. See the limitations page.
jeval Cloud waitlist
Managed runs, saved history and team dashboards are planned. Leave your email to hear about early access.