Quickstart · Jeval

Jeval is a pnpm workspace. Until the packages are published you install from a clone or from locally packed tarballs; the site never claims they are on npm. Node 22 (≥ 20.9) and pnpm 10 are required.

1. Clone and build

git clone <repository-url> jeval && cd jeval
corepack enable            # or: npm i -g pnpm
pnpm install
pnpm build                 # builds @jeval/core, @jeval/provider-jev, @jeval/cli

Run pnpm check to lint, typecheck, build and test everything offline. Nothing here needs a TypeSafe key.

2. Try the bundled example

pnpm --filter jeval-examples cli      # jeval run --mode fixture on examples/data (simulated)
pnpm --filter jeval-examples sdk      # runs examples/sdk-example.ts
open examples/runs/*.html             # self-contained report

Fixture mode uses deterministic simulated answers and is marked SIMULATED in every output. It shows how the framework behaves; it says nothing about how Jev judges.

3. Start your own project

mkdir my-evals && cd my-evals
pnpm --dir ../jeval --filter @jeval/cli exec jeval init .   # or add the packed tarballs to your project
jeval run --mode fixture --html           # simulated answers, offline

jeval init writes jeval.config.json, rubrics.json, a four-case dataset.jsonl, fixtures.json and .env.example without overwriting existing files. To install the CLI into another project, pack the workspace packages (pnpm -r --filter './packages/*' pack) and add the tarballs as dependencies; the repository's scripts/verify-packed.sh does exactly that as a check.

4. Run against Jev

cp .env.example .env         # add TYPESAFE_API_KEY
jeval run --mode live --html --ci         # real Jev; sends cases to TypeSafe
jeval report runs/<run>.json              # self-contained HTML
jeval compare runs/<a>.json runs/<b>.json

Live mode sends the built judge state for each case (input, output, conversation, policy, references and tool events) to TypeSafe using your key. Expected labels and metadata never leave your machine. Usage is measured per request and cost is shown as an estimate from the rate in your config. See Jev integration.

5. Evaluate inside your application

The same API works after your app generates a response. Pass the case, the rubrics and a provider; you get back statuses, reasons, probabilities and request usage.

examples/sdk-example.ts

import { evaluateCase, FixtureProvider, builtinRubric } from "@jeval/core";
import { JevProvider } from "@jeval/provider-jev";

// Real judgments need TYPESAFE_API_KEY and send the case to TypeSafe.
// Without a key this example uses the simulated fixture provider.
const provider = process.env.TYPESAFE_API_KEY
  ? new JevProvider({ model: process.env.TYPESAFE_DEFAULT_MODEL ?? "jev-latest" })
  : new FixtureProvider();

const result = await evaluateCase(
  {
    id: "booking-42",
    input: "Book me a cleaning for Tuesday at 10am.",
    output: "Done! Your cleaning is booked for Tuesday at 10:00.",
    policy: "Only confirm a booking after the book_appointment tool succeeds.",
    toolEvents: [{ id: "t1", name: "book_appointment", status: "failure", error: "slot unavailable" }],
  },
  [builtinRubric("booking-claim"), builtinRubric("policy-compliance")],
  { provider },
);

for (const check of result.checks) {
  console.log(check.rubricId, check.status, "—", check.reason);
}
console.log(result.request?.simulated ? "simulated run" : `model ${result.request?.model}`, result.request?.usage);

Next: Evaluation cases, Rubrics, Reports and CI gates.