Jeval

CLI

jeval init, run, report, compare, benchmark and labels.

The CLI ships as @jeval/cli with a jeval executable. Commands print nothing sensitive by default: no API keys and no case content unless you pass --show.

jeval init [dir]

Creates jeval.config.json, rubrics.json, dataset.jsonl, fixtures.json, .env.example and a README. Existing files are kept.

jeval capture traces.jsonl -o dataset.jsonl

Turns exported traces into cases so you can evaluate last week's real conversations. Formats: OpenAI-style chat message arrays (with tool calls and tool results linked into tool events), OpenTelemetry GenAI spans (JSONL or OTLP envelopes), Langfuse traces with observations, LangWatch traces with spans, or --format generic --map input=path --map output=path … for anything else.--format auto sniffs the shape. The first system message becomes the policy unless --policy-file is given. Tool failure is inferred from explicit error levels, or from outputs that look like errors; pass --no-tool-status-heuristic to drop events whose status the source does not state. Which fields were verified against each vendor's documentation and which were inferred is recorded in the repository's docs/import-formats.md.

jeval capture exports/langfuse-traces.json -o data/last-week.jsonl --metadata-fields userId,tags
jeval estimate --dataset data/last-week.jsonl
jeval run --mode live --dataset data/last-week.jsonl --limit 5 --show

jeval run --mode fixture|live

jeval run --mode fixture --html                 # simulated, offline, labelled SIMULATED
jeval run --mode live --model jev-1.13.0 --ci   # real Jev; exit code follows the CI policy
jeval run --mode live --concurrency 8 --timeout-ms 20000 --max-attempts 3 --label commit=$GIT_SHA

Mode is always explicit. Every case's questions go in one request; transient provider errors (rate limits, timeouts, 5xx) are retried with backoff and Retry-After up to maxAttempts; other errors are recorded on the case and the run continues. Ctrl-C cancels remaining cases and still writes the report. Output is runs/run-<timestamp>-<mode>.json plus HTML with --html.

Sample before you scale

jeval estimate                              # requests, input tokens and cost, nothing sent
jeval run --mode live --limit 5 --show      # judge five cases and print the judged state beside each verdict
jeval run --mode live --max-usd 0.50        # refuse to start if the estimate exceeds the budget

estimate builds the exact state and questions a run would send and sizes them with a character heuristic (characters ÷ 4). It is not a tokenizer; on the example datasets it under-estimated measured usage by about 10%. Compare it with a small live run on your own data. Cost is priced from the pricing entry in your config and shown as an estimate, or reported as unknown.

Long conversations

States above the digest budget (110,000 characters by default, roughly 27k tokens, under Jev's documented 32k state limit) are shortened before sending: the earliest and latest turns are kept (25% / 75% of the budget), the omitted middle is replaced by one marker message, and oversized text fields are cut with a visible note. What was omitted is recorded on every check's evidence. Configure with "digest": { "maxChars": 60000 } or disable with "digest": false.

jeval review run.json --labels human.jsonl

Walks every review result (or --statuses review,fail), shows the reason, the probabilities and, with --dataset, the judged input and output, and records your decision as a label with source: human and your reviewer name. Already-labelled checks are skipped, so you can stop and resume. The labels file feeds jeval benchmark directly, which reports how much of a benchmark rests on human versus synthetic labels.

jeval report run.json

Renders a self-contained HTML file: no scripts, a strict Content-Security-Policy, every string escaped. Cases with critical failures are listed first.

jeval compare baseline.json candidate.json

Aligns checks by case id and rubric id, listing new failures, resolved failures, new reviews and new errors. Different datasets, rubric versions, models, thresholds or modes are flagged; changed rubrics and mixed simulated/live runs are treated as not comparable (exit 2). Add --fail-on-new to exit 1 on new failures.

jeval benchmark run.json --labels labels.jsonl

Scores predicted statuses against separately supplied labels. See the benchmarking page.

jeval labels dataset.jsonl --out labels.jsonl

Extracts the provisional expected entries embedded in a dataset into a labels file, so labels and predictions stay separate.

jeval.config.json

{
  "dataset": "./dataset.jsonl",
  "rubrics": "./rubrics.json",
  "output": "./runs",
  "provider": { "model": "jev-latest", "fixtures": "./fixtures.json", "strictFixtures": false },
  "thresholds": { "pass": 0.75, "fail": 0.5 },
  "concurrency": 4, "timeoutMs": 30000, "maxAttempts": 3,
  "pricing": { "inputPerMillionTokensUsd": 0.042, "asOf": "2026-09-19", "source": "https://docs.typesafe.ai/models" },
  "ci": { "maxFailures": 0, "maxReviewRate": 0, "minDecidedCoverage": 1, "allowSimulated": false }
}

pricing is optional. When present, cost is reported as an estimate with the rate and its date; when absent, cost is reported as unknown, never as zero. Fixture runs never report a cost.