Benchmarking the judge
Score predicted outcomes against labels; tuning versus held-out splits; calibration.
Before trusting a judge on your data, measure it. jeval benchmark compares a run's predicted statuses with labels you supply separately; labels are never sent to the judge.
jeval run --mode live --dataset data/dataset.tuning.jsonl -o runs/tuning.json
jeval benchmark runs/tuning.json --labels data/labels.tuning.jsonl
# adjust thresholds in jeval.config.json, re-run on tuning only, then:
jeval run --mode live --dataset data/dataset.holdout.jsonl -o runs/holdout.json
jeval benchmark runs/holdout.json --labels data/labels.holdout.jsonl --out runs/holdout.benchmark.jsonWhat is reported
- Failure detection over binary labels (pass/fail) where the judge decided: precision, recall, F1 and the confusion counts.
- Abstention: how often the judge returned review or skipped on binary labels, split by whether the label was fail or pass.
- Undecidable labels (review/skipped) scored separately: did the judge also abstain?
- Execution errors, sample counts, label sources, unmatched labels and unlabelled predictions.
- Latency percentiles and token usage from the run. Simulated runs report neither as a measurement.
- Calibration (expected calibration error and bins) only when at least 20 binary-labelled checks carry a single deciding probability; otherwise it says why it was not computed.
Any metric whose denominator is zero is shown as n/a, never 0 or 100%.
Labels
{"caseId":"tune-07","rubricId":"claim-support","expected":"fail","source":"human","reviewer":"ana","note":"price contradicts KB"}The bundled example labels are synthetic and provisional: generated with the cases, not independently human-reviewed. Add human labels with "source": "human" and a reviewer; the benchmark reports counts by source so you can see how much of your evaluation rests on reviewed data.
Tuning versus held-out
Tune thresholds on the tuning split only. Report numbers from the held-out split only, and do not go back and forth.
One recorded live run
On 2026-09-19 both example datasets were judged live by Jev (via Vercel AI Gateway, model id typesafe-ai/jev) with the default thresholds. The held-out split, 10 synthetic cases and 50 checks:
| Metric | Value | Denominator |
|---|---|---|
| Failure detection precision / recall | 100% / 100% | 25 binary labels, all decided (TP 12, FP 0, FN 0, TN 13) |
| Abstention on binary labels | 0 | 25 |
| Agreement on undecidable labels | 92% | 23 of 25 labelled review/skipped |
| Expected calibration error | 0.047 | 22 labelled probabilities |
| Usage | 16,919 input tokens, 10 requests | estimated cost $0.0007 at the documented rate |
| Latency | p50 344 ms, p95 896 ms | 10 requests, one run |
Caveats that matter more than the numbers: the labels are synthetic and two of the disagreements look like label errors; the sets are tiny; it is one run. The tuning split showed six reviews on labelled passes, pointing at a conservative pass threshold for one custom rubric. The full output, the per-case disagreements and the adversarial outcomes are in examples/benchmark.live.md in the repository.
Other judges
To compare Jev with another judge, produce a run JSON in the same report schema from that judge's output (an import script rather than a new provider) and benchmark both against the same held-out labels.