Benchmarking the judge
Score predicted outcomes against labels; tuning versus held-out splits; calibration.
Before trusting a judge on your data, measure it. jeval benchmark compares a run's predicted statuses with labels you supply separately; labels are never sent to the judge.
jeval run --mode live --dataset data/dataset.tuning.jsonl -o runs/tuning.json
jeval benchmark runs/tuning.json --labels data/labels.tuning.jsonl
# adjust thresholds in jeval.config.json, re-run on tuning only, then:
jeval run --mode live --dataset data/dataset.holdout.jsonl -o runs/holdout.json
jeval benchmark runs/holdout.json --labels data/labels.holdout.jsonl --out runs/holdout.benchmark.jsonWhat is reported
- Failure detection over binary labels (pass/fail) where the judge decided: precision, recall, F1 and the confusion counts.
- Abstention: how often the judge returned review or skipped on binary labels, split by whether the label was fail or pass.
- Undecidable labels (review/skipped) scored separately: did the judge also abstain?
- Execution errors, sample counts, label sources, unmatched labels and unlabelled predictions.
- Latency percentiles and token usage from the run. Simulated runs report neither as a measurement.
- Calibration (expected calibration error and bins) only when at least 20 binary-labelled checks carry a single deciding probability; otherwise it says why it was not computed.
Any metric whose denominator is zero is shown as n/a, never 0 or 100%.
The automation curve
Every decided check carries the probability mass behind each side. The benchmark sweeps a decision threshold from 0.5 to 0.95 and reports, for each, what share of checks would be decided automatically and how accurate those decisions were against the labels, with failure precision and recall. --target-accuracy 0.9 picks the lowest threshold that reaches 90% on at least 20 decided checks and tells you what to put in thresholds. This is the honest answer to “is the judge accurate?”: it depends how much you let it decide, and the rest goes to review.
Labels
{"caseId":"tune-07","rubricId":"claim-support","expected":"fail","source":"human","reviewer":"ana","note":"price contradicts KB"}The bundled example labels are synthetic and provisional: generated with the cases, not independently human-reviewed. Add human labels with "source": "human" and a reviewer; the benchmark reports counts by source so you can see how much of your evaluation rests on reviewed data.
Tuning versus held-out
Tune thresholds on the tuning split only. Report numbers from the held-out split only, and do not go back and forth.
Recorded runs on public, labelled data
On 2026-09-20 jeval judged 300 responses from RAGTruth's test split (human span-level hallucination labels; 50 hallucinated and 50 clean responses each for question answering, summarisation and data-to-text) with the claim-support rubric, live via Vercel AI Gateway. Thresholds were fixed on a separate tuning split first.
| Metric | Value | Denominator |
|---|---|---|
| Failure-detection precision / recall | 82.8% / 78.3% | 191 decided cases (TP 72, FP 15, FN 20, TN 84) |
| Abstention (review) | 36.3% | 109 of 300; 58 on hallucinated, 51 on clean |
| Execution errors | 0 | 300 requests |
| Expected calibration error | 0.267 | 300 labelled probabilities |
| Usage / latency | 398,501 input tokens · p50 353 ms | estimated $0.017 at the documented rate |
The same run as an automation curve (decide only when the leading side reaches the threshold; the rest is review):
| Threshold | Decided | Accuracy on decided | Fail precision / recall |
|---|---|---|---|
| 0.50 | 100% | 71.7% | 72.1% / 70.7% |
| 0.60 | 88% | 75.8% | 78.3% / 67.3% |
| 0.70 | 77% | 81.0% | 82.1% / 64.0% |
| 0.75 | 71% | 82.1% | 83.3% / 60.0% |
| 0.80 | 63% | 81.6% | 82.7% / 54.0% |
| 0.90 | 43% | 89.2% | 89.0% / 43.3% |
| 0.95 | 30% | 95.5% | 94.4% / 34.0% |
The same 300 cases judged sentence by sentence with the current default rubric wording (one question per sentence in the same request, about 2.5× the tokens): case-level precision 88.3% and recall 98.1% (2 hallucinated responses missed instead of 20), abstaining on 32%, and the report names the failing sentence. Against 2,413 human sentence labels: precision 76.7%, recall 89.5%, well-calibrated probabilities (ECE 0.08), so a 0.60 threshold decides 96% of sentences at 90% accuracy. The wording and thresholds were chosen on a separate tuning split first; the deterministic grounded-values check on the same run flagged 20 responses at 70% precision.
Read it as: where the judge decides, it is right about four times in five in both directions, and it abstains on a third of cases. It misses subtle conflicts in long summaries (16 of 50 hallucinated summaries judged supported) and over-flags paraphrase in data-to-text (12 of 50 clean descriptions judged unsupported). Calibration is poor, which is why thresholds rather than raw probabilities drive decisions. One run, one day, one rubric wording; the per-task tables, the tuning decision and the HaluEval and τ-bench runs are inexamples/public-benchmarks/RESULTS.md in the repository, with a script that reproduces everything for about $0.06.
Other judges
To compare Jev with another judge, produce a run JSON in the same report schema from that judge's output (an import script rather than a new provider) and benchmark both against the same held-out labels.