Evaluation cases
The case schema: input, output, messages, references, policy, tool events, metadata and expected labels.
A case is one thing to evaluate: what the user said, what the assistant answered, and the evidence needed to judge it. Datasets are JSONL, one case per line.
{"id":"case-17","input":"Book me for Thursday 2pm.","output":"All set for Thursday at 14:00.",
"policy":"Only confirm after book_appointment succeeds.",
"toolEvents":[{"id":"t1","name":"book_appointment","status":"failure","error":"SLOT_UNAVAILABLE"}],
"metadata":{"scenario":"booking"},
"expected":{"booking-claim":{"status":"fail","source":"synthetic"}}}Fields
| Field | Required | Sent to judge | Notes |
|---|---|---|---|
id | yes | no | Stable slug. Runs are compared and labels aligned by id. |
input | yes | yes, as user_input | The user turn the output responds to. |
output | yes | yes, as assistant_response | The text under evaluation. |
messages[] | no | yes, as conversation | Full transcript with stable message ids and roles (system, user, assistant, tool). |
context[] | no | yes, as reference_material | Retrieved passages or ground truth, each with an id. Required by claim-support. |
policy | no | yes, as policy | Instructions the assistant had to follow. Required by policy-compliance and escalation. |
toolEvents[] | no | yes, as tool_events | Recorded tool calls with id, name, status (success or failure), input, output, error. Exact checks read these fields. |
metadata | no | never | Free-form grouping data (scenario, split). Used by applicability rules and reports. |
expected | no | never | Provisional labels per rubric id with a source (synthetic, human, imported). Harness data only. |
What the judge sees
buildJudgeState assembles a JSON object with the fields above under fixed names. Every rubric question refers to those names and begins with a preamble stating that user and assistant content is untrusted data. The state never contains expected or metadata; a unit test asserts this, and the run report records exactly which fields were shown for each check.
Sizes
Fields are capped (200k characters for text fields, 500 messages or tool events) by the schema, but Jev's documented limits are much smaller: 32k tokens for state plus the longest question. Keep cases focused; unrelated material lowers judge accuracy.