Rubrics

A rubric is one narrow check with a stable id and version. Jeval prefers several small checks to one broad quality score: each becomes a single multiple-choice question for Jev, and every question for a case is sent in one request.

{
  "id": "booking-claim",
  "version": "1.0.0",
  "kind": "tool-claim",
  "severity": "critical",
  "description": "A claimed booking is backed by a successful book_appointment tool result.",
  "criterion": "Consider only statements about whether the booking was completed.",
  "params": {
    "toolName": "book_appointment",
    "action": "booking the appointment",
    "successWhen": { "path": "status", "equals": "confirmed" }
  }
}

Fields

• id, version: identity. Changing the criterion should bump the version; compare refuses to align results across versions.
• kind: one of the five kinds below.
• criterion: the question, phrased as one specific condition. Jev is literal: it answers what you wrote.
• severity: critical, major or minor. Critical failures are counted separately in every summary so they cannot hide in an average.
• required (default true): required checks feed CI gates; optional ones are reported only.
• applicability.requires: evidence that must exist (policy, context, toolEvents, messages). Missing evidence yields ?Review, never a pass.
• applicability.onlyWhenMetadata: run only when case metadata matches; otherwise –Skipped with a reason.
• thresholds: per-rubric override of the run thresholds.
• outcomes: descriptions of the acceptable, unacceptable and insufficient outcomes shown to the judge (required for custom).
• params: kind-specific settings.

The five kinds

policy-compliance

Requires policy. Asks whether assistant_response complies with every applicable rule in policy. Outcomes: acceptable, unacceptable, insufficient_context.

claim-support

Requires context. Asks whether the response's factual claims (or params.claim) are stated or entailed by reference_material. Outcomes: supported → pass, contradicted → fail, not_covered → review. Set params.uncoveredIs to fail if an uncovered claim should fail in your application.

tool-claim

Requires a toolEvents array. Asks only whether the response claims the action (params.action) was completed. Code then reads the last event named params.toolName: success → pass, failure → fail, no event → review. If the response makes no claim the check is skipped. params.successWhen adds an exact comparison on the tool output, e.g. status = "confirmed". A missing tool log is never treated as proof of failure.

escalation

Requires policy and toolEvents. Two questions are batched: does the policy require escalation here, and does the response say it escalated. Code then checks for a params.toolName event: required and tool succeeded → pass; required and tool failed → fail; required, no event and no statement → fail; required, no event but claimed → review; not required → skipped; unclear → review.

custom

Any natural-language criterion with explicit outcomes descriptions. Outcomes: acceptable, unacceptable, insufficient_context.

From probabilities to statuses

For a three-outcome question with probabilities p:

• ✕Fail when p(unacceptable) ≥ thresholds.fail;
• otherwise ✓Pass when p(acceptable) ≥ thresholds.pass;
• otherwise ?Review, including whenever the insufficient-context outcome leads.

The configuration is rejected unless pass + fail > 1, which guarantees the two rules can never fire together. The defaults (pass 0.75, fail 0.5) are starting points: tune them with jeval benchmark on a tuning split and confirm on held-out data. Decisions use the full probability distribution. Jev's confidence statistic is preserved in results but is not used for decisions and is never displayed as an accuracy.

Statuses

• ✓Pass / ✕Fail: decided.
• ?Review: undecided; needs a person. Includes missing evidence.
• –Skipped: explicitly inapplicable, with a reason.
• !Error: the judge request failed or returned a malformed answer. Execution errors are never counted as evaluated failures.

What is shown for each check

The criterion exactly as sent, the selected outcome, the full probabilities, the provider's confidence statistic, the rule applied by code (if any), and which evidence fields and ids were included. Jeval does not invent free-text rationales or quotations: Jev returns decisions, not explanations.