Reports and CI gates
Pass-rate denominators, decided coverage, exit codes and how to loosen the policy deliberately.
Rates and their denominators
| Metric | Definition |
|---|---|
| Pass rate (decided) | pass ÷ (pass + fail). Ignores review, skipped and error. |
| Pass rate (applicable) | pass ÷ (all − skipped). Review and error count against it. |
| Decided coverage | (pass + fail) ÷ (all − skipped). How much of the dataset the judge actually decided. |
| Review rate, error rate | review ÷ (all − skipped), error ÷ (all − skipped). |
| Critical failures | Count of failed checks on critical-severity rubrics, always shown separately. |
Usage and latency are recorded per request (one per case), never multiplied by the number of checks in that request.
Exit codes for jeval run --ci
| Code | Meaning |
|---|---|
| 0 | Every gate met and the run is complete. |
| 1 | Quality gate failed: failed required checks or critical failures exceed the limits. |
| 2 | Run incomplete: provider errors, review rate above the limit, coverage below the limit, empty dataset, a required rubric skipped on every case, or a simulated run. |
| 3 | Usage or configuration error (missing key, invalid config, unreadable files). |
The default policy is strict: a review, an error, an empty dataset or a simulated run cannot yield a clean pass. Loosen it deliberately in ci and say why in your repository.
"ci": {
"maxFailures": 0, // failed required checks allowed
"maxCriticalFailures": 0, // always enforced separately
"maxReviewRate": 0.1, // accept up to 10% review on required checks
"minDecidedCoverage": 0.9,
"allowErrors": false,
"allowSkippedRequired": true, // a required rubric may be inapplicable everywhere
"allowEmptyDataset": false,
"allowSimulated": false // never let fixture runs pass CI
}HTML reports
Self-contained, offline, with no JavaScript. All dataset content is escaped, so an assistant output containing HTML or a script tag is shown as text. A simulated run carries a banner; unknown cost is shown as unknown.
GitHub Actions
- run: pnpm install --frozen-lockfile && pnpm build
- run: pnpm --filter jeval-examples cli # offline, simulated: framework smoke test
- if: ${{ secrets.TYPESAFE_API_KEY != '' }}
env: { TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }} }
run: cd evals && jeval run --mode live --ci # real gate, only when a key is configured