TestedliveEval Harnesses
Judge Calibrator
Score the same outputs with an uncalibrated judge and a rubric-calibrated one, against a human panel.
MLMichelle
Eval labelJudge calibration
- Cases
- 6
- Passing
- 5
- Failing
- 1
- Built
- 31 Aug 2026
- Quote until
- Verified until 29 Nov 2026
Try it
The working version.
Mocked for this demo, with the numbers taken from the run that produced it.
Sample 1 of 7
Summarise the customer's current excess positionThe customer's excess is waived under the hardship provision.
Agreement so far · 0 scored
Uncalibrated0%
Rubric-calibrated0%
Exact agreement with the panel. The published κ of 0.71 is the chance-corrected version over 180 samples.
The rubric
- 5Complete and correctAnswers the question, every factual claim is supported by the source, states what it does not know.
- 4Correct, thinNothing wrong, but omits a material qualification a reviewer would expect.
- 3Partially correctThe main claim holds; at least one supporting detail is wrong or unsupported.
- 2MisleadingReads as authoritative and would lead a reader to the wrong action.
- 1WrongThe central claim is false.