Graders
Define repeatable criteria for measuring task outcomes.
A grader defines how a task outcome is scored. For automated evaluation and RL, the two primary choices are an LLM judge and a code verifier. They can be combined when different criteria need different checks.
| Grader | Use when | Guide |
|---|---|---|
| LLM judge | The outcome requires judgment against a rubric | Improve an LLM judge |
| Code verifier | Rules, tests or expected state can establish correctness | Use a code verifier |
Human labeling applies the scoring criteria to a recorded response. Those labels can help validate or improve the automatic grader. A human label on an old response is not an executable reward for a fresh RL attempt.
Design a useful grader
- Measure the outcome that matters, not a proxy that is easy to game.
- Keep inputs and scoring rules explicit and versioned with the Taskset.
- Record inconclusive or unavailable evidence rather than treating it as a pass.
- Review aggregate failures before changing a Harness or training a model.
Use a small number of understandable criteria. A score is evidence for a decision, not a replacement for the authority to make that decision.
Keep criteria and labels connected
Use the same criterion names, score meanings and rubric when humans and the judge assess an example. Retain the grader version with the label. A generic rating is not automatically evidence that the selected judge is correct.
An LLM judge can contain selected golden examples; it can also have a separate dataset of labeled cases used to test it. Adding test cases does not change its prompt. Changing prompt examples or scoring rules creates a new grader version.
Next: Review and label tasks, Evaluation, or Change a grader safely.