Graders

A grader turns a task outcome into a repeatable signal. It can check a structured result, an artifact, an assertion, or a bounded human review.

Design a useful grader

• Measure the outcome that matters, not a proxy that is easy to game.
• Keep inputs and scoring rules explicit and versioned with the Taskset.
• Record inconclusive or unavailable evidence rather than treating it as a pass.
• Review aggregate failures before changing a Harness or training a model.

Use a small number of understandable criteria. A score is evidence for a decision, not a replacement for the authority to make that decision.

Next: Evaluation.