Evaluation
Compare a baseline and candidate against a frozen Taskset.
Evaluation runs a defined model or behavior against a frozen Taskset and records the result, grader output, version identity, and review context.
Evaluate the model or check the grader
These answer different questions:
| Check | Question | Method |
|---|---|---|
| Model evaluation | Does the model perform the task well? | Generate responses on held-out tasks and score them |
| Grader validation | Does the grader apply the intended criteria? | Score existing human-labeled responses and compare judgments |
Checking an existing response does not require generating it again. Neither operation updates model weights. During RL, fresh training attempts are scored by the grader and those rewards guide the optimizer; that is the training step.
Tasks is the place to inspect these results and review responses. The planned Evaluate dialog makes fresh execution versus checking retained responses explicit before starting and shows the cost limit. Baseline and candidate evaluation also remain part of the training workflow.
Before you compare
Use the same Taskset version, grading rules, and meaningful resource limits for the baseline and candidate. Inspect representative failures as well as the aggregate score; a higher score cannot justify a regression in a critical task.
If you change the grader, reevaluate the current model under the new grader before comparing it with a candidate. Keep related task families out of both training and held-out evaluation. Keep examples shown in the judge prompt separate from independent checks of the judge itself.
After an evaluation
Choose one of three outcomes: keep the current release, investigate a bounded Harness change, or approve a candidate for a further training/activation gate. Evaluation does not silently deploy a candidate.