Change a grader safely
Preview
Improve scoring between learning runs while preserving fair comparisons and history.
A grader can evolve as you clarify the desired behavior. Keep its selected version fixed during a run; review changes before using them in later runs.
The preview retains versioned graders and run configurations. Publishing a grader does not automatically move existing task or learning settings to it. On a Model-scoped Grader page, Use revision for Model opens the coordinated adoption review described below.
Decide whether the grader changed
| Update | Grader change? |
|---|---|
| More tasks for the same rules to score | No |
| A different reference answer for a different task | No; it is task data |
| More labeled cases used only to check the judge | No; the check dataset changes |
| Golden examples inserted into the judge prompt | Yes |
| Rubric, score scale, judge model, verifier code or score combination changes | Yes |
Changing an existing task's expected answer still creates a revised task and can require reevaluation. “The grader did not change” does not mean old task results remain valid under different reference data.
Adopt an improvement
- Create a candidate grader version and inspect its rubric, examples or code diff.
- Test it against reviewed examples, including independent checks and earlier failures.
- Resolve important disagreements and review the change before selecting it. Choose Keep current grader to make no configuration change, or Adopt candidate to apply the reviewed revision consistently.
- Make sure the next training batch and its evaluation use compatible selected scoring rules. Recheck score-dependent approvals affected by the change.
- Evaluate both the current model and the candidate under the newly selected grader and the same held-out task set.
Previous runs keep their original scores and grader versions. An in-progress run continues with its captured grader. A draft edit does not alter it.
Avoid misleading improvement
Suppose the current model scored 60% under grader v2 and a candidate scores 80% under v3. That does not establish a 20-point improvement: v3 might simply be more lenient. Rescore the current model under v3 before comparing it with the candidate, and inspect important individual failures.
Likewise, do not use the examples you placed in the judge prompt as independent evidence of judge accuracy. Keep both judge-check examples and model-evaluation tasks separate from the material used to improve them.
Scheduled learning
Schedules use the last approved learning configuration. New tasks and new human labels do not silently change the active grader. A proposed improvement waits for review and adoption before it affects later runs.
Updating a grader does not automatically authorize replaying all previously used training examples. It also does not activate a new model. Review the training result and explicitly accept or reject the candidate.
Return to Model training and continual learning.