Use a code verifier

A code verifier scores a response or action by executing explicit checks. Use one when success can be established from structured fields, tests, artifacts, events or an expected final state.

Automatic grading does not require a human rating on every response. Human review remains useful for finding bad task data, incomplete rules or verifier bugs.

Define the outcome

Suppose a task asks the model to update a customer's address. A useful verifier checks the resulting address and any relevant constraints. It should not pass the task merely because the response says “Your address is updated.”

Specify:

• The response, tool evidence or final state the verifier receives.
• The expected result and rules for partial success or failure.
• The returned score and any mandatory checks.
• The runtime and dependencies required to execute it.

Keep expected answers and private grading state out of the model's task input. If required evidence is missing or the verifier cannot execute, report that condition rather than silently counting it as a successful check.

Check the verifier before relying on it

1. Prepare known passing and failing examples.
2. Include boundary cases, missing evidence and plausible incorrect outputs.
3. Run the checks and inspect the scores and errors.
4. Fix the verifier or task references if the result contradicts the intended rule.
5. Publish and select the reviewed version for evaluation and training.

A deterministic verifier can still be wrong if it checks an incomplete proxy for success. Human review should help validate its rules, not override each fresh RL reward by hand.

Reuse benchmark scoring

A benchmark Profile may already include its verifier and runtime requirements. Preserve the benchmark's scoring behavior and source version rather than replacing it with a generic LLM judge for convenience. Some tasks combine programmatic checks and judge-based assertions; inspect the complete grader.

τ Retail is one example: native grading can compare the resulting retail database state with the expected outcome. The current setup/import preview does not yet qualify its native verifier for managed training. An imported grader or successful local check is not proof of execution inside a hosted run.

When new tasks arrive

The same code can score new tasks with different expected results. That changes the task data, not necessarily the verifier. If you change the rules or code, validate a new grader version and evaluate both the current and candidate models under it.

Next: Change a grader safely.