Model training and continual learning
Preview
Evaluate tasks, review results, improve a grader, and train a candidate model.
Model training changes model weights. Continual learning repeats that process with new approved tasks while checking that the model retains earlier skills. The grader supplies the scores that guide reinforcement learning.
This guide explains the learning workflow. In the preview, Evaluate on a Model's Tasks page and Train on its Model page open preparation dialogs before any run is reserved. Reviewed labels can also prepare a grader candidate for checking and explicit adoption. Availability depends on the selected task environment and grader; importing a benchmark does not establish that it can run or train.
Choose your path
| Your starting point | Guide |
|---|---|
| You want to label good and bad responses | Review and label tasks |
| Success requires judgment | Improve an LLM judge |
| Success can be checked with rules or tests | Use a code verifier |
| Your scoring criteria or golden examples changed | Change a grader safely |
Tasks, Model and Runs
Tasks is where you inspect examples, responses and scores, and perform human review. The Model page owns training configuration and starting a training run. Runs contains execution progress, costs, results and candidate decisions.
On a Model's Tasks page, select task definitions and choose Evaluate. The dialog distinguishes Hosted model (fresh responses) from Authored fixtures (retained examples), and shows the Taskset revision, population and maximum-cost description before its run button becomes available. Evaluation does not change model weights.
On the Model page, choose Train to review the approved-task count, starting model, grader version, held-out evaluation and maximum spend before Start training reserves work. If human review identified a judge problem, improve and validate the grader first. A label or saved grader candidate does not automatically rewrite the selected grader.
A continual-learning cycle
- Collect tasks from an approved source, imported Taskset or reviewed work.
- Evaluate attempts and inspect the results. Human labeling is optional when the automatic grader already checks the intended outcome reliably.
- Approve useful tasks for training. A failed response can identify a useful task; approval does not mean copying that failed response.
- If the grader misunderstood your criteria, validate and adopt a revised grader before starting the next run.
- Train: the current model generates fresh attempts, the selected grader scores them, and the optimizer uses those rewards to update the model.
- Evaluate the candidate on separate held-out tasks and compare it with the current model under the same scoring rules.
- Review the candidate and explicitly decide whether to use it.
With scheduled learning enabled, runs use the approved configuration and eligible tasks. Task arrival does not change the judge prompt. No eligible batch means no training job. Candidate acceptance remains separate.
What the model learns from
| Component | Receives |
|---|---|
| Model making an attempt | Task instructions, permitted context, tools and environment observations |
| Grader | The response or actions, scoring rules and permitted reference information |
| RL optimizer | Generated attempts and their rewards |
| Evaluation report | Results for comparison and human inspection |
Human labels can help develop and test the grader. They are not automatically inserted into the model's prompt. Private reference answers and judge examples stay with the grader. Saving a corrected answer does not by itself train the model to reproduce it.
Start from a benchmark
A benchmark Profile can supply tasks, environment instructions and native scoring. Inspect the source, selected cases, grader and execution requirements before running it. You should not have to recreate scoring that the benchmark already supplies.
For example, τ Retail tasks can test whether a customer request produced the correct database outcome. Its native verifier can check that outcome without asking you to label every conversation. A conversation that sounds successful can still fail the state check.
The τ Retail setup/import preview does not yet establish a complete hosted evaluation-to-training path. Managed execution of its native verifier remains unqualified. Check readiness rather than treating a visible Profile as a trainable benchmark.
Training methods
Hosted continual learning currently uses GRPO, a reinforcement-learning method. Supervised training from corrected demonstrations and preference optimization are different methods; recording a correction or preference does not select either automatically. A Refiner proposal also does not launch model training.
See Evaluation for fair comparisons and Continuous learning for the separate Harness refinement and recurring-review loop.