Skip to content
OpenPond
Join community
Evals/Review and label tasks

Review and label tasks

Apply a grader's criteria to responses and decide which tasks are useful for learning.

A task describes what the model should attempt. A response records one attempt. A human label records your judgment of that response against specific criteria. The same task can have several responses with different scores.

The preview's response review loads the exact attached grader criteria and scale. It retains that grader/rubric identity with ratings while keeping corrections and training eligibility as separate decisions.

Four decisions with different effects

DecisionWhat it recordsWhat it does not authorize
Score a responseYour judgment against the exact grader criteria.Training use or a weight update.
Admit an examplePermission for that exact source/task evidence to enter training.Copying an incorrect answer or accepting a candidate.
Correct an exceptionA proposed better answer with retained original evidence.Bypassing independent verification.
Accept a candidateA decision on a trained version and its comparison evidence.Unrelated serving authority.

Ordinary Model evidence is reviewed in the Model's Tasks queue using Needs review. Its scheduler requires explicit human admission, the matching source and task definition, the training split and unused evidence. Creating a Dataset or enabling a schedule does not create a review assignment for every row.

A Model owner can use Submit Dataset tasks for review from the Dataset's Continual learning tab to submit selected tasks from the exact published training split. The resulting receipt links to the same Model Tasks queue. This action submits evidence only; it creates no training admission or global Inbox assignment. See the Dataset handoff.

Conversation policies can automatically grade, verify and admit qualified examples under their saved rules. Their unresolved exceptions and candidate decisions appear in Inbox. This is a different admission path from ordinary Model evidence.

Review a response

  1. Open a response in an Experiment or an assigned item in Inbox, then open its human review controls.
  2. Read the attached grader's rubric or checks and its scoring scale.
  3. Compare the automatic result with the evidence. Decide whether the model failed, the grader was wrong, or the task lacks enough information.
  4. Record a label and explanation against the relevant criterion. Mark cases you cannot assess rather than assigning an arbitrary score.
  5. Under Training use, decide separately whether the task is suitable for training, then choose Save review.

For a support judge, a criterion might be “Checks refund eligibility before promising a refund.” A failed label should explain which promise or missing check supports the judgment. That gives a judge improvement a concrete target.

Distinguish the review outcomes

FindingNext step
Model failed; grader correctly detected itConsider the task for training with the current grader
Grader gave the wrong scoreUse the label to check/refine the grader before relying on that score
Task input or expected result is wrongCorrect the task, then evaluate the revised example
You can supply a better answerSave a correction/reference and review its intended use
Evidence is insufficient or reviewers disagreeKeep uncertainty visible and resolve it before treating the label as authoritative

Approving a task for RL means allowing fresh attempts at it. It does not approve copying an incorrect response. A suggested answer is also different from a human score: one shows what to say; the other judges a particular attempt.

Resolve conversation exceptions

Open the assigned exception and inspect the original answer, grader evidence, missing context and any proposed correction. Exclude unsuitable evidence or save a correction for independent verification. The correction counts toward a training batch only after its required checks pass.

A policy set to Pause this cycle for Inbox exceptions waits for unresolved decisions. With Continue with independently verified examples, the qualified remainder may proceed only if it still meets the minimum batch and evaluation requirements. Neither choice makes a failed or uncertain example eligible.

Candidate approval is separate: inspect the baseline/candidate comparison, regression checks and retained cost before deciding. A failed or inconclusive comparison keeps the current version. See Models for version and serving decisions.

Use labels to improve scoring

Selected labels can become examples in the judge prompt or test cases for the judge. Record which role each example serves. A test of the judge hides the human score from it and compares the returned score with your judgment.

Start with diverse successes, failures and ambiguous cases. Ten or fifteen examples can reveal useful disagreements, but the count alone does not prove that the judge is reliable.

From a retained Model task rating, choose Use label to check Reward, then Check / refine for the selected grader. Select reviewed labels as development examples and continue into the editor. Add separate independent validation fixtures and check both the published grader and candidate on the unchanged fixtures. Saving a label or candidate alone does not change the active grader or train the model.

Next: Improve an LLM judge or use a code verifier.

View as Markdown