Turn Repeated Work into a Taskset and Evaluation

When a task keeps recurring, a good next step is not automatically training. First turn a representative, approved example into a Taskset: an immutable, reviewable collection of tasks and expected evidence.

1. Start with a real, approved example

Choose completed work that represents the behavior you care about. Review it, remove information you are not allowed to use, and preserve the provenance. Collecting an example does not authorize it for training; that decision remains separate.

2. Make the success condition explicit

For each task, record the input, expected outcome or grading rule, provenance, split, and needed attachments. Add a grader that can distinguish a useful result from a superficial one. A reliable evaluation is more valuable than a large pile of vague examples.

3. Freeze an evaluation split

Keep held-out evaluation tasks separate from examples used to refine the behavior. Run the evaluation against the baseline first. That gives later changes—Harness updates, Agent changes, or model candidates—a point of comparison.

4. Decide what layer should change

A Taskset supports evaluation first. The evidence may show that a small Skill or Harness adjustment is enough. If behavior truly needs model training, the next step requires approved data, a candidate, budget, frozen evaluation, and an activation decision.

The loop is reviewed work → versioned Taskset → measured evaluation → explicit decision. Read the Tasksets guide before training.