Harness Refiner: Testing a Persistent Learning Loop
Harness Refiner reviews completed work and changes the behavior around a model only when the evidence supports a reusable correction. That behavior lives in an immutable Harness: instructions, Skills, Agents, approved memory, and tool capabilities.
This is not model-weight training. Refiner does not compute gradients or update parameters. It produces a new, inspectable Harness release, which can be used by the next task and rolled back later.
We now test that system in two separate layers:
- a controlled qualification verifies that the complete mechanism can abstain, route, change, roll back, transfer, and persist review state; and
- a natural 20-task systems trace exercises the same sequence on distinct held-out work.
Keeping those layers separate matters. A system can have working control paths without finding a justified change in one natural run.
Qualification: 6 of 6 scenarios passed
The qualification uses six versioned scenarios with deterministic expected outcomes.
| Scenario | Required outcome | Result |
|---|---|---|
| Clean result | Abstain without changing the Harness | Passed |
| Ambiguous evidence | Retain the current release | Passed |
| Wrong system owner | Route outside the Harness | Passed |
| Supported correction | Apply one bounded change with rollback | Passed |
| Distinct follow-up | Improve verified reward with the new release | Passed |
| Recurring cross-run issue | Persist, deduplicate, and continue review | Passed |
The suite used the released local Work, Refiner, validation, immutable release, rollback, cross-run review, and scheduler paths. It proves those paths can produce the intended decisions under controlled evidence.
Benchmark: twenty tasks, forty attempts
The natural Taskset contains ten adaptation tasks and ten fact-distinct held-out tasks. Each task used an explicit deterministic output contract. An uncalibrated model judge was not part of the primary reward, so the run is not a validated semantic-quality benchmark.
The controlled sequence is:
baseline Harness → 10 adaptation tasks
baseline Harness → 10 held-out tasks
task 1 → Refiner → next Harness release
task 2 → Refiner → next Harness release
...
task 10 → Refiner → freeze final Harness
final Harness → 10 held-out tasksThe run fixed the OpenPond Chat model, DeepSeek V4 Flash, low reasoning effort, local Work runtime, tools, sampling configuration, seed, Taskset release, and spend ceiling. Every foreground attempt, Refiner invocation, reward, artifact, and Harness transition was recorded in content-addressed evidence.
What happened
The run completed all forty foreground attempts and all twenty canonical task
pairs. Refiner reviewed each adaptation result and returned no_action ten
times. The initial and final Harness hashes are identical.
| Measurement | Baseline | Repeated candidate |
|---|---|---|
| Adaptation contracts passed | 5/10 | 3/10 |
| Held-out contracts passed | 3/10 | 4/10 |
| Foreground tokens across all tasks | 11,063,871 | 10,728,586 |
One Refiner invocation failed and recovered through the admitted retry without repeating the foreground task. Total observed spend was $0.888575, including 222,498 Refiner tokens.
The result is a systems trace, not evidence of learning or broad task quality. There was no accepted Harness change, so the pass-count and token differences are repeat-run variation rather than evidence of learning. The useful result is operational: the complete loop ran, preserved its immutable lineage, recovered a failed review, and declined to write an unsupported persistent rule.
The follow-up is a new, separately versioned Taskset: policy-visible criteria, deterministic artifact and format checks, calibrated semantic grading, criterion-level diagnoses, adversarial grader tests, and an A/A variance control. It will be evaluated and published only after those gates pass.
Immediate and cross-run review
The product has two review lanes with the same correction policy.
- Immediate review examines one settled task and may make a narrowly supported low-risk update for the next task.
- Cross-run review groups authorized evidence from separate tasks, persists a candidate when recurrence is not yet strong enough, and revisits it through activity or scheduled review.
Neither lane changes the Harness merely because a task failed. The system must identify a reusable root behavior, establish that the Harness owns it, fit the change inside an advertised capability, and pass critique and validation.
Historical studies
An earlier 50-task observation completed every review, routed five problems to other system layers, and proposed no Harness change. An August 11 paired run accepted one short-message rule and found a narrow transfer to related held-out messages, while the complete held-out efficiency result moved in the opposite direction.
Those studies remain available as historical trajectories. The current model card leads with the six-scenario qualification and the sealed 20-task result, including every task pair and its content hash.
The public model card shows the results. The complete Taskset, protocol, deterministic verifier, and content-addressed aggregate are in the OpenPond benchmark directory.