OpenPond model card · Harness Refiner v1
Across the same twenty tasks, one Harness refinement reduced foreground tokens on thirteen tasks and cut total usage by 41.38%, without changing the model, runtime, or tools.
Token result
13 / 20
tasks used fewer tokens
7 tasks used more tokens
All 20 tasks
−41.38%
5,599,785 → 3,282,811
Adaptation
5/10 lower
−28.35% in aggregate
Held out
8/10 lower
−57.87% in aggregate
Per-task token direction
Each square is one paired task under the baseline and refined Harness.
Adaptation
5 lower·5 higher
Held out
8 lower·2 higher
Foreground tokens
Provider-reported totals across the same paired tasks.
All 20 tasks
−2,316,974 raw change
Adaptation
−887,153 raw change
Held out
−1,429,821 raw change
What the run showed
Refiner cut total foreground usage by 41.38%, but the reduction was not consistent across the suite: thirteen tasks used fewer tokens and seven used more. This run did not reach the intended 20/20 per-task efficiency target.
The refinement loop
Refiner waits until work settles, learns once from the adaptation cohort, then proves the release on replay and unseen tasks.
- 01
Work
Run the adaptation tasks with one pinned Harness release and capture the result, tools, artifacts, grades, and usage.
- 02
Refine once
Review the completed cohort together and propose one small, reusable Harness change—or no change at all.
- 03
Validate
Independently critique the proposal, enforce bounded schemas and permissions, then create an immutable release.
- 04
Replay
Run adaptation and held-out tasks again with the same model, runtime, tools, seed, and sampling policy.
Evidence in, bounded update out
Refiner sees the requests, visible answers, tool outcomes, recoveries, artifacts, grades, usage, and the pinned Harness. The model decides what is reusable; deterministic validation owns schemas, permissions, safe paths, and release identity.
A controlled comparison
- Same model revision and high reasoning effort
- Same local Work runtime and ten tools
- Same seed, temperature, top-p, and output limit
- Frozen research evidence replayed across each pair
Inspect the complete run
The public Taskset includes all twenty tasks, fixtures, graders, and the deterministic runner. The technical note covers the evidence contract, paired protocol, and measured result.