Harness Refiner: A Controlled Post-Work Behavior Loop

A strong final answer can hide repeated, avoidable work. OpenPond's Harness makes that behavior inspectable and releasable.

After a task settles, Harness Refiner can inspect concrete evidence, decide whether anything reusable was learned, and propose the smallest justified change. A new task can use the improved Harness without replaying every old transcript.

Harness Refiner runs are ordinary Model Runs, with their configuration, quality gate, token accounting, Harness lineage, and result history together.

The Harness is the behavior around the model

The model is only one layer of an agent. Its Harness also carries:

instructions and approved memory;
Skills and specialized Agents;
tool capabilities and operating policy;
immutable release metadata.

Every task pins one Harness release before it starts. Refinement happens after the result settles, so it cannot rewrite the task that produced the evidence. An accepted change creates a new immutable release. The old result remains reproducible and the update can be inspected or rolled back.

settled task
  → bounded evidence
  → model-authored decision
  → independent critique
  → deterministic validation
  → new Harness release or no action

Why post-turn review works

During a task, a strong model can often recover from an error. Recovery is useful, but it can also hide a repeated tax: the agent may make the same wrong first attempt on every future task, spend several tool calls repairing it, and still return a correct result.

Post-turn review can see both sides. It receives the request, the visible answer, artifacts, bounded tool events, failures, recoveries, validation, and the exact Harness sources loaded for the turn. It can distinguish a harmless one-off from a general behavior that should change.

Refiner is deliberately model-driven. The model decides whether an event matters, whether recovery fixed the root cause, whether differently worded events are related, and which Harness layer owns the improvement. Code handles the parts that should not be interpretive: authorization, evidence bounds, schemas, safe paths, atomic application, release identity, and rollback.

Work, refine once, validate, and replay

Harness refinement is a post-work release process. The benchmark first runs a fixed adaptation cohort with the baseline Harness and records bounded evidence. Refiner reviews that cohort once and either proposes a reusable source change or takes no action. An independent critique and deterministic validation then decide whether the proposal can become a new immutable Harness release.

The candidate release replays both the adaptation cohort and a distinct held-out cohort. This makes two questions visible: did the proposed behavior repair the failures that motivated it, and did it generalize without reducing quality? A small up-front refinement cost is useful only when later foreground work saves enough to repay it.

A public, provider-neutral engine

The contracts and model-driven policy live in the open-source @openpond/harness package. Applications provide the model, persistence, authorization, and scheduling adapters:

import { authorLocalHarnessRefinementWithModel } from "@openpond/harness";

const decision = await authorLocalHarnessRefinementWithModel({
  evidence: completedTurnEvidence,
  stream: modelStream,
  signal,
});

The model may return no_action, propose a prompt, memory, Skill, or Agent edit, or route the evidence to runtime, product, Evaluation, or model-training review. A second model critique checks proposed edits for task-specific facts, overfitting, and unsupported instructions before deterministic validation.

This keeps the learning engine portable. Desktop can run it with local files and SQLite. A hosted product can use the same package with its own durable storage, authorization, and compute adapters.

Measuring generalization, not memorization

OpenPond ships one canonical Harness Refiner benchmark as a public, content-addressed Taskset. It contains twenty ordinary real-world requests: ten adaptation cases and ten distinct held-out cases from the same behavioral families.

baseline Harness → 10 held-out tasks
baseline Harness → 10 adaptation tasks → Refiner
candidate Harness → the same 10 adaptation tasks
candidate Harness → the same 10 held-out tasks

The comparison fixes the model, effort, local Desktop Work runtime, tools, Taskset release, sampling policy, and run configuration. Search and fetch results are captured in a content-addressed evidence snapshot during baseline work and replayed for the matching candidate task, so live-source drift cannot be mistaken for a Harness effect.

The primary question is whether each of the same twenty tasks uses fewer foreground provider tokens after refinement. Task quality is tracked separately as a safety check so a shorter failed answer cannot look like an efficiency win.

Artifact tasks test whether an agent can create and verify useful files. Research tasks test source selection, stopping behavior, calibration, and linked reporting. The prompts never mention Harness internals, tool names, or workspace paths.

What the measured run showed

The revision-3 run completed the forty-attempt protocol with one baseline and one Refiner-produced Harness. Refiner added one general instruction: when someone asks for a message, email, report, or summary in a plain turn, return the complete content inline by default instead of replacing it with an artifact.

Measurement	Baseline	Refined	Aggregate change
All twenty foreground tokens	5,599,785	3,282,811	−41.38%
Adaptation foreground tokens	3,128,871	2,241,718	−28.35%
Held-out foreground tokens	2,470,914	1,041,093	−57.87%
Tasks using fewer tokens	—	13/20	Target 20/20
Adaptation tasks lower	—	5/10
Held-out tasks lower	—	8/10

The aggregate reduction was substantial but uneven. Five adaptation tasks and eight held-out tasks used fewer tokens; the other seven tasks used more. The run therefore reached 13/20 on its intended per-task efficiency target, not 20/20.

Quality was secondary context. It rose from 13/20 to 19/20 overall, although one travel report grew beyond the 4,096-token output ceiling and ended mid-sentence.

From Harness updates to continuous model improvement

Harness refinement fixes the cheapest correct layer first. Repeated evidence that survives held-out Evaluation can later support a Taskset-backed model improvement recommendation. That recommendation still has to pass explicit quality, privacy, budget, and activation gates before SFT, preference learning, or reinforcement learning is appropriate.

The principle is simple: let the model judge meaning, keep the evidence and authority bounded, make every behavior change reversible, and measure the next task rather than trusting the explanation.

The complete benchmark Taskset, fixtures, graders, and runner are available in the OpenPond benchmark directory. The implementation is in OpenPond PR #104, and the public model card shows the per-task and aggregate token comparison.