How the OpenPond Harness Refiner Works

The OpenPond Harness Refiner reviews completed Agent work and decides whether the evidence supports a durable improvement. It can leave the Agent alone, recommend evaluation coverage or work in another system, or propose one exact change to memory, instructions, a Skill, or an Agent definition.

@openpond/harness packages that review process so it can run inside OpenPond or another Agent system. The integration supplies the completed run and the editable sources. The Refiner turns that context into a bounded, reviewable decision.

host Agent/runtime
  -> completed turn, trace, artifacts, and current sources
  -> bounded Refiner evidence packet
  -> model-driven review and deterministic admission
  -> no action, external route, or exact source proposal
  -> host validation, application, versioning, and rollback

The result is a proposal with evidence and a precise target, ready for the system's normal validation and release process.

The review starts after the foreground task is complete. It is not part of the Agent's response loop, so it can inspect the final answer, artifacts, tool failures, recoveries, and evaluation results together. The caller decides when to run it: after every turn, after a failure, from a queue, or when a group of similar incidents is ready for review.

Inputs

The Refiner begins with context from the system that ran the task:

the Agent loop and model-provider credentials;
conversation history, tool traces, outputs, and artifacts;
memory, instructions, Skills, and Agent definitions;
authorization, privacy filtering, and evidence retention;
source storage, validation, release activation, and rollback.

The source format is flexible. A support bot might keep its prompt in a database. A coding Agent might load Markdown Skills from a repository. A workflow system might represent roles and procedures as JSON documents. The integration maps the editable parts into a small catalog:

type RefinerSource = {
  path: string;
  kind: "memory" | "instruction" | "skill" | "agent";
  content: string;
  loaded: boolean;
};

loaded matters. It distinguishes a rule that was actually available to the reviewed run from a file that exists now but could not have influenced that run. The evidence contract also separates the admitted release from the current release so a review cannot rewrite history after sources change.

That gives the Refiner two views of the Harness. The admitted source records what the Agent saw during the run. The current source records what can be edited now. If another proposal changed a Skill in between, the review still has the original source for attribution and the current source for conflict checking.

OpenPond Agent SDK projects map naturally to these source layers. Other Agent systems can map the same operations to their own storage model.

Review

Evidence

The Refiner uses a bounded evidence schema. The packet contains the current turn, a short prior conversation window, timeline events, artifacts, diagnostics, execution counters, matching prior incidents, admitted source, current editable source, and the exact evidence IDs a decision may cite.

Every route or proposal declares an evidence basis. A single_deterministic basis is allowed when one incident exposes a concrete failure mechanism and a reusable prevention rule. A recurrent_independent basis requires at least two independent incidents. Similar wording, the same tool name, or the same artifact type does not count as independent evidence. Material counterevidence is recorded alongside the supporting IDs.

Policy

The policy asks the model to separate a reusable Harness problem from an ordinary successful turn, a conversation-specific fact, or a defect owned by another system layer.

The completed result matters, but success is not the only signal. An Agent may eventually produce the right file after taking an avoidable path through invalid tool calls, retries, or a rule it failed to follow on the first attempt. The review looks for the smallest change that would prevent that path next time. It does not copy task content, customer data, benchmark wording, or transient file paths into the Harness.

Model adapter

The model provider plugs in through a small streaming adapter. The response is parsed into a structured decision:

import {
  authorLocalHarnessRefinementWithModel,
  type LocalHarnessRefinerModelStream,
} from "@openpond/harness/refiner";

const stream: LocalHarnessRefinerModelStream = async function* ({
  messages,
  signal,
}) {
  const result = await model.generate({ messages, signal });
  yield { text: result.text };
};

const decision = await authorLocalHarnessRefinementWithModel({
  evidence,
  stream,
  signal: new AbortController().signal,
});

The adapter receives the complete message list and an abort signal. The package handles the timeout, collects the streamed text, parses the JSON, and makes one bounded repair request if the first response is malformed. Provider credentials and model selection stay in the integration.

Critique

A proposed mutation receives a mandatory independent model critique. A no_action result also receives a challenge when the evidence contains a recovery or tool failure that may reveal a reusable prevention rule.

The critique sees the original evidence and the first decision. It checks the failure mechanism, ownership, target layer, evidence IDs, exact edit, and expected effect. It can keep the proposal, narrow it, change it to an external route, or return no_action. A clean no_action on an ordinary successful run does not need this second pass.

Admission

Deterministic checks run on the final result. A proposal cannot cite evidence IDs that the host did not admit, target a capability the host did not advertise, or escape the supported source boundaries. Invalid proposals fail closed to no_action.

The operation shape is checked too. create requires new content, update requires one exact find and replace, and delete cannot carry replacement content. These checks do not decide whether an edit is wise; they make sure the model returned an operation the integration can inspect and apply safely.

Three outcomes

The Refiner has three top-level outcomes:

Outcome	What it says
`no_action`	The evidence does not justify durable intervention.
`route`	The issue belongs to runtime, product, Taskset, or training ownership.
`propose`	One exact memory, prompt, Skill, or Agent source operation is justified.

A failed grade may show that a Taskset needs coverage or that its grader is wrong. A tool outage may belong to runtime or product infrastructure. A behavior may need model training rather than another prompt rule. These become routes instead of persistent Agent text.

A taskset route is a recommendation to enter an evaluation-authoring workflow. The Harness package does not synthesize or run the Taskset. Portable Taskset, grader, run, and evaluation contracts live in @openpond/evals, while the host still supplies execution and persistence.

The other external routes work the same way. runtime covers execution and tooling behavior, product covers application behavior, and training marks a problem better addressed in model weights. A route records ownership and an expected outcome; it is not a generic failure label.

When the outcome is propose, it remains intentionally small. An update looks roughly like this:

{
  decision: "propose",
  route: "skill",
  operation: "update",
  target: "skills/pdf-editing.md",
  find: "Use the default PDF writer.",
  replace: "Use the PDF writer only for new documents; preserve existing PDFs with the supported incremental editor.",
  createContent: null,
  evidenceBasis: {
    kind: "single_deterministic",
    supportingEvidenceIds: ["tool-failure-17"],
    counterevidence: []
  },
  summary: "Prevent a reproduced incompatible PDF edit path."
}

The proposal identifies the exact change, but it still goes through the system's validation and release process before becoming active.

Validation

The system confirms that the target is current, applies the operation to its source format, runs the right tests or evaluations, presents the change for review when required, and creates a new version only after validation passes.

This keeps a model decision from silently becoming production state. In OpenPond, immutable Harness releases, workspaces, overlays, proposal receipts, targeted validation, advancement, merge, and rollback make the full path auditable. Other systems can map the same decision into their existing configuration and release process.

For an update, the integration first checks that find still matches the target exactly. It can then build the candidate source in a workspace, run targeted evals, and record the result in a validation receipt. Passing validation can advance the candidate into a new immutable Harness release. The prior release remains available for rollback.

Repeated issues can also live as cross-run candidates before a source change is proposed. Those candidates keep their supporting incidents, counterevidence, authorization state, and resolution history together. A later successful run can resolve a candidate without an edit; revoked evidence can remove it from consideration.

The rest of the Harness

The Refiner is one part of @openpond/harness. The package also contains immutable Agent snapshots and Harness releases, content-addressed identities, workspace and overlay contracts, improvement observations, trigger detection, cross-run candidate lifecycle, proposal and validation receipts, advancement and rollback records, tool declarations, model identities, and trace contracts.

The model-driven Refiner sits inside that protocol. The prompt supplies semantic judgment; the schemas, evidence bounds, critique pass, admission rules, immutable identities, and receipts make the judgment inspectable.

The pieces fit together like this:

host: run work, own sources, authorize evidence, apply changes
harness package: define evidence, review policy, decisions, lineage, receipts
model: judge whether the bounded evidence supports an intervention

The Refiner studies bounded evidence, decides whether an intervention is warranted, and produces the next action. Review and validation stay in the path.

Install it with npm install @openpond/harness, and pair it with @openpond/evals when routed evaluation work needs portable Tasksets, graders, and run receipts. The package README in the OpenPond repository documents the standalone provider adapter and evidence boundary.