Skip to content
OpenPond
Join community
Evals/Evaluation overview

Evaluation overview

Measure models and agents with datasets, graders, and experiments.

Evaluation answers whether a model or agent performs a defined task well. It also lets you check whether the grader measures the intended outcome. Neither operation changes model weights.

Follow the Console workflow

  1. Open Datasets, create or import examples, and inspect their input, references, provenance, and execution requirements.
  2. Open Graders, define the scoring criteria, and validate them against known passing, failing, and ambiguous examples.
  3. Open Experiments, select the dataset and target, review execution mode and limits, and start the comparison.
  4. Inspect case-level results, scores, errors, and retained evidence before deciding what to improve.
ResourcePurpose
DatasetsVersioned tasks and their evaluation contract
GradersRepeatable scoring, including required gates
ExperimentsExecutions and results for a chosen target and dataset
EnvironmentsThe runtime contract required to reproduce attempts
Human reviewLabels and explanations for recorded responses

Publish reviewed releases before comparing targets. Keep dataset and grader versions fixed across a baseline and candidate comparison. Separate held-out tasks from training and from examples used to improve the judge.

If an experiment identifies a repeatable failure, choose the smallest effective change: fix task data, improve a grader, adjust the Harness, or prepare a separately approved Training run.

Use the open-source package

@openpond/evals provides portable Taskset, grader, run-manifest, receipt, and comparison contracts. It does not execute Desktop or sandbox sessions, resolve credentials, or persist artifacts. Your host owns execution adapters, permissions, and evidence storage.

bash
npm install @openpond/evals

Validate retained receipt data at the boundary before using it in your application:

ts
import { readFile } from "node:fs/promises"; import { AttemptReceiptSchema } from "@openpond/evals"; const rawReceipt = JSON.parse(await readFile("./attempt-receipt.json", "utf8")); const receipt = AttemptReceiptSchema.parse(rawReceipt); console.log(receipt);

Schema parsing checks the receipt shape; it does not prove that the host executed the task or that referenced artifacts are authentic. Use the package verification operations with the pinned release and evidence before accepting a result. The package README owns its current APIs and subpaths.

To execute hosted experiments, use the supported OpenPond CLI, Console, or openpond-sdk/experiments. That execution client uses stable operation IDs, whole-experiment spending caps, immutable configuration, cancellation, retained results, and comparison. See Experiments.

View as Markdown