Skip to content
OpenPond
Join community
Evals/Experiments

Experiments

Run fresh attempts or score recorded evidence against pinned Dataset and grader releases.

An evaluation experiment retains the target, Dataset release, graders, task population and execution context for one inspectable run. Open Experiments or start from a Dataset or Model. New Experiment setup compares outputs without training the selected target. Training and adapter configuration start from Models; historical training runs remain under Models → Runs and their original links.

The setup keeps Model, Dataset, Graders, Execution and Review together. Moving between tabs preserves your selections. Review the exact target, task population, grading and spending before starting. Save retains the draft; saving or changing tabs does not execute tasks or enable a Model's continual-learning policy.

Pick the question before the execution mode

ModeWhat executesWhat the result establishes
Fresh model executionThe selected model generates responses to held-out tasks.Performance of that exact model/configuration on those tasks.
Model with Harness or agent behaviorA published behavior and permitted environment produce attempts.Performance of the complete agent workflow, including tools.
Recorded responsesGraders inspect retained attempts without regenerating them.Quality of that captured work or the grader; not fresh-generation reproducibility.

Availability depends on the Dataset, target, runtime and captured evidence. Evaluation performs inference and grading; it has no optimizer that updates the target's weights.

What happens inside a task

A tool failure, missing response or unavailable judge is evidence to inspect, not a passing case. A model judge is another inference dependency; its output is not ground truth merely because it is fluent. Validate it on known passing and failing examples and use Human review for ambiguous disagreements.

Worked example: compare structured support answers

Publish an independent holdout Dataset with two illustrative cases:

TaskInputExpected behavior
damaged-order{"ticket":"My parcel arrived damaged"}Valid JSON with action: "replace" and a useful request for a photo.
late-order{"ticket":"My parcel is three days late"}Valid JSON with action: "track" and a useful tracking reply.

Pin a structural JSON check and a separately reviewed support-quality grader. Keep the exact same Dataset/grader versions and task seeds for the baseline and candidate; do not train on these holdout cases.

  1. Open New → Experiment to configure a standard test. You can also use the Dataset's experiment action.
  2. In Model, name the run, select the exact baseline Model and choose fresh execution. A saved adapter requires its compatible exact Model identity; choosing its original base would evaluate a different model.
  3. In Dataset, select the released holdout and task population. Inspect scoring in Graders and request/runtime limits and accepted spending controls in Execution.
  4. In Review, check target/release identities and approve execution. Saving a draft or changing tabs does not run tasks.
  5. Repeat for the candidate with unchanged holdout/grader versions and limits. Inspect individual answers and failure evidence as well as the metric.

Equivalent TypeScript SDK / API

OpenPondModelTasksetRunsClient uses the real POST /v1/model-taskset-runs operation. This complete request example uses illustrative IDs and a hash: replace them with your actual released Dataset, selected model and task IDs. modelId is the available inference identity, not an arbitrary model name. modelProjectId can be null for a standalone run.

typescript
import { OpenPondModelTasksetRunsClient } from "openpond-sdk/model-taskset-runs"; const client = new OpenPondModelTasksetRunsClient({ baseUrl: process.env.OPENPOND_API_URL ?? "https://api.openpond.ai", apiKey: process.env.OPENPOND_API_KEY!, teamId: "your-workspace", }); const run = await client.create({ schemaVersion: "openpond.modelTasksetRunRequest.v1", teamId: "your-workspace", operationId: "support-holdout-base-001", // Retain this identity on retry. modelProjectId: null, name: "Support holdout baseline", taskset: { id: "your-published-holdout", revision: 1, contentHash: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", }, policy: { kind: "hosted_chat", modelId: "your-available-inference-model", maxOutputTokens: 256, temperature: 0, topP: 1, }, population: [ { receiptId: "support-base-damaged-001", taskId: "damaged-order", seed: "17", fixtureId: null }, { receiptId: "support-base-late-001", taskId: "late-order", seed: "17", fixtureId: null }, ], }); console.log(run.summary.id, run.summary.status); const current = await client.get(run.summary.id); if (current.summary.resultAvailable) { const result = await client.result(run.summary.id); console.log(result.run.summary, result.receipts, result.metric); }

Create the candidate run with a new operation/receipt identity and the candidate's supported policy binding. Reuse the original identities only when recovering the same run. A candidate-artifact policy binding is an exact typed reference from the candidate's supported evaluation flow, not a replacement modelId fabricated from its label. For a Harness target, use its released hosted_harness policy; recorded-response grading has its own operation. Do not change this example to fixture and describe the output as model performance.

Representative result and evidence

A terminal summary can expose the following partial projection:

json
{ "status": "completed", "totalCount": 2, "counts": {"pending": 0, "running": 0, "completed": 2, "failed": 0, "cancelled": 0}, "cleanupComplete": true, "resultAvailable": true, "score": 0.5 }

This is an illustrative response excerpt, not a full wire response or an actual accepted benchmark. For a pass-rate metric, one passing case out of two yields 0.5; the pinned metric definition decides how real grades are aggregated. A candidate scoring 1.0 on this tiny holdout would justify inspecting those two cases, not claiming broad reliability. The full result retains its admitted request/manifest, per-task receipts, metric and integrity hash.

For example, the damaged-order task could retain this decoded answer:

json
{"action":"replace","reply":"Please send a photo of the damage."}

An illustrative case review, rather than an API response shape, would read:

EvidenceObservation
Task input{"ticket":"My parcel arrived damaged"}
JSON structure checkPass: the response parses and contains the required fields.
Expected-action checkPass: action is replace.
Support-quality reviewThe reply asks for a photo; inspect the pinned grader's actual grade and rationale.

The answer and observations above are not recorded results. In a real run, open the task's response and grading evidence in the experiment, or inspect result.receipts from the SDK call above. Keep the task/receipt IDs, grader release and any tool/verifier evidence beside the grade. A plausible answer without a retained grade must not be counted as a passing case.

Inspect results and decide what to change

Evaluation executes or grades the selected population and retains the result. On a completed Experiment, inspect the task responses, grades and pinned versions. Open Diagnostics for retained execution evidence. Use those records to identify the responsible layer before preparing a separate change.

A diagnosis can point to the model, grader, task or environment. Improving a grader needs independent validation before adopting the new scoring rules. Training needs separately admitted training tasks and approved spending; held-out examples cannot be repurposed just because an evaluation failed. Follow Human review for admission decisions and Continual learning for the scheduled path. The evaluation tutorial connects these steps.

Recover, compare and continue

  • Lost reply: reopen or read the original run before retrying. Keep its operation ID; a fresh identity can create duplicate execution and charges.
  • Cancel: follow cancellation through terminal population accounting and confirmed cleanup. Do not treat a request acknowledgement as released capacity.
  • Compare: hold task release, grading rules, seeds and meaningful limits fixed. If a grader changes, re-evaluate the baseline before claiming a gain. Keep grader-development examples separate from independent validation.
  • Training: open Models to configure a separate training run. Review the exact starting version, Dataset, graders, admitted examples and spending. An evaluation does not authorize copying responses, changing weights or activating serving. See Training.
  • Recurring learning: recurring model updates belong to the Model's continual-learning policy, not a repeated evaluation button. Saved drafts do not override permissions or runtime qualification.

Sources

Compatibility

Execution depends on the selected target, released Dataset and graders, runtime readiness, permissions, and accepted spending limits.

View as Markdown