Experiments
Run fresh attempts or score recorded evidence against pinned Dataset and grader releases.
An evaluation experiment retains the target, Dataset release, graders, task population and execution context for one inspectable run. Open Experiments or start from a Dataset or Model. New Experiment setup compares outputs without training the selected target. Training and adapter configuration start from Models; historical training runs remain under Models → Runs and their original links.
The setup keeps Model, Dataset, Graders, Execution and Review together. Moving between tabs preserves your selections. Review the exact target, task population, grading and spending before starting. Save retains the draft; saving or changing tabs does not execute tasks or enable a Model's continual-learning policy.
Pick the question before the execution mode
| Mode | What executes | What the result establishes |
|---|---|---|
| Fresh model execution | The selected model generates responses to held-out tasks. | Performance of that exact model/configuration on those tasks. |
| Model with Harness or agent behavior | A published behavior and permitted environment produce attempts. | Performance of the complete agent workflow, including tools. |
| Recorded responses | Graders inspect retained attempts without regenerating them. | Quality of that captured work or the grader; not fresh-generation reproducibility. |
Availability depends on the Dataset, target, runtime and captured evidence. Evaluation performs inference and grading; it has no optimizer that updates the target's weights.
What happens inside a task
A tool failure, missing response or unavailable judge is evidence to inspect, not a passing case. A model judge is another inference dependency; its output is not ground truth merely because it is fluent. Validate it on known passing and failing examples and use Human review for ambiguous disagreements.
Worked example: compare structured support answers
Publish an independent holdout Dataset with two illustrative cases:
| Task | Input | Expected behavior |
|---|---|---|
damaged-order | {"ticket":"My parcel arrived damaged"} | Valid JSON with action: "replace" and a useful request for a photo. |
late-order | {"ticket":"My parcel is three days late"} | Valid JSON with action: "track" and a useful tracking reply. |
Pin a structural JSON check and a separately reviewed support-quality grader. Keep the exact same Dataset/grader versions and task seeds for the baseline and candidate; do not train on these holdout cases.
- Open New → Experiment to configure a standard test. You can also use the Dataset's experiment action.
- In Model, name the run, select the exact baseline Model and choose fresh execution. A saved adapter requires its compatible exact Model identity; choosing its original base would evaluate a different model.
- In Dataset, select the released holdout and task population. Inspect scoring in Graders and request/runtime limits and accepted spending controls in Execution.
- In Review, check target/release identities and approve execution. Saving a draft or changing tabs does not run tasks.
- Repeat for the candidate with unchanged holdout/grader versions and limits. Inspect individual answers and failure evidence as well as the metric.
Equivalent TypeScript SDK / API
OpenPondModelTasksetRunsClient uses the real
POST /v1/model-taskset-runs operation. This complete request example uses
illustrative IDs and a hash: replace them with your actual released Dataset,
selected model and task IDs. modelId is the available inference identity, not
an arbitrary model name. modelProjectId can be null for a standalone run.
import { OpenPondModelTasksetRunsClient } from "openpond-sdk/model-taskset-runs";
const client = new OpenPondModelTasksetRunsClient({
baseUrl: process.env.OPENPOND_API_URL ?? "https://api.openpond.ai",
apiKey: process.env.OPENPOND_API_KEY!,
teamId: "your-workspace",
});
const run = await client.create({
schemaVersion: "openpond.modelTasksetRunRequest.v1",
teamId: "your-workspace",
operationId: "support-holdout-base-001", // Retain this identity on retry.
modelProjectId: null,
name: "Support holdout baseline",
taskset: {
id: "your-published-holdout",
revision: 1,
contentHash: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
},
policy: {
kind: "hosted_chat",
modelId: "your-available-inference-model",
maxOutputTokens: 256,
temperature: 0,
topP: 1,
},
population: [
{ receiptId: "support-base-damaged-001", taskId: "damaged-order", seed: "17", fixtureId: null },
{ receiptId: "support-base-late-001", taskId: "late-order", seed: "17", fixtureId: null },
],
});
console.log(run.summary.id, run.summary.status);
const current = await client.get(run.summary.id);
if (current.summary.resultAvailable) {
const result = await client.result(run.summary.id);
console.log(result.run.summary, result.receipts, result.metric);
}Create the candidate run with a new operation/receipt identity and the
candidate's supported policy binding. Reuse the original identities only when
recovering the same run. A candidate-artifact policy binding is an exact typed
reference from the candidate's supported evaluation flow, not a replacement
modelId fabricated from its label. For a Harness target, use its released
hosted_harness policy; recorded-response grading has its own operation. Do not
change this example to fixture and describe the output as model performance.
Representative result and evidence
A terminal summary can expose the following partial projection:
{
"status": "completed",
"totalCount": 2,
"counts": {"pending": 0, "running": 0, "completed": 2, "failed": 0, "cancelled": 0},
"cleanupComplete": true,
"resultAvailable": true,
"score": 0.5
}This is an illustrative response excerpt, not a full wire response or an actual accepted benchmark. For a pass-rate metric, one passing case out of two yields 0.5; the pinned metric definition decides how real grades are aggregated. A candidate scoring 1.0 on this tiny holdout would justify inspecting those two cases, not claiming broad reliability. The full result retains its admitted request/manifest, per-task receipts, metric and integrity hash.
For example, the damaged-order task could retain this decoded answer:
{"action":"replace","reply":"Please send a photo of the damage."}An illustrative case review, rather than an API response shape, would read:
| Evidence | Observation |
|---|---|
| Task input | {"ticket":"My parcel arrived damaged"} |
| JSON structure check | Pass: the response parses and contains the required fields. |
| Expected-action check | Pass: action is replace. |
| Support-quality review | The reply asks for a photo; inspect the pinned grader's actual grade and rationale. |
The answer and observations above are not recorded results. In a real run,
open the task's response and grading evidence in the experiment, or inspect
result.receipts from the SDK call above. Keep the task/receipt IDs, grader
release and any tool/verifier evidence beside the grade. A plausible answer
without a retained grade must not be counted as a passing case.
Inspect results and decide what to change
Evaluation executes or grades the selected population and retains the result. On a completed Experiment, inspect the task responses, grades and pinned versions. Open Diagnostics for retained execution evidence. Use those records to identify the responsible layer before preparing a separate change.
A diagnosis can point to the model, grader, task or environment. Improving a grader needs independent validation before adopting the new scoring rules. Training needs separately admitted training tasks and approved spending; held-out examples cannot be repurposed just because an evaluation failed. Follow Human review for admission decisions and Continual learning for the scheduled path. The evaluation tutorial connects these steps.
Recover, compare and continue
- Lost reply: reopen or read the original run before retrying. Keep its operation ID; a fresh identity can create duplicate execution and charges.
- Cancel: follow cancellation through terminal population accounting and confirmed cleanup. Do not treat a request acknowledgement as released capacity.
- Compare: hold task release, grading rules, seeds and meaningful limits fixed. If a grader changes, re-evaluate the baseline before claiming a gain. Keep grader-development examples separate from independent validation.
- Training: open Models to configure a separate training run. Review the exact starting version, Dataset, graders, admitted examples and spending. An evaluation does not authorize copying responses, changing weights or activating serving. See Training.
- Recurring learning: recurring model updates belong to the Model's continual-learning policy, not a repeated evaluation button. Saved drafts do not override permissions or runtime qualification.
Sources
- Biderman et al., Lessons from the Trenches on Reproducible Evaluation of Language Models — evaluation sensitivity, comparison and reproducibility.
- EleutherAI Language Model Evaluation Harness — primary implementation and task/metric documentation; these external benchmarks are not OpenPond acceptance results.
- OpenPond model-taskset run contracts and client — request, retained population, receipts and result-integrity behavior.
Compatibility
Execution depends on the selected target, released Dataset and graders, runtime readiness, permissions, and accepted spending limits.