Adaptation time and repeat rate
Understand the retained correction measurements, their exact calculations, and what missing evidence means.
Ponder separates two correction outcomes: how long a verified adaptation took and how often the corrected mistake appeared again. Adaptation is measured as elapsed time. Neither number is the optimizer's learning rate or a promise that every submitted correction has been learned.
Read the scope first
The summary identifies:
- workspace, Model and training run;
- exact checkpoint and artifact;
- the correction-receipt window, opportunity window and observation time;
- resolved and pending correction counts;
- separate Evaluation and Live repeat evidence.
Compare like scopes and windows. A result for one checkpoint does not establish the behavior of its original base, a later candidate or every serving target.
Adaptation time
For a resolved correction, the elapsed time is:
adaptation time = first verified improvement time − correction received time
summary adaptation time = median of resolved correction timesA retained receipt is only the start. Verification requires actual training exposure, a failed parent check, and the candidate passing all declared checks under the frozen correction-verification plan. Batch reservation or consumption alone is not evidence that the optimizer used the correction.
The summary reports more than the median:
| Value | Meaning |
|---|---|
| Median adaptation time | Elapsed receipt-to-first-verification time among resolved corrections with training exposure. |
| Resolved / tracked corrections | How much of the scoped correction population contributes resolved durations. |
| Pending corrections | Scoped, non-withdrawn corrections without a retained verified-improvement time. |
| Oldest pending correction | Age of the oldest pending correction at the end of the window. |
Illustrative calculation: resolved times of 1 hour, 3 hours and 8 hours give a median of 3 hours. If two other corrections are still pending and the oldest is 12 hours old, report those two pending cases and 12-hour age alongside the median. Excluding them from the median does not mean they adapted instantly.
No resolved durations means Not measured, not zero seconds. A fast median with many old pending corrections is incomplete evidence. Candidate verification also does not establish when a version became served; that timestamp is separate.
Repeat rate
A relevant opportunity is an assessed attempt linked to the declared corrected behavior for the exact scoped Model/checkpoint/artifact. Its classification is confirmed repeat, passed or unscored.
scored relevant opportunities = confirmed repeats + passed opportunities
repeat rate = confirmed repeats / scored relevant opportunities
percentage = repeat rate × 100For example, 4 confirmed repeats and 16 passed opportunities give 4 / 20 = 20%. Three additional unscored opportunities remain visible but are excluded from that denominator. This is an illustrative calculation, not an OpenPond measurement.
- 0 repeats / 20 scored opportunities = 0% for that observed population.
- 0 scored opportunities = Not measured, even if there are unscored attempts.
- An unrelated grader failure is not a confirmed repeat.
- Ordinary benchmark accuracy is not a substitute for correction-specific checks.
- Opportunity identity prevents the same retained opportunity being counted twice in the scoped summary.
Evaluation and Live stay separate
| Source | What it represents | Interpretation |
|---|---|---|
| Evaluation | Declared correction checks against retained evaluation attempts. | Behavior on the declared checks and populations. |
| Live | Relevant user-visible attempts linked to the corrected behavior. | Observed behavior in the covered live population. |
Do not pool these denominators without an explicit new analysis. A compact R badge uses Live when a measured Live rate exists; otherwise it can show Evaluation. Inspect the source label and counts in Correction outcomes. The A badge shows measured adaptation time; missing values may omit a badge.
A low repeat rate on a tiny or narrowly selected population is a narrow result. Review the denominator, unscored count, declared checks, baseline failure and candidate evidence before claiming improvement or generalization.
Where to inspect the evidence
Open the exact run's Correction outcomes, then follow the scoped run, correction/task policy and evaluation links. Check blockers when exposure or verification is pending. These reads summarize retained evidence; they do not run fresh inference or training.
Keep these measurements separate from:
- optimizer learning rate, which changes update size;
- replay fraction, which allocates earlier training examples in a batch;
- import or training frequency, which controls scheduling;
- broad adaptation/retention claims from research benchmarks.
See Research and evaluation for the broader questions, and How it works for admission, training and candidate gates.
Contract reference
The public correction-metrics contracts define retained observations, scoped windows, nullable measurements and separate Evaluation/Live counts. UI availability depends on retained evidence and the running release.
Compatibility
These outcomes require retained correction, training-exposure and verification evidence for the exact Model/run/checkpoint/artifact. Missing measurements are not zero.