Repeat rate
Calculate confirmed repeats among scored relevant opportunities, keeping Evaluation and Live evidence separate.
Repeat rate measures how often a corrected mistake appears again among scored relevant opportunities. It describes an observed population, not every possible future attempt or the model's general benchmark accuracy.
The calculation
scored relevant opportunities = confirmed repeats + passed opportunities
repeat rate = confirmed repeats / scored relevant opportunities
percentage = repeat rate × 100For each source s, Cₛ counts confirmed repeats and Pₛ counts passed opportunities. Unscored opportunities stay visible but are excluded from the denominator.
A relevant opportunity is an assessed attempt linked to the declared corrected behavior for the exact scoped Model, run, checkpoint and artifact. Its classification is confirmed repeat, passed or unscored. An unrelated grader failure is not a confirmed repeat. Retained opportunity identity prevents the same opportunity being counted twice in a summary.
Illustrative calculation, not an OpenPond measurement: 4 confirmed repeats and 16 passed opportunities give 4 / 20 = 20%. Three additional unscored opportunities remain visible but do not change that denominator.
| Evidence | Report |
|---|---|
| 0 confirmed repeats / 20 scored opportunities | 0%, for that observed population. |
| 0 scored opportunities | Not measured, even if unscored attempts exist. |
| Many unscored opportunities | Report their count alongside the rate; do not treat them as passes. |
Evaluation and Live are separate sources
| Source | What it represents | Interpretation |
|---|---|---|
| Evaluation | Declared correction checks against retained evaluation attempts. | Behavior on the declared checks and populations. |
| Live | Relevant user-visible attempts linked to the corrected behavior. | Observed behavior in the covered live population. |
Calculate each source's numerator and denominator independently. Do not pool Evaluation and Live without an explicit new analysis. A compact R badge uses Live when a measured Live rate exists; otherwise it can show Evaluation. Inspect its source label and counts in Correction outcomes.
Read the scope and denominator
Check the workspace, Model, run, exact checkpoint and artifact, correction window, opportunity window and observation time. A result for one checkpoint does not describe its base, later candidates or every serving target.
A low rate on a tiny or narrowly selected population is a narrow result. Review the scored and unscored counts, declared checks, failed baseline and candidate evidence before claiming improvement or generalization. Ordinary benchmark accuracy does not replace correction-specific checks.
Open the exact run's Correction outcomes, then follow the retained correction/task policy and evaluation links. These reads summarize existing evidence; they do not run fresh inference or training.
Repeat rate is distinct from adaptation time, which ends at the first verified improvement. It is also separate from replay fraction, optimizer learning rate and import or training frequency. See Research and evaluation for the broader questions.
Contract reference
The public correction-metrics contracts define retained opportunity classifications, scoped windows and separate Evaluation/Live counts. UI availability depends on retained evidence and the running release. The formula above explains the calculation; it is not a new measured result.
Compatibility
Requires retained correction-specific opportunities for the exact Model/run/checkpoint/artifact. No scored opportunities means Not measured.