An agent is random, so one run does not show how good it is. Two metrics score the same agent over runs, and they move in opposite directions.
Meaning
| Metric | Question | Success when | Analogy |
|---|---|---|---|
pass@k | Can it ever | At least one of the runs works | A job interview, where one great answer gets you hired |
pass^k | Does it always | All runs work | The job itself, where you must deliver every day |
pass@k is what a demo shows. pass^k is what customers experience, since every shopper is one run (tau-bench, Yao et al. 2024).
Formulas
If the chance that one run succeeds is :
- is 1 minus the chance that all runs fail, and it grows towards 1 as grows
- is the chance that all runs succeed, and it shrinks towards 0 as grows
- with and , pass@4 is but
pass^4is
From data, run the task times and count successes. The standard estimates are:
Worked example
The agent runs the shopping task 8 times, with 6 successes and 2 failures, and .
- pass@4 , since with only 2 failures any 4 runs include a success
pass^4, the chance that 4 shoppers in a row are all served correctly
The same agent scores 100% one way and 21% the other way. For an agent that spends money, report pass^k.
Remembering tip
pass@kis a job interview (one good try is enough), andpass^kis the job (every day counts).
Code
from math import comb
def pass_at_k(n, c, k):
"""Estimated chance that at least one of k runs succeeds."""
return 1 - comb(n - c, k) / comb(n, k)
def pass_hat_k(n, c, k):
"""Estimated chance that all k runs succeed."""
return comb(c, k) / comb(n, k)
print(f"pass@4 = {pass_at_k(8, 6, 4):.2f} pass^4 = {pass_hat_k(8, 6, 4):.3f}") # 1.00, 0.214
p = 0.75
print(f"p = 0.75: pass@4 = {1 - (1 - p) ** 4:.3f} pass^4 = {p ** 4:.3f}") # 0.996, 0.316Notes:
comb(n - c, k)is 0 when there are fewer than failures, which is why pass@4 is exactly 1.0 here- a naive agent that slips one time in four has a pass@4 near 1 but a
pass^4of about 0.3
Counting successes needs a clear definition of success, see Agent Evaluation.