An agent is random, so one run does not show how good it is. Two metrics score the same agent over runs, and they move in opposite directions.

Meaning

MetricQuestionSuccess whenAnalogy
pass@kCan it everAt least one of the runs worksA job interview, where one great answer gets you hired
pass^kDoes it alwaysAll runs workThe job itself, where you must deliver every day

pass@k is what a demo shows. pass^k is what customers experience, since every shopper is one run (tau-bench, Yao et al. 2024).

Formulas

If the chance that one run succeeds is :

  • is 1 minus the chance that all runs fail, and it grows towards 1 as grows
  • is the chance that all runs succeed, and it shrinks towards 0 as grows
  • with and , pass@4 is but pass^4 is

From data, run the task times and count successes. The standard estimates are:

Worked example

The agent runs the shopping task 8 times, with 6 successes and 2 failures, and .

  • pass@4 , since with only 2 failures any 4 runs include a success
  • pass^4 , the chance that 4 shoppers in a row are all served correctly

The same agent scores 100% one way and 21% the other way. For an agent that spends money, report pass^k.

Remembering tip

pass@k is a job interview (one good try is enough), and pass^k is the job (every day counts).

Code

from math import comb
 
def pass_at_k(n, c, k):
    """Estimated chance that at least one of k runs succeeds."""
    return 1 - comb(n - c, k) / comb(n, k)
 
def pass_hat_k(n, c, k):
    """Estimated chance that all k runs succeed."""
    return comb(c, k) / comb(n, k)
 
print(f"pass@4 = {pass_at_k(8, 6, 4):.2f}   pass^4 = {pass_hat_k(8, 6, 4):.3f}")   # 1.00, 0.214
 
p = 0.75
print(f"p = 0.75: pass@4 = {1 - (1 - p) ** 4:.3f}   pass^4 = {p ** 4:.3f}")        # 0.996, 0.316

Notes:

  • comb(n - c, k) is 0 when there are fewer than failures, which is why pass@4 is exactly 1.0 here
  • a naive agent that slips one time in four has a pass@4 near 1 but a pass^4 of about 0.3

Counting successes needs a clear definition of success, see Agent Evaluation.