A task succeeds only if every step succeeds, like a string of old fairy lights where one dead bulb turns off the whole string. Engineers call this a series system.
Formula
| Piece | Meaning |
|---|---|
| The chance that step succeeds | |
| The number of steps | |
| If every step has the same chance , multiply by itself times | |
| Caveat | It assumes that steps fail independently, so it is a rough first estimate and not an exact law |
Worked example
A 20 step task, where each step succeeds with :
- without a checker, , so the agent finishes the task only about 1 time in 3
- add a checker that catches and fixes 60% of step errors, so the chance of a good step becomes
- with the checker, , which nearly doubles the success
A "95% accurate" agent is fragile on long tasks. Short tasks and good checks beat a slightly better model.
Remembering tip
Long chain, weak chain. Shorten the chain or add a checker at each link.
Code
import numpy as np
print(f"No checker : {0.95 ** 20:.3f}") # 0.358
print(f"With checker : {(0.95 + 0.05 * 0.6) ** 20:.3f}") # 0.668
rng = np.random.default_rng(0)
def simulate(p_step, n_steps, catch=0.0, n_tasks=100_000):
"""Fraction of simulated tasks with no uncaught error. A step error is caught with probability catch."""
error = rng.random((n_tasks, n_steps)) > p_step # True where a step fails
caught = rng.random((n_tasks, n_steps)) < catch # True where a checker fixes it
return np.mean(~(error & ~caught).any(axis=1)) # a task works if no error is left uncaught
print(f"Simulated, no checker : {simulate(0.95, 20):.3f}") # about 0.358
print(f"Simulated, with checker: {simulate(0.95, 20, catch=0.6):.3f}") # about 0.668Notes:
.any(axis=1)checks each task (each row) for at least one uncaught error- the simulation gives the same numbers as the formula, which supports the formula
- the best checker is a rule in code, see Guarded Agent Loop
Errors are not the only thing that grows with steps, the cost grows too, see Runaway Loops and Agent Cost.