A runaway agent never stops. If you send a friend to buy a charger and every shop is sold out, a sensible friend phones you after the third shop. A runaway agent keeps searching, rephrasing and retrying, and every step costs money.

Stop rules

RuleExample in the shopping task
A step limitStop after 15 steps
A budgetStop after spending, say, 0.50 USD on the LLM
A "going in circles" checkStop if the same search returns the same results twice
Ask a person"Nothing under 50 dollars arrives by Friday, raise the limit?"

Cost of a loop

Every step re-reads everything so far: a fixed part of tokens, plus new tokens at each step. Over steps the total tokens read is:

Worked example with , and a price of 3 USD per million tokens:

  • a single call with one search result reads tokens, which costs 0.0075 USD
  • 10 steps read tokens, which costs about 0.13 USD
  • the ratio is , so 10 steps cost 17 times one call
  • the total grows with , because each step re-reads a growing history

A sold-out shop with no stop rule, stopped only by a safety cap of 100 steps, reads tokens, or over 8 USD to buy nothing.

A missing stop rule is expensive

At 100,000 shoppers a day, a missing stop rule is expensive. Summarising old steps and caching repeated text keep the cost down.

Code

S, D = 2_000, 500                   # fixed tokens, new tokens per step
PRICE = 3 / 1_000_000               # USD per input token
 
def tokens_read(n_steps):
    """Total tokens read when every step re-reads the whole history: n*S + d*n(n-1)/2."""
    return sum(S + D * t for t in range(n_steps))
 
single = S + D
print(f"10 steps : {tokens_read(10):,} tokens = USD {tokens_read(10) * PRICE:.4f}")   # 42,500, 0.1275
print(f"One call : {single:,} tokens = USD {single * PRICE:.4f}")                      # 2,500, 0.0075
print(f"Ratio    : {tokens_read(10) / single:.0f}x")                                    # 17x
print(f"100 steps: USD {tokens_read(100) * PRICE:.2f}")                                 # 8.03

A stop rule against going in circles can be added to the loop like this:

seen = set()
for step in range(max_steps):
    call = llm(goal, history)
    key = (call["name"], str(call["arguments"]))
    if key in seen:                                   # the same call twice: going in circles
        history.append({"call": call, "obs": "stopped: going in circles, asking a person"})
        break
    seen.add(key)
    # ... run the tool as before

Notes:

  • tokens_read(n) is the same as written as a sum
  • a repeated call is a sign that the agent is going in circles

Cost is one side of reliability, and the other side is how often the agent succeeds, see Pass At K and Pass Hat K.