The LLM proposes and the code decides. Each safeguard lives in code, where the LLM cannot talk its way past it, like the access control in RAG.

Safeguards

SafeguardWhat it stops in the poisoned listing task
Approved sellers onlyBuying from TechDealz88
Mandate check in codeThe 65.18 total
Human approval showing the real totalA misleading summary
Step and spending limitsEndless retries
A log of every stepUnanswerable disputes

No single safeguard is enough. Together they turn a successful trick from "money gone" into "a refused cart and a log entry".

Code

The same loop as the bare loop, with 4 safeguards:

def run_guarded(goal, llm, shop, mandate, ask_user, max_steps=15):
    """The agent loop with safeguards: the LLM proposes, code checks, and only then does anything happen."""
    tools, history = make_tools(shop), []
 
    for step in range(max_steps):                                 # 1. step limit
        call = llm(goal, history)
        if call["name"] == "finish":
            history.append({"call": call, "obs": call["arguments"]["answer"]})
            return history
        problems = validate_call(call)                            # 2. known tools, valid arguments
        if problems:
            history.append({"call": call, "obs": {"status": "error", "problems": problems}})
            continue
        entry = {"call": call}
        if call["name"] == "checkout":
            cart = shop.current_cart()
            ok, reasons = mandate.allows(cart)                    # 3. mandate check in code
            if not ok:
                history.append({"call": call, "obs": {"status": "refused",
                                                      "product_id": cart["product_id"], "reasons": reasons}})
                continue
            if not ask_user(cart):                                # 4. human approval
                history.append({"call": call, "obs": "stopped: you declined"})
                return history
            entry["approved"] = True
        entry["obs"] = tools[call["name"]](**call["arguments"])   # code runs the tool
        history.append(entry)
    history.append({"call": {"name": "stop", "arguments": {}},   # out of steps: hand over to a person
                    "obs": "Step limit reached: escalating to a person."})
    return history

Notes:

  • validate_call is from Tool Call Validation and mandate.allows is from Agent Mandate (AP2)
  • a refusal goes back to the LLM as an observation, so even a gullible LLM then buys the VoltMate instead
  • in the poisoned listing task the LLM is still fooled and still proposes to pay 65.18, and the mandate check refuses it
  • in the sold-out task the step limit stops the loop after 15 steps, which costs about 0.25 USD instead of over 8 USD
  • the LLM did not get any smarter, and all of the improvement came from the code around it

Remembering tip

Limit, Known, Mandate, Approve. The four safeguards are the step limit, known tools with valid arguments, the mandate check, and human approval.

Writing these controls down, with named people responsible, is governance, see Agentic AI Governance (IMDA).