Overview

  • an agent is an MLLM that keeps looking, deciding and acting until a goal is done
  • loop: perceive, reason, act
  • three pillars to build one: environment and benchmark, data, and model and system
  • it also covers robot VLAs (only the action space changes), then future trends and safety risks

A multimodal agent is a system built on an MLLM that repeatedly observes, decides and acts until a goal is met. An MLLM only answers. An agent also takes actions that change its environment.

From MLLM to agent

  • perceive: screenshots, camera images or sensor data are the agent's "eyes"
  • reason: the MLLM is the "brain" that decides the next step
  • act: actions are code, mouse clicks, key presses or robot motions, and they change the environment, which gives a new observation

Agent domains and their benchmarks:

  • coding: SWE-bench
  • web: WebArena, Mind2Web
  • mobile: AndroidWorld
  • physical: robots, for example Figure Helix

Three pillars for building agents

  • environment and benchmark: reconfigurable, expandable and real, with execution-based evaluation (the result is checked by running it); example: OSWorld
  • data: diverse modalities and large-scale trajectories (recorded sequences of observations and actions) across many tasks; examples: AgentTrek and TACO
  • model and system: a unified vision, language, reasoning and action model, with long context and memory; examples: AGUVIS, BLIP-3-Video and GenS

The capabilities needed are grounding (linking words to the right thing on screen or in the world), reasoning and memory.

Vision-language-action models for robots

A robot agent uses the same recipe as a GUI agent. Only the action space changes, from pyautogui clicks to motor commands.

  • a camera image and an instruction, such as "pick up the mug", go into a vision-language model
  • an action head outputs action tokens or a flow, which become robot commands (joint angles, gripper)
  • RT-2 (2023): fine-tunes a VLM so that actions are emitted as text tokens, so web knowledge transfers to manipulation
  • OpenVLA (2024): an open 7B model trained on about 1M real robot episodes
  • pi0 and pi0.5 (2024 to 2025): a VLM backbone plus a flow-matching action expert for smooth, high-frequency control; pi0.5 generalises to unseen homes
  • V-JEPA 2 (2025): learns a video world model, then plans actions in feature space

More on these models is in VLA.

Where agents are heading

Recent directions:

  • harder benchmarks: OSWorld 2.0 has 108 workflows that each take a human about 1.6 hours, with about 318 tool calls against about 30 in OSWorld 1.0
  • vision-language-action (VLA) models for robots: RT-2, OpenVLA, pi0, pi0.5
  • world models for planning (V-JEPA 2)
  • a move from imitation (SFT) to reinforcement learning in real environments

Open challenges:

  • grounding: clicking the right pixel, and staying robust when the layout changes
  • long horizons: small errors at each step compound over hundreds of steps
  • memory: keeping constraints across long tasks and long videos
  • safety: untrusted screen content, permissions and irreversible actions
  • evaluation: high scores on short tasks can overstate real-world readiness

Safety and reliability

RiskWhy it happensMitigation
Prompt injection from the screenA web page or document can contain text that the agent reads as an instructionTreat all observed content as data, not commands; separate the instruction and content channels
Irreversible actionsDeleting files, sending messages, making paymentsRequire confirmation for risky actions; sandbox or dry-run first
Silent failureThe agent reports a success it did not achieveExecution-based checks; verify the final state; log every step
PrivacyScreenshots capture passwords, messages and personal dataMask sensitive regions; keep data local; limit retention

An agent that acts in the real world needs the same care as any system with write access.