Overview
- an agent is an MLLM that keeps looking, deciding and acting until a goal is done
- loop: perceive, reason, act
- three pillars to build one: environment and benchmark, data, and model and system
- it also covers robot VLAs (only the action space changes), then future trends and safety risks
A multimodal agent is a system built on an MLLM that repeatedly observes, decides and acts until a goal is met. An MLLM only answers. An agent also takes actions that change its environment.
From MLLM to agent

- perceive: screenshots, camera images or sensor data are the agent's "eyes"
- reason: the MLLM is the "brain" that decides the next step
- act: actions are code, mouse clicks, key presses or robot motions, and they change the environment, which gives a new observation
Agent domains and their benchmarks:
- coding: SWE-bench
- web: WebArena, Mind2Web
- mobile: AndroidWorld
- physical: robots, for example Figure Helix
Three pillars for building agents
- environment and benchmark: reconfigurable, expandable and real, with execution-based evaluation (the result is checked by running it); example: OSWorld
- data: diverse modalities and large-scale trajectories (recorded sequences of observations and actions) across many tasks; examples: AgentTrek and TACO
- model and system: a unified vision, language, reasoning and action model, with long context and memory; examples: AGUVIS, BLIP-3-Video and GenS
The capabilities needed are grounding (linking words to the right thing on screen or in the world), reasoning and memory.
Vision-language-action models for robots

A robot agent uses the same recipe as a GUI agent. Only the action space changes, from pyautogui clicks to motor commands.
- a camera image and an instruction, such as "pick up the mug", go into a vision-language model
- an action head outputs action tokens or a flow, which become robot commands (joint angles, gripper)
- RT-2 (2023): fine-tunes a VLM so that actions are emitted as text tokens, so web knowledge transfers to manipulation
- OpenVLA (2024): an open 7B model trained on about 1M real robot episodes
- pi0 and pi0.5 (2024 to 2025): a VLM backbone plus a flow-matching action expert for smooth, high-frequency control; pi0.5 generalises to unseen homes
- V-JEPA 2 (2025): learns a video world model, then plans actions in feature space
More on these models is in VLA.
Where agents are heading
Recent directions:
- harder benchmarks: OSWorld 2.0 has 108 workflows that each take a human about 1.6 hours, with about 318 tool calls against about 30 in OSWorld 1.0
- vision-language-action (VLA) models for robots: RT-2, OpenVLA, pi0, pi0.5
- world models for planning (V-JEPA 2)
- a move from imitation (SFT) to reinforcement learning in real environments
Open challenges:
- grounding: clicking the right pixel, and staying robust when the layout changes
- long horizons: small errors at each step compound over hundreds of steps
- memory: keeping constraints across long tasks and long videos
- safety: untrusted screen content, permissions and irreversible actions
- evaluation: high scores on short tasks can overstate real-world readiness
Safety and reliability
| Risk | Why it happens | Mitigation |
|---|---|---|
| Prompt injection from the screen | A web page or document can contain text that the agent reads as an instruction | Treat all observed content as data, not commands; separate the instruction and content channels |
| Irreversible actions | Deleting files, sending messages, making payments | Require confirmation for risky actions; sandbox or dry-run first |
| Silent failure | The agent reports a success it did not achieve | Execution-based checks; verify the final state; log every step |
| Privacy | Screenshots capture passwords, messages and personal data | Mask sensitive regions; keep data local; limit retention |
An agent that acts in the real world needs the same care as any system with write access.