Overview
- OSWorld is a real computer test bed where agents do tasks on Ubuntu, Windows and macOS
- the result is checked by running a script on the final state of the machine
- it covers what the agent sees (screenshot, a11y tree, Set-of-Marks) and how it acts (pyautogui code)
- it ends with progress over time and lessons, such as screenshot-only agents being the long-term goal
OSWorld is the first scalable real computer environment for testing multimodal agents. The agent must complete tasks on real operating systems and real applications, and the result is checked by running a script on the final state of the machine.
Environment

- virtual machines run Ubuntu, Windows and macOS
- 369 tasks use real web and desktop apps, file input and output, and workflows that span several apps
- each task has an instruction, an initial-state setup and an execution-based evaluation script
Observations and actions
What the agent observes:
- a screenshot
- the accessibility (a11y) tree, a text description of the elements on screen
- Set-of-Marks, numbered boxes drawn on the screenshot so the agent can refer to elements
- terminal output
What the agent does: it writes executable pyautogui code for the mouse and keyboard (for example moveTo, click, write, press, hotkey, scroll, dragTo), plus three special actions: WAIT, FAIL and DONE.
Interaction loop
obs = env.reset(task)
while not done:
act = agent(instruction, obs, history)
obs, done = env.step(act)
reward = evaluate(env.final_state)The environment is reset to the task's start state. In each step the agent reads the instruction, the latest observation and the history, and outputs an action. At the end, the final state is evaluated.
Progress

Task success rate:
- best model in the OSWorld paper (2024): 12.2%
- Claude 3.5 Sonnet with 15 steps (October 2024): 14.9%
- OpenAI CUA (January 2025): 38.1%
- human baseline: 72.4%
- Claude Opus 4.8 on OSWorld-Verified (2026): 83.5%
The setups differ (step limits, OS images, verified re-runs), so the trend matters more than the exact gaps.
Lessons from the analysis
- a higher screenshot resolution helps
- a longer text history helps, but a history made only of screenshots does not help and is costly
- agents are not robust to changes in UI layout and to noise
- screenshot-only agents are weaker today, but they are the long-term goal, because screenshots work for every app
Note
OSWorld is the environment pillar of a multimodal agent system.