Overview
- AgentTrek makes agent training data from web tutorials, with no human labelling
- pipeline: collect, tag and rephrase, guided replay, evaluate
- it belongs to the data pillar of multimodal agents
AgentTrek is a method for making agent training data from web tutorials. It solves the data problem of GUI agents without human annotators.
Problem and idea
- problem: agent trajectories (step-by-step records of what an agent sees and does) need costly human annotation, and there is no free corpus of them to crawl
- idea: the web already contains many step-by-step GUI tutorials written for humans
Pipeline

- collect: filter tutorials out of web snapshots
- tag and rephrase: an LLM extracts the task description, the prerequisites and the steps
- guided replay: a VLM agent follows the steps in a real browser
- evaluate: an evaluator keeps only the successful runs, with the full video, the HTML record, the agent's thoughts and its actions
Note
AgentTrek is one of the data sources for a multimodal agent. Another is TACO, which builds chains of thought and tool actions, see TACO.