Overview

  • AgentTrek makes agent training data from web tutorials, with no human labelling
  • pipeline: collect, tag and rephrase, guided replay, evaluate
  • it belongs to the data pillar of multimodal agents

AgentTrek is a method for making agent training data from web tutorials. It solves the data problem of GUI agents without human annotators.

Problem and idea

  • problem: agent trajectories (step-by-step records of what an agent sees and does) need costly human annotation, and there is no free corpus of them to crawl
  • idea: the web already contains many step-by-step GUI tutorials written for humans

Pipeline

  1. collect: filter tutorials out of web snapshots
  2. tag and rephrase: an LLM extracts the task description, the prerequisites and the steps
  3. guided replay: a VLM agent follows the steps in a real browser
  4. evaluate: an evaluator keeps only the successful runs, with the full video, the HTML record, the agent's thoughts and its actions

Note

AgentTrek is one of the data sources for a multimodal agent. Another is TACO, which builds chains of thought and tool actions, see TACO.