Overview

  • simulators are virtual 3D worlds where agents train and test safely and at scale
  • the ecosystem has three layers: tasks and metrics, simulators, and environments (3D scene datasets)
  • example simulators and scene sets: DeepMind Lab, VizDoom, AI2-THOR, SUNCG, Matterport3D, Gibson and Replica

Embodied AI simulators are virtual 3D worlds where agents can be trained and tested safely and at scale. Most embodied AI work starts in a simulator, because learning in the real world is dangerous, expensive, slow and hard to benchmark.

Three layers of the ecosystem

  • tasks, metrics and algorithms: EmbodiedQA (Das et al.), language grounding (Chaplot et al., Hermann and Hill et al.), Interactive QA (Gordon et al.), Vision-Language Navigation (Anderson et al.) and visual navigation (Zhu and Gordon et al., Savva et al., Wu et al.)
  • simulators: House3D, AI2-THOR, MINOS, Gibson, CHALET, HoME, VirtualHome, AdobeIndoorNav and Matterport3DSim
  • environments (3D scene datasets): SUNCG, Matterport3D and Stanford 2D-3D-S

Example simulators and environments

  • DeepMind Lab: a game-like 3D maze world
  • VizDoom: based on the game Doom; tasks can be given in language, such as "Go to the short blue torch"
  • AI2-THOR: interactive indoor rooms where objects such as doors can be used
  • SUNCG: synthetic houses with furniture, such as kitchens
  • Matterport3D, Gibson and Replica: scans of real buildings, so the views look photorealistic

Note

Training in simulation raises the question of how well the agent works on a real robot, which is the topic of Sim2Real Transfer.