Overview
- simulators are virtual 3D worlds where agents train and test safely and at scale
- the ecosystem has three layers: tasks and metrics, simulators, and environments (3D scene datasets)
- example simulators and scene sets: DeepMind Lab, VizDoom, AI2-THOR, SUNCG, Matterport3D, Gibson and Replica
Embodied AI simulators are virtual 3D worlds where agents can be trained and tested safely and at scale. Most embodied AI work starts in a simulator, because learning in the real world is dangerous, expensive, slow and hard to benchmark.
Three layers of the ecosystem
- tasks, metrics and algorithms: EmbodiedQA (Das et al.), language grounding (Chaplot et al., Hermann and Hill et al.), Interactive QA (Gordon et al.), Vision-Language Navigation (Anderson et al.) and visual navigation (Zhu and Gordon et al., Savva et al., Wu et al.)
- simulators: House3D, AI2-THOR, MINOS, Gibson, CHALET, HoME, VirtualHome, AdobeIndoorNav and Matterport3DSim
- environments (3D scene datasets): SUNCG, Matterport3D and Stanford 2D-3D-S
Example simulators and environments
- DeepMind Lab: a game-like 3D maze world
- VizDoom: based on the game Doom; tasks can be given in language, such as "Go to the short blue torch"
- AI2-THOR: interactive indoor rooms where objects such as doors can be used
- SUNCG: synthetic houses with furniture, such as kitchens
- Matterport3D, Gibson and Replica: scans of real buildings, so the views look photorealistic
Note
Training in simulation raises the question of how well the agent works on a real robot, which is the topic of Sim2Real Transfer.