Overview
- EQA asks a question the agent can only answer after moving to find the answer
- so it joins navigation with visual question answering
- two datasets: the first one built on SUNCG, and a later one built on Matterport3D with point clouds
Embodied Question Answering (EQA) asks an agent a question about its environment that it can only answer by moving around to find the answer. The answer is not visible from the start position.
Examples

- question: "What color is the car?"
- the agent moves forward, forward again, turns left and turns left again until it sees the car
- answer: "Orange!"
Another example: "What is to the left of the shower?" The agent must navigate to the bathroom, find the shower and answer "A cabinet!"
So EQA joins navigation (finding the right place) with visual question answering (answering about what it sees).
EQA in SUNCG
The first EQA work used SUNCG houses, which are synthetic 3D homes. The agent sees RGB, depth, semantic segmentation and a top-down map. Later, the SUNCG data became the subject of a legal dispute: the company Planner5D claimed the 3D files were taken from it and given to Facebook's AI researchers, so SUNCG stopped being available.
EQA Matterport dataset
A newer version, EQA in photorealistic environments with point cloud perception, built its dataset from Matterport3D scans:
- it uses human colour names for objects
- goal locations are carefully selected so that the target of the question is visible
- it generates point clouds from the scanner data, not from the mesh, so the agent perceives the scene as 3D points
Note
EQA needs the agent to remember what it has seen, which is the topic of Environment Representation for Navigation.