Vision-and-language navigation is the task of following a natural language route instruction through an environment the agent can only see one viewpoint at a time. A typical instruction is "exit the bedroom, turn left down the hall and stop in the kitchen".
What makes it hard
- the instruction is given once at the start, and the agent has to keep track of how much of it is already done
- the observation is partial, so the agent sees one view rather than a map
- the stopping decision is part of the task, because arriving and knowing you arrived are different
Recurrent agent shape
- a context encoder reads the instruction text
- a visual encoder, historically a CNN, reads the frame at each time step
- a recurrent action policy carries state across time steps and emits one action per step
- attention over the instruction lets the agent look at the clause it is currently executing
Why attention matters here
- not every part of an input is important given the context, so the model learns which words and which image regions to weight
- the same grounding ability answers questions about a scene, such as which colour is illuminated on a traffic light
The mechanism that does the weighting is described in Self-Attention.