Overview
- VLN means following a full route instruction, such as "exit the bedroom, turn left and stop in the kitchen"
- hard parts: tracking how much of the instruction is done, seeing one view at a time, and knowing when to stop
- three settings: VLN (Matterport3D), StreetLearn and Touchdown, and LANI
- it ends with the Matterport3D simulator, nDTW scoring, and the shape of a recurrent VLN agent
Vision-and-language navigation is the task of following a natural language route instruction through an environment the agent can only see one viewpoint at a time. A typical instruction is "exit the bedroom, turn left down the hall and stop in the kitchen".
What makes it hard
- the instruction is given once at the start, and the agent has to keep track of how much of it is already done
- the observation is partial, so the agent sees one view rather than a map
- the stopping decision is part of the task, because arriving and knowing you arrived are different
Instruction-guided visual navigation
VLN is one setting of instruction-guided visual navigation. The agent gets a route instruction, such as "Leave the bedroom, and enter the kitchen. Walk forward, and take a left at the couch. Stop in front of the window."
Three major settings:
- Vision-and-Language Navigation (VLN): indoor environments from the Matterport3D dataset plus human directions (Interpreting visually-grounded navigation instructions in real environments)
- StreetLearn: Google Street View plus Google Maps directions; the related Touchdown task adds natural-language navigation and spatial reasoning in street scenes, for example "Head northwest on W 39th St toward 8th Ave, turn right at the 1st cross street onto 8th Ave, turn left onto W 47th St"
- LANI: a simulated quadcopter in an open environment with landmark objects, for example "Go around the pillar on the right hand side and head towards the boat, circling around it clockwise"; work on LANI maps instructions to actions with visual goal prediction and to continuous control actions with position-visitation prediction
Matterport3D simulator
- based on the Matterport3D dataset (Chang et al. 2017)
- contains 10,800 panoramic images from 90 buildings
- has high visual diversity
- feasible trajectories are set by a navigation graph: the agent moves between fixed viewpoints connected by edges, instead of moving freely

The human directions were collected by showing a person a route in the panoramic viewer and asking them to write directions for a smart robot. A public leaderboard for VLN is hosted on EvalAI, and it reports path length, navigation error, oracle success, success and SPL. The human baseline has a success rate of 0.86 and an SPL of 0.76.
Path matters
Success rate and SPL only check where the agent stops. For instruction following, the agent should also follow the described route, so VLN is also scored with normalized Dynamic Time Warping (nDTW), see Navigation Evaluation Metrics.
Recurrent agent shape
- a context encoder reads the instruction text
- a visual encoder, historically a CNN, reads the frame at each time step
- a recurrent action policy carries state across time steps and emits one action per step
- attention over the instruction lets the agent look at the clause it is currently executing
Why attention matters here
- not every part of an input is important given the context, so the model learns which words and which image regions to weight
- the same grounding ability answers questions about a scene, such as which colour is illuminated on a traffic light
The mechanism that does the weighting is described in Self-Attention.