Overview
- visual navigation means getting from A to B using only the agent's own camera, then saying STOP
- classic approach: mapping plus localization, solved together as SLAM
- three goal types: PointGoal, ObjectGoal and AreaGoal, plus four levels of how much the agent saw the place before
- it ends with why this is a learning problem and what is still open
Visual navigation is the task of moving from A to B in an environment E using egocentric perception. The agent only sees what its own camera sees, and it must decide when it has arrived and say STOP.

The definition leaves four questions: how is the goal specified, how familiar is the environment, how is navigation evaluated, and why is this a learning problem.
Mapping and localization
Classic navigation joins two problems:
- mapping: building a map of the environment from what the agent observes
- localization: estimating where the agent is in that map

These two problems depend on each other, like the chicken and the egg. A good map needs a known position, and a known position needs a good map. Solving both together is Simultaneous Localization and Mapping (SLAM), see Simultaneous Localisation and Mapping (SLAM).

Goal specification
- PointGoal: go to a point given as coordinates relative to the start, for example
- ObjectGoal: go to an object, given as a label (for example "lamp") or as an image of the object
- AreaGoal: go to an area, given as a label (for example "kitchen") or as an image of the area
Environment exposure before the task is given
From least to most exposure:
- no prior exploration: the environment has never been seen; this matches a temporary environment or an important task, and it is the main setting today
- pre-recorded exploration: the environment was seen but not acted in, for example from teleoperation or from another agent's experience
- fixed-time budget for exploration: the agent may explore for a limited time first; this matches a new environment where many tasks are expected
- unlimited access: the agent may do anything in the test environment, like an agent that lives in your home; most Atari and MuJoCo RL results are in this setting, and so is any paper that evaluates new tasks in its training environments
Evaluation
The two main metrics are success rate and SPL, described in Navigation Evaluation Metrics.
Why is this a learning problem
Robotics has done PointGoal navigation for a long time, and SLAM plus planning systems work. Studies that compare the two approaches found:
- learned navigation is more robust
- classic navigation is more performant (in some settings)
- both are below human level
For ObjectGoal and AreaGoal, learning helps more, because the world has strong regularities that can be learned. A pot is probably in the kitchen. To go from the bathroom to the back lawn, the agent should first head for an exit. Learned agents can use these patterns to navigate more efficiently.
Open problems
PointGoal navigation with RGB-D and GPS and compass sensors is essentially solved. Open questions remain:
- assumptions: drop unrealistic sensors such as a perfect GPS and compass
- sample efficiency: reaching 0.96 SPL takes about 80 years of experience in simulation
- ObjectGoal and AreaGoal are still fairly open
- how to trade off exploration against path length when there are several targets
- how best to represent the environment for navigation
- how robust learned agents are to domain shift
- how learned and classic approaches can help each other
- how trained PointGoal agents can be used as parts of other models