Overview
- two basic tasks that tell an agent where it is
- visual odometry tracks how the agent moved over time, either with a hand-built pipeline or end-to-end (DeepVO)
- localization finds the agent on a known map, as a belief that narrows down as it sees more
Visual odometry and localization are two basic tasks that tell an embodied agent where it is. Visual odometry tracks how the agent has moved. Localization finds the agent's position in a known map.
Visual odometry
Visual odometry estimates the change in pose over time using visual perception. Pose means the position and the orientation of the agent.

Conventional method, a pipeline of hand-designed steps:
- take the image sequence
- feature detection: find distinctive points in each image
- feature matching (tracking): follow the same points across frames, with outlier rejection to drop wrong matches
- motion estimation: compute how the camera moved, using camera calibration
- bundle adjustment: refine all poses together, with scale estimation
- output the poses
End-to-end method (for example DeepVO):
- the video goes through a CNN that extracts features from pairs of consecutive frames
- an RNN carries information across time
- the network outputs the pose directly at each step
Related work: DeepVO (end-to-end visual odometry with deep recurrent convolutional networks) and VINet (visual-inertial odometry as a sequence-to-sequence learning problem, which also uses motion sensor data).
Localization
Localization estimates where the agent is inside a map.

- the agent keeps a belief, which is a probability over all positions in the map
- at the start () the belief is spread over many places
- as the agent moves and sees more, the belief narrows down to the ground truth position
Related work: Active Neural Localization, and Global Pose Estimation with an Attention-based Recurrent Network.
Note
Localization is one half of the problem in Visual Navigation. The other half is mapping.