Overview

  • two basic tasks that tell an agent where it is
  • visual odometry tracks how the agent moved over time, either with a hand-built pipeline or end-to-end (DeepVO)
  • localization finds the agent on a known map, as a belief that narrows down as it sees more

Visual odometry and localization are two basic tasks that tell an embodied agent where it is. Visual odometry tracks how the agent has moved. Localization finds the agent's position in a known map.

Visual odometry

Visual odometry estimates the change in pose over time using visual perception. Pose means the position and the orientation of the agent.

Conventional method, a pipeline of hand-designed steps:

  1. take the image sequence
  2. feature detection: find distinctive points in each image
  3. feature matching (tracking): follow the same points across frames, with outlier rejection to drop wrong matches
  4. motion estimation: compute how the camera moved, using camera calibration
  5. bundle adjustment: refine all poses together, with scale estimation
  6. output the poses

End-to-end method (for example DeepVO):

  • the video goes through a CNN that extracts features from pairs of consecutive frames
  • an RNN carries information across time
  • the network outputs the pose directly at each step

Related work: DeepVO (end-to-end visual odometry with deep recurrent convolutional networks) and VINet (visual-inertial odometry as a sequence-to-sequence learning problem, which also uses motion sensor data).

Localization

Localization estimates where the agent is inside a map.

  • the agent keeps a belief, which is a probability over all positions in the map
  • at the start () the belief is spread over many places
  • as the agent moves and sees more, the belief narrows down to the ground truth position

Related work: Active Neural Localization, and Global Pose Estimation with an Attention-based Recurrent Network.

Note

Localization is one half of the problem in Visual Navigation. The other half is mapping.