Overview

  • these metrics measure how well a navigating agent reaches its goal
  • success rate only counts arrivals, and SPL also rewards taking a short path
  • nDTW checks whether the agent followed the route the instruction described, not just where it ended

Navigation evaluation metrics measure how well an embodied agent reaches its goal. Some only count arrivals, some also reward short paths, and some check whether the agent followed the path that an instruction described.

Success rate (SR)

  • the percentage of episodes in which the agent actually reaches the goal
  • "reach" usually means stopping within a small distance of the goal, about 0.2 m

Success weighted by Path Length (SPL)

SPL cares about success and efficiency:

where:

  • is the number of episodes, so the formula averages over episodes
  • is the binary success of episode : 1 if the agent reached the goal, 0 if not
  • is the shortest path length from the start to the goal in episode
  • is the agent's path length, the distance it actually travelled
  • is 1 when the agent took the shortest path, and smaller when it took a longer path

A failed episode scores 0. A successful but wandering episode scores less than 1.

Normalized Dynamic Time Warping (nDTW)

For instruction following, the path matters, not only the endpoint. Two paths can end at the same place but follow very different routes.

  • Dynamic Time Warping lines up each point of the agent's path with the closest matching point of the reference path , in order
  • nDTW turns the total matching distance into a score between 0 and 1
  • an nDTW of 1.000 means the agent's path matches the reference path exactly, and scores near 0.063 mean the two paths are very different

Note

nDTW is used for Vision-and-Language Navigation, where the instruction describes the route.