Overview

  • here the agent can ask for help or directions while it moves, instead of getting one instruction at the start
  • three examples: language-based assistance, Help, Anna! and Vision-and-Dialog Navigation

Dialog-guided visual navigation lets the agent ask for directions or for help while it navigates. Instead of getting one instruction at the start, the agent and a helper talk during the task.

Three examples

Vision-based navigation with language-based assistance (imitation learning with indirect intervention):

  • the agent is told "Find a towel in the kitchen"
  • when it is lost, it says "I am lost! Help!"
  • a helper replies with a short instruction, such as "Turn 60 degrees right, go forward, turn left"
  • the agent continues and finally reaches the goal

Help, Anna! (visual navigation with natural multimodal assistance):

  • the agent must "Find a mug"
  • when it asks for help, Anna gives a language instruction with an image, such as "Enter the bedroom and turn left immediately. Walk straight to the carpet in the living room."
  • the agent follows the route, can ask again at a later point, and succeeds

Vision-and-Dialog Navigation:

  • a navigator and an oracle talk in natural language, and the oracle can see more of the route than the navigator
  • the navigator gets a hint, such as "The goal room contains a mat"
  • it asks questions such as "Into the hall or the office?" and "Should I go upstairs?"
  • the oracle answers, for example "Left into the hall. Follow it to a living room" and "Yeah, head up the stairs", until the navigator stops in the goal room

Note

Dialog-guided navigation extends the one-instruction setting of Vision-and-Language Navigation.