Overview
- here the agent can ask for help or directions while it moves, instead of getting one instruction at the start
- three examples: language-based assistance, Help, Anna! and Vision-and-Dialog Navigation
Dialog-guided visual navigation lets the agent ask for directions or for help while it navigates. Instead of getting one instruction at the start, the agent and a helper talk during the task.
Three examples
Vision-based navigation with language-based assistance (imitation learning with indirect intervention):

- the agent is told "Find a towel in the kitchen"
- when it is lost, it says "I am lost! Help!"
- a helper replies with a short instruction, such as "Turn 60 degrees right, go forward, turn left"
- the agent continues and finally reaches the goal
Help, Anna! (visual navigation with natural multimodal assistance):

- the agent must "Find a mug"
- when it asks for help, Anna gives a language instruction with an image, such as "Enter the bedroom and turn left immediately. Walk straight to the carpet in the living room."
- the agent follows the route, can ask again at a later point, and succeeds
Vision-and-Dialog Navigation:

- a navigator and an oracle talk in natural language, and the oracle can see more of the route than the navigator
- the navigator gets a hint, such as "The goal room contains a mat"
- it asks questions such as "Into the hall or the office?" and "Should I go upstairs?"
- the oracle answers, for example "Left into the hall. Follow it to a living room" and "Yeah, head up the stairs", until the navigator stops in the goal room
Note
Dialog-guided navigation extends the one-instruction setting of Vision-and-Language Navigation.