Overview

  • a navigation task where the goal is an object described in words, such as "go to the small red object"
  • focus: linking words for colour, size and position to what the agent sees
  • methods: Gated-Attention and Guided Feature Transformation (GFT)

Grounded language learning is a navigation task where the goal is an attributed object described in language. It is sometimes called ObjectGoal++ because it extends ObjectGoal with words for colour, size and position.

Task

  • the goal is given in language, for example "go to the small red object", "the target is left of the hair dryer" or "go to any green object"
  • the focus is on language learning: linking words to things the agent sees
  • research often studies generalisation to compositionally novel instances, which are new combinations of known words, such as "small red" when the agent only learned "small blue" and "large red"

Examples

  • in a simulated 3D world, the instruction "blue object next to the green object" must be grounded in the visual scene (Grounded Language Learning in a Simulated 3D World; Understanding Grounded Language Learning Agents)
  • in VizDoom, "Go to the short blue torch" is solved with Gated-Attention architectures, which combine the instruction and the image features by multiplying the image channels with an attention vector computed from the text
  • Guided Feature Transformation (GFT) is a neural language-grounding module for embodied agents, which uses the language to transform the visual features

Note

When the instruction describes a whole route instead of one target, the task becomes Vision-and-Language Navigation.