Overview
- a navigation task where the goal is an object described in words, such as "go to the small red object"
- focus: linking words for colour, size and position to what the agent sees
- methods: Gated-Attention and Guided Feature Transformation (GFT)
Grounded language learning is a navigation task where the goal is an attributed object described in language. It is sometimes called ObjectGoal++ because it extends ObjectGoal with words for colour, size and position.
Task
- the goal is given in language, for example "go to the small red object", "the target is left of the hair dryer" or "go to any green object"
- the focus is on language learning: linking words to things the agent sees
- research often studies generalisation to compositionally novel instances, which are new combinations of known words, such as "small red" when the agent only learned "small blue" and "large red"
Examples
- in a simulated 3D world, the instruction "blue object next to the green object" must be grounded in the visual scene (Grounded Language Learning in a Simulated 3D World; Understanding Grounded Language Learning Agents)
- in VizDoom, "Go to the short blue torch" is solved with Gated-Attention architectures, which combine the instruction and the image features by multiplying the image channels with an attention vector computed from the text
- Guided Feature Transformation (GFT) is a neural language-grounding module for embodied agents, which uses the language to transform the visual features
Note
When the instruction describes a whole route instead of one target, the task becomes Vision-and-Language Navigation.