In a modular robot pipeline, perception is the stage that turns raw sensor data into symbols that a planner can use. The planner (often an LLM) cannot read a waveform or a pixel grid directly. It needs text it can reason over, and object positions it can act on.

Why perception is a separate stage

An end-to-end VLA takes the camera image and the instruction and outputs an action in one model. A modular pipeline splits this into steps, and each step has a clear input and output:

  • each model can be swapped or tested on its own
  • a failure can be traced to one stage, for example a missed detection or a wrong transcript
  • the cost is extra latency, because every stage adds its own inference time

The 2 perception models

ModalityModelInputOutput
AudioWhisper large-v2spoken command as a waveformtext transcript
VisionOWLv2camera image and text queriesbounding boxes with scores
  1. Whisper large-v2
    • an encoder-decoder speech recognition model that converts a spoken command into text
  2. OWLv2
    • an open-vocabulary object detector that finds objects named in free text, without a fixed class list

How they connect

graph TD
    MIC["Microphone"] --> ASR["Whisper large-v2"]
    ASR --> TXT["Text command"]
    TXT --> LLM["LLM Planner"]
    LLM --> Q["Object queries"]
    CAM["Camera image"] --> DET["OWLv2"]
    Q --> DET
    DET --> BOX["Bounding boxes"]
    BOX --> POSE["3D position from depth"]
    POSE --> ROBOT["Pick and place primitive"]

The two models meet at the planner. Whisper gives the planner the task in words, and the planner gives OWLv2 the object names to search for. For example, the command put the red cube in the blue bowl becomes the queries a red cube and a blue bowl.

Open-vocabulary detection is what makes this chain work. A closed-set detector trained on a fixed list of classes cannot search for an object that the planner names at run time.