Overview
- ImageBind puts six kinds of data into one shared space, with images as the hub
- trick: it only needs (image, other data) pairs, because pairs with images are easy to find
- payoff: emergent alignment, so two kinds of data that were never paired, such as audio and text, still end up close
- compare it with LanguageBind, which uses language as the hub
ImageBind is a unified multimodal encoder that embeds six modalities into one joint representation space centred on images: images and videos, text, audio, depth, thermal data and IMU (motion sensor) data.
Paired data between every two modalities is rare. For example, few datasets pair audio with depth. But paired data between images and almost every other modality is common:
- web images with text captions
- depth sensor data with images
- web videos with their sound
- thermal camera data with images
- egocentric (first-person) videos with IMU readings
ImageBind only needs (image, X) pairs for each modality X. Each modality has its own encoder, and each encoder is trained to bring its output close to the matching image embedding.
So every other modality is trained only against images, and no two non-image modalities are ever trained against each other.

Emergent alignment
Because every modality is aligned to images, two modalities that were never paired become aligned too. For example, audio and text end up close to each other without any audio-text pairs. This is called emergent alignment. In the figure, solid lines are naturally aligned pairs and dotted lines are emergent alignment.
Why it helps an MLLM
Well-aligned representations of all modalities in one space make it easier for an LLM to understand them, because one connector can serve all of them.
Related binding methods
- LanguageBind: the same idea, but language is the anchor instead of images
- VATT: video, audio and text go through Transformers; video and audio are aligned first, then text is aligned to the joint video-audio space, using contrastive losses
- CLAP: CLIP-style contrastive learning between audio and text
Note
ImageBind is useful when a new modality has few pairs: the new modality only needs to be bound to an existing hub such as images. The training signal that pulls matching pairs together is Contrastive Learning.