Overview

  • long videos make far too many tokens, so the model must keep only what matters
  • two approaches: online memory (BLIP-3-Video squeezes a video into 32 to 128 tokens) and offline memory (GenS picks the right frames)
  • GenS generates the relevant frame spans, in place of plain CLIP similarity search

Long video memory is the problem of letting an MLLM understand long videos without running out of context. A long video becomes hundreds of thousands of tokens, so the model needs a way to keep only what matters. There are two approaches: online memory and offline memory.

Online memory: BLIP-3-Video

  • long videos turn into hundreds of thousands of tokens if every frame is encoded
  • a temporal encoder reads the per-frame image features and summarises the whole video into only 32 to 128 tokens
  • comparable models use about 4,608 tokens, against 32 tokens for BLIP-3-Video
  • the LLM then reads these few tokens with the text question

Offline memory: GenS

  • the usual method retrieves frames with CLIP similarity to the question, but this misses temporal links and multi-hop reasoning (questions that need several moments of the video)
  • GenS (Generative Frame Sampler) uses a VideoLLM that generates the relevant frame spans, each with a confidence score
  • it is trained on the GenS-Video-150K dataset, with an average video length of 647.5 seconds
  • VideoQA accuracy improves as more frames are available to choose from

Note

Memory is one of the three capabilities a multimodal agent needs, together with grounding and reasoning.