Video diffusion extends image diffusion to a sequence of frames. Frame by frame image diffusion is not enough, because each frame would be generated independently and the video would flicker.
Two named challenges
- temporal consistency, so the same object stays the same across frames
- autoregressive generation of arbitrary length, so a clip is not limited to a fixed number of frames
How we can overcome this
- the model is built around a spatial-temporal VAE, which compresses across space and across time
- diffusion then runs inside that latent space
- generation continues autoregressively, so a long clip is produced piece by piece
Diffusion is not only for images
Here are a few named applications:
- text to video generation
- image editing and composition
- visual illusions and anagrams
- video game simulation and world models
- diffusion policy, which generates robot actions rather than pixels
The image version of the same process is in Standard Diffusion Models.
Note
Two more application families sit beside the list above.
- fine-grained control, where an extra control signal such as an edge map or a pose is added to a text to image model so the layout is steered and not only the content
- video editing, where a region of an existing clip is animated, swapped, moved or extended rather than generated from scratch