A plain diffusion model generates whatever it likes. Conditioning and guidance are how we tell it what to generate. There are 4 main ideas:
1. Class conditioning with a one-hot label
- the class label is passed to the denoiser as an extra input
- the limitation is that it only allows generation of a fixed set of classes, so anything outside the label list cannot be asked for
2. Text conditioning
- a text encoder turns the prompt into an embedding vector
- that embedding is fed to the diffusion network alongside the noisy image
- this removes the fixed class list, because any sentence can be encoded
3. Classifier guidance
- a separate image classifier pushes the sample towards the wanted class during sampling
- the practical problem is that a normal classifier has only seen clean images, and sampling works on noisy ones
- the fix is to fine-tune the classifier on noisy images
4. Classifier-free guidance
- the same denoiser is run twice, once with the condition and once without it
- the two noise predictions are mixed, so no separate classifier is needed at all
- raising the guidance weight pushes samples harder towards the condition, at the cost of diversity
Note
- classifier-free guidance does not need a classifier trained on noisy images, that is classifier guidance
The mixing formula and the weight are written out in Latent Diffusion Models (LDM).