Tweedie's formula is the link between the noise a diffusion model predicts and the score function of the noisy distribution. It is the reason denoising can be described as a gradient step.
Score function
- the score function is the gradient of the log density of the noisy distribution
- it points in the direction where the data becomes more likely, so following it moves a noisy sample towards the data manifold
What Tweedie's formula says
- by Tweedie's formula, the predicted noise is approximately the negative score function scaled by sigma
- so a network trained to predict noise has, without being told, learned the score
- one denoising step is therefore a gradient step that walks the sample towards higher density
Note
- denoising can be written as a gradient step using the score function of the noisy distribution
- the one step estimate of the clean image can be recovered from the noise prediction, and it is the expected clean image given the noisy input
The network that produces the noise prediction is described in Standard Diffusion Models.