Standard Diffusion Models

This is often called Denoising Diffusion Probabilistic Models (DDPM). Standard Diffusion Models contain 2 different processes:

1. Forward Process (Add Noise)

• we first start with a full clear image
• next we slowly corrupt it by added small amounts of gaussian noise over $T$ steps (normally its $T=1000$)
• at time step $t$, a small amount of noise is added according to a fixed variance schedule (${\beta }_1,{\beta }_2,\dots ,{\beta }_T$)
• Instead of calculating how much noise to add at each step 1000 times over, we can use a direct closed form formula that allows us to calculate the final output at any time step:

$$x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$$

(where $ϵ\sim \mathcal{N}(0,I)$ is pure random noise, and ${\bar{\alpha }}_t$ is a predefined scaling parameter calculated from the noise schedule $\beta$)

2. Reverse Process (Denoise)

Starting with pure static noise at time step $x_{1000}$, we can go back to $x_0$ using a U-Net.

3 Training Process

1. Pick a clean real image $x_0$ from your dataset.
2. Pick a random timestep $t$ between $1$ and $1000$.
3. Sample a random noise vector $ϵ\sim \mathcal{N}(0,I)$.
4. Use the closed-form shortcut formula to generate the noisy image $x_t$.
5. Feed $x_t$ and $t$ into the U-Net.
6. The U-Net predicts the noise: ${ϵ}_{\text{pred}}=\text{U-Net}(x_t,t)$.
7. Calculate the loss using standard Mean Squared Error (MSE):

$$\text{Loss}=\Vert ϵ-{ϵ}_{\text{pred}}\Vert {}^2$$

4 Inference Process

To generate a brand-new image from scratch after training:

1. Start with pure random Gaussian static noise $x_{1000}$.
2. Run a loop backward from $t=1000$ down to $t=1$:
   • Pass $x_t$ and timestep $t$ into the trained U-Net to predict noise ${ϵ}_{\text{pred}}$.
   • Subtract a fraction of that predicted noise from $x_t$ to get a slightly cleaner image $x_{t-1}$.
   • Add a tiny bit of random jitter (to preserve diversity)
3. At step $t=0$, a brand new $x_0$ image appears

5 Notes

• one network handles every timestep, so a diffusion model does not need a separately trained network for each one
• the timestep $t$ is passed to the denoiser as an additional input, and it is positionally encoded
• the network is trained to predict the noise, and not the clean image directly
• a basic approach runs roughly 1000 denoising steps
• during sampling, after removing a little noise the standard DDPM update also adds back a bit of noise, which keeps diversity, and some variants skip this
• training uses the easy direction, so cats to noise, because that direction is simple to simulate while noise to cat is hard
• in the reverse process the model moves random noise gradually onto the manifold of real images

The score function behind all of this is in Tweedie's Formula.