Latent Diffusion Models (LDM)

This model is a combination of VAE + Diffusion Models.

Problem with Standard Diffusion Models

If we have a 1024 x 1024 image with 3 colour channels, that's over 3 million values per image. We need to pass all 3 million values through a massive U-Net 1000 times sequentially to clean noise step by step. This takes up alot of GPU vram.

Solution

Main idea is to perform diffusion inside of a compressed VAE bottleneck.

1. Pre-train a VAE

First, we train a VAE on the dataset so it learns how to compress 1024 x 1024 spectrogram (1024, 1024) into a compact 2D vector representation (64, 64) in the latent space. Since image was 1024 by 1024, we essentially have a 2D spatial latent grid of 64 x 64 x 4.

Once VAE is trained we freeze the weights. It acts as an ultra-fast and efficient compressor.

2. Run Diffusion inside Latent Space

Now, instead of adding Gaussian noise to the full 1024 by 1024 image $x_0$, we pass $x_0$ through the frozen VAE Encoder to get a clean vector embedding $z_0$.

From this latent space embedding of the first input image, $z_0$, we will perform diffusion here, in latent space. We will feed noisy latent $z_t$ and timestep $t$ into the U-Net. The U-Net predicts the noise inside the latent space.

$$\text{Loss}=\Vert {ϵ}_{\text{latent}}-\text{U-Net}(z_t,t)\Vert {}^2$$

3. Generation in Latent Space

To generate a new 1024 by 1024 image from scratch:

1. Start with pure random noise in the tiny $64\times 64$ shape ($z_{1000}$).
2. Run the U-Net for 1,000 steps inside the tiny $64\times 64$ space to get a clean latent $z_0$.
3. ONLY AT THE VERY END, hand $z_0$ to the frozen VAE Decoder. The decoder unzips $z_0$ into the full-scale $1024\times 1024$ image

4. Class-Conditional Embedding

We want to generate a certain class of image. Each class for example Class 1, Class 2 have their own vector embeddings. What we can do is embed a class vector into every denoising step.

5. Classifier-Free Guidance (CFG)

Sometimes the U-Net ignores the class embedding to produce a weak, generic blend of drone signals. Classifier-Free Guidance (CFG) is a mathematical trick during inference/generation to amplify each classes' features.

At each step the U-Net calculates 2 noise predictions:

1. $N_{\text{uncond}}$: Noise prediction without looking at the class label.
2. $N_{\text{cond}}$: Noise prediction with the class label.

It then pushes the generation strongly toward the class features using a guidance scale factor $w$:

$$\text{Final Noise}=N_{\text{uncond}}+w\cdot (N_{\text{cond}}-N_{\text{uncond}})$$

By turning $w>1$ (e.g., $w=7.5$), you force the model to create sharp, unambiguous signatures for that specific drone class.

6. FiLM

This is Feature-wise Linear Modulation (FiLM). We use FiLM to pass feature embeddings and timestamps into the U-Net. Instead of forcing a 1D vector (vector embeddings) into a 2D image, we use the embedding vector to scale and shift the activations inside the U-Net's feature maps using basic linear algebra:

$$\text{FiLM}(x)=\gamma \cdot x+\beta$$

• The class label (e.g., Class 3: DJI Phantom) passes through a small linear layer to generate two parameter vectors: $\gamma$ (gamma, scale) and $\beta$ (beta, shift).
• Inside the U-Net, right after a convolutional layer produces a feature map $x$:
  • $x$ is multiplied element-wise by $\gamma$ (turning up the volume on important drone features)
  • $\beta$ is added to $x$ (shifting the feature baseline)