Kullback-Liebler Divergence (KL Divergence)

Motivation

We need to understand or measure the difference between the true probability distribution and our observed probability distribution.

KL Divergence is a type of statistical distance between 2 probability distributions $P(x)$ which is our true distribution, vs the probability distribution $Q(x)$ which is the approximating model distribution. KL divergence is also known as the information gain or relative entropy.

KL Equation for Guassian

$$KL=0.5\ast ({\mu }^2+{\sigma }^2-1-2ln(\sigma ))$$

• this equation is used for squashing vector embeddings in the latent space into having a $\mu$ of 0 and $\sigma$ of 1
• this is for the Variational Autoencoder (VAE)