ELBO stands for evidence lower bound. It is the quantity a VAE actually maximises, because the thing we really want is impossible to compute directly.
Why the posterior is intractable
- we want the posterior , which is the distribution over latent codes given an image
- its denominator is obtained by marginalising out the latent variables, and that denominator is called the evidence
- for a nonlinear likelihood such as a neural network, that integral cannot be computed
- so the exact posterior is out of reach
Variational inference
- variational inference approximates the intractable posterior with a simpler tractable distribution
- it turns inference into optimization, because we now tune to sit as close as possible to the true posterior
- closeness is measured by the Kullback-Liebler Divergence (KL Divergence)
Why we maximise ELBO instead
- the KL divergence between and the true posterior still contains the intractable evidence term
- by Jensen's inequality, the KL divergence is always greater than or equal to zero
- that makes ELBO a lower bound on the evidence, and never an upper bound
- minimising the KL divergence is therefore equivalent to maximising the ELBO
- ELBO can be computed and differentiated, so it is what we optimise
Note
- ELBO is a lower bound on the evidence
- no datapoint shares its latent embedding with another, which is exactly why the loss decomposes into one term per datapoint