Probabilistic Latent Semantic Analysis (pLSA)

Introduction

Find a model that can generate the data observed within a term-document matrix. In otherwords, we want a function $P(w,d)$ that generates the corresponding entry in the word-document matrix given the word $w$ and document $d$.

$$P(w,d)=P(d)P(w|t)P(t|d)$$

LSA vs pLSA

	LSA	pLSA
Basis	easily scalable SVD based decomposition, relatively straightforward linear algebra technique	less scalable, higher complexity Probabilistic modelling, which views the generation of a document as a probabilistic mixture of topics, so pLSA is less scalable than LSA due to its higher complexity
Interpretability	less interpretable can reveal semantic relationships between words and documents, but it may not provide human-interpretable topics, since it does not explicitly model topics in a probabilistic way	more interpretable it explicitly defines topics as probability distributions over words, which lets users interpret topics from their word distributions
Core equation	$A=U_k\cdot S\cdot V_k^T$	$P(w,d)=P(d){\sum }_{t}P(w\mid t)P(t\mid d)$
Input	a document-term matrix, usually filled with TF-IDF scores	the same term-document matrix, treated as observed counts to be generated
What it produces	$U_k$ for documents, $S$ for topic importance, and $V_k^T$ for terms	$P(w\mid t)$ and $P(t\mid d)$
Value range of the output	any real number, so weights can be negative	probabilities between 0 and 1, which sum to 1
How it is solved	closed form, from one SVD computation	iteratively, with expectation maximisation
Determinism	deterministic, so the same input gives the same output	depends on initialisation, so results vary between runs
Topic ordering	sorted by singular value, so topic 0 is the strongest	no inherent ordering

pLSA gives LSA a probabilistic form, and the model that goes further by adding a prior is Latent Dirichlet Allocation (LDA).