Latent Sementic Analysis (LSA)

Aim

LSA aims to uncover the underlying semantic structure of text data and transform it into a lower-dimensional space. This is something like an autoencoder, where we reduce the input's dimensionality, and then expand it back out again.

Introduction

LSA works by generating a document-term matrix, which is usually populated with TF-IDF scores rather than raw counts, since TF-IDF gives a better representation of the significance of a word. The matrix is then decomposed, typically with SVD, into sub-matrices. Like LDA, LSA requires the user to set the number of topics $k$.

Algorithm

Main Idea

Factorise the document-term matrix into 3 smaller matrices, then keep only the $k$ largest components.

1. Generate a document-term matrix of shape $m\times n$ using TF-IDF

• $m$ is the number of documents and $n$ is the number of terms
• the entries can be raw counts, but TF-IDF scores are usually used

	$T_1$	$T_2$	$T_3$	$...$	$T_n$
$D_1$	$0.2$	$0.1$	$0.3$	$...$	$0.5$
$D_2$	$0.1$	$0.3$	$0.2$	$...$	$0.4$
$...$	$...$	$...$	$...$	$...$	$...$
$D_m$	0.3	0.5	0.3	$...$	$0.3$

2. Apply Singular Value Decomposition (SVD) to break the matrix down

$$A=U_k\cdot S\cdot V_k^T$$

Matrix	Shape	Name	Meaning
$A$	$m\times n$	Term-Document Matrix	the original matrix we computed above
$U_k$	$m\times k$	Document-Topic Matrix	document vectors expressed in terms of topics
$S$	$k\times k$	Topic Importance	diagonal matrix of singular values, so it ranks how strong each topic is
$V_k^T$	$k\times n$	Term-Topic Matrix	term vectors expressed in terms of topics

• keeping only the $k$ largest singular values is what performs the dimensionality reduction, so $n$ term columns collapse into $k$ topic columns
• the singular values in $S$ are sorted from largest to smallest, so topic 0 explains the most variance

LSA in gensim

Using the same COVID dataset and the same preprocessing as LDA, up to data_words.

1. Stem words to reduce noise

from nltk.stem.porter import PorterStemmer
p_stemmer = PorterStemmer()
 
def stem_words(texts):
    return [[p_stemmer.stem(word) for word in simple_preprocess(str(doc))
             ] for doc in texts]

2. Build the document-term matrix

doc_term_matrix = [id2word.doc2bow(twt) for twt in data_words]

3. Build the LSA model and extract the top 10 keywords

from gensim.models import LsiModel
 
lsa_model = LsiModel(doc_term_matrix, num_topics=num_topics, id2word=id2word)
print(lsa_model.print_topics(num_topics=num_topics, num_words=10))

Output, showing the first 2 of the 7 topics:

[(0, '0.600*"covid" + 0.406*"coronavirus" + 0.177*"supermarket" + 0.145*"consumer" + ' '0.136*"amp" + 0.128*"stay" + 0.125*"food" + 0.120*"prices" + 0.112*"turkey" + ' '0.103*"online"'),

(1, '0.427*"amp" + 0.350*"prices" + -0.342*"coronavirus" + -0.162*"turkey" + ' '0.156*"rs" + 0.129*"home" + 0.127*"oil" + 0.124*"output" + 0.118*"stay" + ' '0.104*"price"'),

(2, '0.455*"coronavirus" + -0.442*"covid" + 0.240*"amp" + -0.188*"consumer" + ' '0.182*"please" + 0.133*"store" + 0.101*"masks" + 0.094*"let" + 0.087*"stay" + ' '-0.082*"online"'),

(3, '0.289*"stay" + -0.266*"prices" + -0.243*"coronavirus" + -0.199*"turkey" + ' '-0.187*"output" + 0.185*"home" + 0.157*"store" + 0.151*"going" + ' '0.144*"consumer" + 0.131*"let"'),

(4, '-0.406*"amp" + 0.217*"stay" + 0.207*"prices" + -0.199*"supermarket" + ' '0.187*"output" + 0.186*"says" + 0.172*"store" + 0.134*"due" + 0.124*"cut" + ' '0.120*"let"'),

(5, '0.374*"supermarket" + -0.319*"consumer" + -0.284*"online" + 0.194*"social" + ' '-0.138*"amp" + 0.136*"would" + -0.132*"shopping" + 0.129*"covid" + ' '0.104*"nhs" + 0.102*"going"'),

(6, '0.371*"consumer" + -0.250*"online" + -0.226*"turkey" + -0.152*"covid" + ' '-0.141*"stay" + -0.136*"shopping" + 0.128*"pandemic" + 0.119*"due" + ' '0.113*"output" + 0.108*"food"')]

How to read the output:

• the first number 0, 1, 2, ... represents the topic_id
• there is no topic name, just topic id, naming the topic is our job
• each coefficient in each topic is one entry in $V_k^T$ (term-topic matrix)
• the coefficients are SVD loadings, not probabilities, so they do not sum to 1
• weights can be negative, as with -0.342*"coronavirus" in topic 1, which means the word pushes a document away from that topic
• this is the main reason LSA is harder to interpret than LDA, since a negative weight has no plain meaning
• we set num_words=10 means we only see the top 10 words for each topic
• num_topics is the $k$ that we decide at the start to set the number of singular values to keep
• the class is called LsiModel because LSA is also known as Latent Semantic Indexing

Advantages and Limitations

Advantages	Limitations
Dimensionality reduction: LSA reduces the dimensionality of the data by representing documents as mixtures of topics, which simplifies the data representation	Lack of interpretability: the reduced semantic space lacks human interpretability, so it captures semantic relationships without giving direct insight into the meaning of specific dimensions
Semantic understanding: LSA captures the underlying semantic structure of text, so it can identify and measure semantic similarities between words and documents even when they share no exact word overlap	Preprocessing sensitivity: LSA is highly sensitive to the preprocessing used on the data, such as stop word removal and stemming

Understanding the output

Applying SVD on the document-term matrix gives us 3 matrices.

what you want to do	matrix you use
compare documents, cluster them, classify them, retrieve them	$U_{k}S$, the document-topic matrix
read what each topic means, or compare words to each other	$V_k^T$, the term-topic matrix
know which topics matter most	$S$, the singular values

The decomposition step that LSA relies on is Singular Value Decomposition (SVD).