Latent Dirichlet Allocation (LDA)

Aim

LDA aims to find the representative words for a topic and classify a document based on the occurances of these words.

Introduction

LDA has 2 parts:

1. words belonging to a document
2. words belonging to a topic

LDA uses the Bag of Words (BoW) approach where the order and sementic of a word is not considered. LDA requires the user to set the predefined $k$ number of topics.

Algorithm

Main Idea

Go through each document in the corpus and randomly assign each word in the document to one of $k$ topics.

For each document $d$, go through each word $w$ and find:

• p(topic t|document d) which is the probability of topic $t$ given a document $d$
• what this represents is the proportion of words found in document $d$ that are labelled/assigned the topic $t$
• p(word w|topic t) is the probability of getting the word $w$ when we look into a topic $t$
• another way to see this is:

$$p(w|t)=\frac{\text{the number of word w occurances that are assigned to topic t}}{\text{total number of word occurances assigned to topic t}}$$

Therefore, we can get the final probability of word $w$ belonging to topic $t$:

$$\text{p(word w belongs to topic t) = p(topic t|document d)}\cdot \text{p(word w|topic t)}$$

LDA in gensim

Dataset: COVID tweets, from https://www.kaggle.com/datasets/datatattle/covid-19-nlp-text-classification

1. Remove punctuation and lowercase

# Load the regular expression library
import re
 
# Remove punctuation
tweets['OriginalTweet_processed'] = \
tweets['OriginalTweet'].map(lambda x: re.sub('[@#,\.!?]', '', x))
 
# Convert the tweets to lowercase
tweets['OriginalTweet_processed'] = \
tweets['OriginalTweet_processed'].map(lambda x: x.lower())
 
# Print out the first rows of tweets
tweets['OriginalTweet_processed'].head()

2. Tokenise and remove stop words

import gensim
from gensim.utils import simple_preprocess
import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords
 
stop_words = stopwords.words('english')
stop_words.extend(['https', 'tco'])
 
def sent_to_words(sentences):
    for sentence in sentences:
        # deacc=True removes punctuations
        yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))
 
def remove_stopwords(texts):
    return [[word for word in simple_preprocess(str(doc))
             if word not in stop_words] for doc in texts]
 
data = tweets.OriginalTweet_processed.values.tolist()
data_words = list(sent_to_words(data))
 
# remove stop words
data_words = remove_stopwords(data_words)
 
print(data_words[:1][0][:30])

Note: 'https' and 'tco' are parts of shared hyperlinks within tweets, so they are added to the stop word list.

3. Build the dictionary and the bag of words corpus

import gensim.corpora as corpora
 
# Create Dictionary
id2word = corpora.Dictionary(data_words)
 
# Create Corpus
texts = data_words
 
# Term Document Frequency
corpus = [id2word.doc2bow(text) for text in texts]
 
# View
print(corpus[:1][0][:30])

Output:

[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1), (10, 1)]

4. Train the LDA model

from pprint import pprint
 
# number of topics
num_topics = 7
 
# Build LDA model
lda_model = gensim.models.LdaMulticore(corpus=corpus,
                                       id2word=id2word,
                                       num_topics=num_topics)
 
# Print the Keyword in the 10 topics
pprint(lda_model.print_topics())
 
doc_lda = lda_model[corpus]

Output, showing the first 2 of the 7 topics:

[(0,
  '0.017*"coronavirus" + 0.013*"covid" + 0.012*"consumer" + 0.009*"amp" + '
  '0.007*"supermarket" + 0.007*"turkey" + 0.006*"products" + 0.005*"would" + '
  '0.005*"care" + 0.005*"prices"'),
 (1,
  '0.014*"covid" + 0.012*"food" + 0.009*"demand" + 0.007*"coronavirus" + '
  '0.007*"due" + 0.006*"prices" + 0.006*"pandemic" + 0.005*"soars" + '
  '0.005*"customers" + 0.005*"social"')]

Reading the output

• each topic prints as a weighted word list, and the coefficients are $p(w\mid t)$
• so 0.017 for "coronavirus" in topic 0 means 1.7 percent of the word occurrences assigned to topic 0 are "coronavirus"
• the topics arrive as numbers 0 to 6 with no names, so you read the word lists and name them yourself
• doc_lda = lda_model[corpus] gives the topic mixture $p(t\mid d)$ for every document
• num_topics = 7 is the $k$ used in this example, and you choose it yourself

Notes on the pipeline

• doc2bow converts each document into a list of (word_id, count) pairs, so (0, 1) means word id 0 appears once
• id2word maps those ids back to the actual words, which is why the model needs it to print readable topics
• LdaMulticore has no fixed random seed by default, so the topic numbering and contents change between runs
• the visualisation below comes from the pyLDAvis library, which is a separate tool

Visualization

Advantages and Limitations

Advantages	Limitations
Dimensionality reduction: LDA reduces the dimensionality of the data by representing documents as mixtures of topics, which simplifies the data representation	No semantics: LDA treats words independently of their context in a document, so the semantics of a word are not captured
Interpretability: the topics generated by LDA are represented as lists of words, which makes them interpretable	Preprocessing sensitivity: LDA is highly sensitive to the preprocessing used on the data, such as stop word removal and stemming