Singular Value Decomposition (SVD)

Aim

SVD aims to factorise a given matrix into three separate matrices to uncover hidden patterns and structure within the data.

Introduction

Given an input matrix (a term-document matrix for NLP), SVD decomposes the matrix into 3 separate matrices:

Matrix	Name	Meaning
$U$	left singular vectors matrix	represents relationships between rows in the original data, which often represents the relationship between terms and topics
$\Sigma$	diagonal matrix of singular values	captures the importance of each singular vector, where higher values indicate more significance
$V^T$	right singular vectors matrix	represents relationships between columns in the original data, which often represents the relationship between documents and topics

k-SVD

Main Idea

k-SVD is a variation of SVD and it only takes into account the $k$ largest singular values to approximate the data.

$$D=U\cdot S\cdot V^T$$

• only the first $k$ columns of $U$ are kept
• only the top-left $k\times k$ block of $S$ is kept
• only the first $k$ rows of $V^T$ are kept
• the discarded blocks are what produces the dimensionality reduction, so the result is an approximation rather than an exact equality

SVD in Scikit-Learn

Using a truncated SVD function.

1. Build the input matrix with TF-IDF

vect = TfidfVectorizer(stop_words=stop_words, smooth_idf=True)
input_matrix = vect.fit_transform(tweets.OriginalTweet).todense()
input_matrix = np.asarray(input_matrix)

2. Fit truncated SVD

from sklearn.decomposition import TruncatedSVD
 
svd_modeling = TruncatedSVD(n_components=4, algorithm='randomized', n_iter=100, random_state=122)
svd_modeling.fit(input_matrix)
 
components = svd_modeling.components_
vocab = vect.get_feature_names_out()

• n_components=4 is the $k$ in k-SVD
• algorithm='randomized' uses randomised SVD, which avoids computing the full decomposition
• n_iter=100 is the number of iterations for that randomised solver
• random_state=122 fixes the seed, which is needed because the randomised algorithm is not deterministic
• components_ has shape $k\times n$, so rows are components and columns are terms

3. Function to obtain the various topics from the list

topic_word_list = []
def get_topics(components):
    for i, comp in enumerate(components):
        terms_comp = zip(vocab, comp)
        sorted_terms = sorted(terms_comp, key=lambda x: x[1], reverse=True)[:7]
        topic = " "
        for t in sorted_terms:
            topic = topic + ' ' + t[0]
        topic_word_list.append(topic)
        print(topic_word_list)
    return topic_word_list
 
get_topics(components)

• zip(vocab, comp) pairs each term with its weight in that component
• sorted(..., reverse=True)[:7] keeps the 7 highest weighted terms

4. Main topics distilled by SVD

['  co 19 covid coronavirus food online shopping']

['  co 19 covid coronavirus food online shopping', '  oil prices covid 19 rise co dow']

['  co 19 covid coronavirus food online shopping', '  oil prices covid 19 rise co dow', '  online shopping pandemic new home consumers j90i0ij4cj']

['  co 19 covid coronavirus food online shopping', '  oil prices covid 19 rise co dow', '  online shopping pandemic new home consumers j90i0ij4cj', '  food coronavirus demand stock bank help know']

['  co 19 covid coronavirus food online shopping',
 '  oil prices covid 19 rise co dow',
 '  online shopping pandemic new home consumers j90i0ij4cj',
 '  food coronavirus demand stock bank help know']

The 4 components:

Component	Top 7 terms
0	co, 19, covid, coronavirus, food, online, shopping
1	oil, prices, covid, 19, rise, co, dow
2	online, shopping, pandemic, new, home, consumers, j90i0ij4cj
3	food, coronavirus, demand, stock, bank, help, know

• the print sits inside the loop, so the growing list is printed once per component, and only the last line is the final result
• "co" and "j90i0ij4cj" are link fragments from the raw tweets, so they are preprocessing artefacts
• component 1 is a clear oil price topic, and component 2 is a clear online shopping topic

Advantages and Limitations

Advantages	Limitations
Semantic relationships: SVD captures semantic relationships between terms and documents, and the resulting low-dimensional representations often contain meaningful information about the underlying structure of the data	Scaling sensitivity: SVD is sensitive to the scaling of features, so it is essential to standardise or scale features appropriately before applying SVD
Interpretability: the reduced dimensions are often more interpretable, which makes it easier to understand and analyse the data, and it aids in tasks like topic modeling and sentiment analysis	Choosing dimensionality: deciding the appropriate number of dimensions (singular values) to retain can be subjective, and an incorrect choice may lead to information loss or overfitting

SVD keeps the directions with the largest singular values, and Principal Component Analysis (PCA) does the same after centring the data first.