Term Frequency-Inverse Document Frequency (TF- IDF)

Term Frequency refers to how often a term appears across a particular document, so it is one number per word per document
Inverse Document Frequency refers to the relative rarity of a term in the collection of documents, this is one number per unique word throughout corpus

Calculating TF

1. Raw count of how often a term appears in the document, $t{f}_{t,d}$
2. Term frequency adjusted for document length, $\frac{t{f}_{t,d}}{n\cdot words}$
3. Logarithmically scaled, $1+log(t{f}_{t,d})$
4. Boolean Frequency (1 if term appears in document, 0 if term does not appear in document)

Calculating IDF

$$idf(t,D)=log(\frac{N}{count(d\in D:t\in d)})+1$$

• $N$ is the total number of documents in the corpus ($D$)
• The denominator is the number of documents where the term $t$ appears

Calculating TF-IDF

$$tf(t,d)\cdot idf(t,d)$$

• the higher the TF-IDF score the more important or relevant the term is
• as the term is less relevant the TF-IDF score approaches $0$

TF-IDF Vectorization

This involves calculating the TF-IDF score for every unique word in the corpus relative to that document and then putting that information into a vector

Each document in the corpus has its own vector of all the same N dimensions. N here represents the total number of unique words in the corpus. Words that are in Document A but not in Document B will just show 0 under that slot in Vector B.

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
 
data1 = "I'm designing a document and don't want to get bogged down in what the text actually says"
data2 = "I'm creating a template for various paragraph styles and need to see what they will look like."
data3 = "I'm trying to learn more about some features of Microsoft Word and don't want to practice on a real document"
 
df1 = pd.DataFrame({'First_Para':[data1], 'Second_Para':[data2], 'Third_Para':[data3]})
vectorizer = TfidfVectorizer()
doc_vec = vectorizer.fit_transform(df1.iloc[0])
 
df2 = pd.DataFrame(doc_vec.toarray().transpose(), index=vectorizer.get_feature_names_out())
 
df2.columns = df1.columns
print(df2)

Advantages and Disadvantages

Advantages	Limitations
Efficiency: it is computationally cheap to calculate TF-IDF scores, especially compared to more complex NLP techniques	No semantics: it treats words as independent units, so it ignores meaning and the relationships between words
Language agnostic: it can be applied to documents in any language, which makes it useful for multilingual text analysis	Term frequency bias: frequent terms tend to receive higher weights, which can overemphasise common words that carry little information
No supervised training: it needs no training phase with labelled data, so it applies to many tasks without a labelled dataset	Vocabulary size: the number of unique terms across the corpus becomes a computational problem on large datasets, and rare terms may not get accurate IDF values

TF-IDF scores a term inside 1 document, and Best Match 25 (BM25) scores a whole document against a query instead.