Source
Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree
Definition
Text classification assigns a text to one or more predefined categories based on its content. The classifier is a mapping from an input text to an output label (class, category) .
Applications:
- news stories into World, Business, SciTech, Sports and so on (e.g. Google News)
- email into spam and non-spam, or support emails into Mac, Windows and other
- movie reviews into favourable, unfavourable or neutral
- student essays into grades A to F
Machine Learning in Brief
The model is , where are the model parameters.
Two phases:
- training: given labelled examples , estimate by minimising the prediction error on the training set
- testing: apply to examples never seen in training and compare the predicted with the true label
- the available data is split into a training set and a testing set
Representing Text
A classifier needs numbers, so the text is pre-processed first:
- remove stop words: prepositions (in, at, on, of, to), pronouns (he, she, they, it) and punctuation
- stemming: reduce words to a root form, e.g. walk, walker, walked, walking all become walk
- vectorisation: turn each text into a numeric vector, the simplest being bag-of-words, which counts how often each word appears and ignores word order
Classifiers
There are hundreds of classifiers, e.g. Naive Bayes, support vector machines, decision trees (random forest, XGBoost) and neural networks (deep learning).
Python Example
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score
data = [("This is a technology news article.", "Technology"),
("Learn how to cook a delicious recipe.", "Cooking"),
("Stock market analysis and investment strategies.", "Finance")]
# 12 examples in 6 categories in the lecture
texts, labels = zip(*data)
vectorizer = CountVectorizer() # bag-of-words counts
X = vectorizer.fit_transform(texts)
X_train, X_test, y_train, y_test = train_test_split(
X, labels, test_size=0.2, random_state=42)
classifier = MultinomialNB() # Naive Bayes on word counts
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(accuracy_score(y_test, y_pred))
new = vectorizer.transform(["Smartphone buying guide"])
print(classifier.predict(new)[0]) # TechnologyWith only 12 training texts the lecture run gets a test accuracy of , and it even labels "Get the latest updates on mobile technology." as Fashion. A tiny training set gives unreliable predictions, so real systems need far more labelled data. Web search, one of the search applications from the start of the search topic, relies on the same kind of text representation to match pages to keywords.