Logistic Regression

Logistic regression is a supervised classification model. It learns how input features affect the probability of a class.

For sentiment analysis, the input can be a TF-IDF vector for a review. The output can be the probability that the review is positive.

How It Makes a Prediction

The binary classification process:

• Multiply each feature by its learned weight and add a bias to obtain a score
• Apply a sigmoid function to convert the score into a probability between 0 and 1
• Use a decision threshold to select a class

A positive weight increases the score when that feature increases. A negative weight decreases it. Training adjusts the weights to reduce classification loss.

Three Different Outputs

Method	Output for each review	Use
predict	One class label	Compute accuracy, precision, recall, or F1
predict_proba	One probability per class	Inspect confidence or apply a threshold
decision_function	A decision score	Rank examples or compute binary ROC-AUC

The probability columns follow model.classes_. For labels 0 and 1, column 0 refers to class 0 and column 1 refers to class 1. See the LogisticRegression API.

Classification and regression

Logistic regression predicts class probabilities. Linear regression predicts a numerical value that can fall outside the probability range.

Evaluation

Area Under Curve (AUC-ROC) measures how well scores rank positive examples above negative examples. It can use positive-class probabilities or decision scores. Thresholded class labels discard much of this ranking information. See the ROC evaluation guide.

---

Code

1. Define training and test reviews

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
 
train_review = [
    "excellent enjoyable movie",
    "great enjoyable film",
    "excellent great acting",
    "enjoyable wonderful story",
    "terrible boring movie",
    "awful boring film",
    "terrible awful acting",
    "boring dreadful story",
]
train_sent = [1, 1, 1, 1, 0, 0, 0, 0]
 
test_review = [
    "excellent enjoyable story",
    "terrible boring story",
    "great wonderful movie",
    "awful dreadful movie",
]
test_sent = [1, 0, 1, 0]

train_review contains eight strings. train_sent gives one label per string. The test reviews are separate inputs used after training.

2. Convert text into a shared feature space

tv = TfidfVectorizer()
train_review_tfidf = tv.fit_transform(train_review)
test_review_tfidf = tv.transform(test_review)
 
print(train_review_tfidf.shape)
print(test_review_tfidf.shape)

(8, 12)
(4, 12)

The operations:

• fit_transform learns the vocabulary and IDF values from the training reviews, then creates their vectors
• transform uses those same learned values for the test reviews
• Each row is one review and each column is one vocabulary term
• Both matrices have 12 columns, so each feature has the same meaning in both datasets

3. Create, train, and use the model

log_regression = LogisticRegression()
log_regression.fit(train_review_tfidf, train_sent)
 
y_pred = log_regression.predict(test_review_tfidf)
probabilities = log_regression.predict_proba(test_review_tfidf)
 
print(log_regression.classes_)
print(y_pred)
print(probabilities.shape)

[0 1]
[1 0 1 0]
(4, 2)

fit receives review vectors and target labels. predict returns four labels. predict_proba returns four rows with two probabilities per row.

4. Select positive-class probabilities

y_pred_proba = probabilities[:, 1]
print(y_pred_proba.round(3))

Read [:, 1] as all rows, column 1. The result has one value per test review. probabilities[: :, 1] has the same meaning. The selected column is positive because classes_ is [0, 1].

5. Calculate evaluation results

fpr, tpr, thresholds = metrics.roc_curve(test_sent, y_pred_proba)
auc = metrics.roc_auc_score(test_sent, y_pred_proba)
f1 = metrics.f1_score(test_sent, y_pred)
 
print("AUC:", auc)
print("F1:", f1)

AUC: 1.0
F1: 1.0

roc_curve returns false-positive rates, true-positive rates, and the thresholds that produce them. Use fpr, tpr, _ when the threshold array is unnecessary. AUC uses the probability scores; F1 uses the predicted labels.

These results describe the four test reviews defined above.