Random Forest

A random forest combines predictions from many decision trees. Each tree learns a series of feature-based decisions.

For example, a tree can use the number of positive words in a review to decide which branch to follow. A leaf at the end of a branch supplies a prediction.

How the Trees Differ

Random forests use two sources of randomness:

• Bootstrap sampling gives each tree a training sample drawn with replacement, so a row can appear more than once
• Feature sampling selects a random subset of features to consider at each split

The feature subset can change between splits within the same tree. The trees can train independently.

These differences reduce the similarity between tree errors. Combining the results can reduce the variance of individual trees. This is a form of bagging. See the random forest guide.

Combining Predictions

Task	Combination
Classification	Combine class predictions or class probabilities
Regression	Average numerical predictions

For classification, majority voting is a useful general explanation. Scikit-learn's RandomForestClassifier averages the class probabilities from its trees. It selects the class with the highest average probability. See the RandomForestClassifier API.

Important Settings

Setting	Meaning
n_estimators	Number of trees
max_depth	Maximum depth of each tree
max_features	Number of features considered at each split
bootstrap	Whether each tree uses a sample drawn with replacement
random_state	Seed used to reproduce random choices

Deeper trees can learn more detailed patterns. A depth limit can reduce their tendency to fit noise. More trees increase computation.

Random forests handle nonlinear feature relationships. Text still needs numerical features, such as word counts, TF-IDF values, or embedding values.

---

Code

1. Create review features

This small example uses two manually selected words so each feature is easy to inspect.

from sklearn.ensemble import RandomForestClassifier
 
train_texts = [
    "excellent excellent excellent film",
    "excellent excellent acting",
    "excellent story",
    "boring boring boring film",
    "boring boring acting",
    "boring story",
]
y_train = [1, 1, 1, 0, 0, 0]
 
def features(text):
    words = text.lower().split()
    return [words.count("excellent"), words.count("boring")]
 
X_train = [features(text) for text in train_texts]
print(X_train)

[[3, 0], [2, 0], [1, 0], [0, 3], [0, 2], [0, 1]]

Read the feature function:

• lower() gives both word searches the same letter case
• split() converts the string into a list of words
• count(...) counts an exact word in that list
• The returned list has a fixed order: positive-word count, then negative-word count
• The list comprehension applies the function to every review

X_train contains six rows and two columns. y_train contains the six sentiment labels.

2. Create and train the forest

forest = RandomForestClassifier(
    n_estimators=100,
    max_depth=3,
    max_features=1,
    bootstrap=True,
    random_state=42,
)
forest.fit(X_train, y_train)

This model contains 100 trees. Each split considers one of the two features. The maximum tree depth is three. fit learns the trees from the feature rows and target labels.

3. Predict new reviews

test_texts = ["excellent film", "boring film"]
X_test = [features(text) for text in test_texts]
 
predictions = forest.predict(X_test)
probabilities = forest.predict_proba(X_test)
 
print(forest.classes_)
print(predictions)
print(probabilities.shape)

[0 1]
[1 0]
(2, 2)

The results:

• classes_ gives the class order used by the probability columns
• predictions contains one label per test review
• probabilities contains two rows and two class probabilities per row
• probabilities[:, 1] selects the positive-class probability for both reviews

The same features function is used for training and prediction. This keeps the meaning and order of the two columns consistent.

Feature selection limits the information

Both wonderful film and awful film become [0, 0] in this example. The forest receives identical inputs for those two reviews. A larger vocabulary can preserve that difference.