Hyperparameter Tuning

Hyperparameter tuning compares model settings and selects those that perform best on validation data.

A parameter is learned during training, such as a neural network weight. A hyperparameter controls the model or its training, such as the learning rate or batch size.

What can be tuned

Common choices include:

Setting	What it controls
Learning rate	Size of each weight update
Batch size	Number of samples used for one training update
Epoch limit	Maximum number of full passes through training data
Hidden size	Number of features stored in a hidden representation
Dropout rate	Fraction of activations removed during training
Naive Bayes alpha	Strength of count smoothing
Vectorizer ngram_range	Whether features include single words or word sequences

The best value depends on the dataset, model, and scoring rule. A setting that works well for one task can perform poorly on another.

Practical tuning order

These values are starting points. The best values still depend on the data and model.

Priority	Settings	Guidance
Tier 1: big knobs	Learning rate, batch size, regularisation	Start here because these settings can decide whether training is stable and whether the model overfits
Tier 2: model architecture	Embedding size, hidden units, number of layers, sequence length, n-gram range	Tune model capacity and the amount of context after the main training settings work
Tier 3: final tuning	Optimiser, exact dropout rate, schedules, gradient clipping	Use these settings for smaller final gains

The slides give these practical starting ranges:

• Start Adam near 1e-3 and search learning rates from about 1e-1 to 1e-5
• Treat batch sizes from 32 to 128 as small and 256 or more as large
• Search dropout rates from about 0.2 to 0.5
• Use about 1e-4 as a typical starting value for weight decay
• Use 100 to 300 dimensions as a common range for static word embeddings

A small batch can support generalisation but takes more updates. A large batch can use parallel hardware efficiently but can increase overfitting risk. See Batch Size and Epochs.

Search strategies

Strategy	How it selects trials	Main trade-off
Grid search	Tries every defined combination	Complete for the grid, but expensive when the grid has many settings
Random search	Samples combinations from the search space	Covers a large space with fewer trials
Bayesian optimisation	Uses earlier trial results to choose the next trial	Efficient for expensive models, but more complex to configure

Grid search details

Grid search tries every combination in a defined set of values. It uses K-Fold Cross-Validation to score each combination across several data splits.

For three smoothing values and two vectorizer settings, there are six combinations. With three folds, the search performs 18 training runs. It then refits the selected combination once on all development data.

Random search samples combinations from a defined space instead of trying every combination. It can reduce the number of runs when the search space is large.

Resource-aware tuning

Each trial is a full training run, so tuning uses time and compute resources.

Here is a practical process:

1. Use 10 to 20 percent of the data or only 2 to 3 epochs for an initial search
2. Remove configurations that do not reach the baseline result
3. Stop a run when validation accuracy improves by less than 1 percent over 3 epochs
4. Stop when the loss stays flat or diverges for about 5 epochs
5. Run several configurations in parallel when the hardware supports it

The first 5 to 10 trials often give the largest gains. After about 20 trials, the lecture expects gains below 0.5 percent unless the dataset or model is very large. This is a guide for deciding when the cost is larger than the likely gain. See Early Stopping.

Practical caveats

• Validation overfitting: Repeated tuning on one validation split can make the selected settings fit that split too closely. Use cross-validation or keep a final test set untouched
• Compute bias: More compute permits more trials. Report the resources used together with model performance
• Reproducibility: Fix random seeds when you compare trials. Runs can differ even when the hyperparameters are the same
• Transferability: Retune when the dataset or task changes. The best settings do not transfer automatically
• Diminishing returns: A gain below 0.5 percent can have less value than its compute cost

Choose the scoring rule first

The score determines which candidate wins:

• Accuracy measures the fraction of correct predictions
• Macro F1 calculates F1 for each class, then gives each class equal weight
• A loss measures prediction error, so lower loss is better

Use the same selected score throughout the search. With scikit-learn, scoring="f1_macro" explicitly selects macro F1. When scoring is omitted, MultinomialNB uses its default accuracy score.

Separate selection from final evaluation

Validation data helps choose hyperparameters. The highest validation score is therefore part of the selection process. It can give an optimistic estimate of performance on new data.

Use a separate test set after selection. Another option is nested cross-validation, which puts the complete tuning process inside an outer evaluation loop. See the scikit-learn comparison of nested and ordinary cross-validation.

Keep the vectorizer inside the model pipeline. This lets each training fold learn its own vocabulary before the corresponding validation fold is transformed.

---

Code

Search a small text classifier

This complete example uses 12 development documents and two separate test documents. Class 1 means positive; class 0 means negative.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import f1_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
 
texts = [
    "good bright film", "bad dull film",
    "good fun story", "bad slow story",
    "good warm acting", "bad weak acting",
    "good enjoyable movie", "bad boring movie",
    "good exciting plot", "bad confusing plot",
    "good lovely ending", "bad awful ending",
]
labels = [1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0]
test_texts = ["good enjoyable film", "bad dull movie"]
test_labels = [1, 0]
 
pipeline = Pipeline([
    ("vectorizer", CountVectorizer()),
    ("classifier", MultinomialNB()),
])
param_grid = {
    "classifier__alpha": [0.1, 0.5, 1.0],
    "vectorizer__ngram_range": [(1, 1), (1, 2)],
}
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
search = GridSearchCV(
    pipeline,
    param_grid,
    cv=cv,
    scoring="f1_macro",
    refit=True,
)
search.fit(texts, labels)
 
predictions = search.predict(test_texts)
print(search.best_params_)
print(search.best_score_)
print(predictions.tolist())
print(f1_score(test_labels, predictions, average="macro"))

Follow the data through the search

Read these lines by their jobs:

• Pipeline joins preprocessing and classification into one estimator
• The names vectorizer and classifier identify its two steps
• classifier__alpha means the alpha setting inside the classifier step
• vectorizer__ngram_range=(1, 1) uses single words
• vectorizer__ngram_range=(1, 2) uses single words and pairs of adjacent words
• cv makes three balanced folds, each with eight training documents and four validation documents
• search.fit(texts, labels) tries the six combinations and records their validation scores
• refit=True trains the selected pipeline again using all 12 development documents
• search.predict(test_texts) applies that fitted vectorizer and classifier to the two test documents

Inside one fold, the vectorizer produces a sparse matrix with shape (8, V), where V is that fold's vocabulary size. The matching validation matrix has shape (4, V). The final prediction array has shape (2,).

For these simple test documents, the predicted labels are [1, 0] and macro F1 is 1.0. These two examples demonstrate the code path. They provide little evidence about performance on varied reviews.

Read the result fields

The search stores several results:

Field	Meaning
best_params_	Selected setting values
best_score_	Mean validation score for those settings
best_estimator_	Selected pipeline after the final refit
cv_results_	Scores, timings, and settings for all candidates

Print each candidate's settings and mean score:

for params, score in zip(
    search.cv_results_["params"],
    search.cv_results_["mean_test_score"],
):
    print(params, round(score, 3))

In cv_results_, the word test refers to the held-out fold within cross-validation. The separate test_texts are used only after the search finishes.

All six candidates score 1.0 on this small dataset. The search selects the first tied candidate, with alpha=0.1 and single-word features. These equal scores give no evidence that this setting is better than the other candidates.

The trailing underscore identifies attributes created by fitting. Access them after fit completes. See the GridSearchCV API.