K-Fold Cross-Validation

K-fold cross-validation evaluates a model design by training it on several parts of the same dataset. Each run uses a different group of samples for validation.

It is often used during Hyperparameter Tuning to compare model settings.

How it works

Split the development data into k groups, called folds. For each fold:

• Use that fold as the validation set
• Use the other k - 1 folds as the training set
• Fit a fresh model and measure its validation score
• Store the score, then continue with the next fold

After all runs, calculate the mean of the k scores. Each sample is used for validation once and for training k - 1 times.

For 100 samples and k = 5, each run uses 80 training samples and 20 validation samples. The process trains five separate models of the same design.

Start each fold from a fresh model

A model that keeps weights from the previous fold has already learned from some samples in its new validation set. Rebuild the model and its optimiser for each fold.

Choosing the split

The splitter determines which samples belong together:

Splitter	Behaviour	Suitable situation
KFold	Divides sample positions into groups	General independent samples
StratifiedKFold	Keeps class proportions approximately equal across folds	Classification, especially with unequal class counts
Leave-one-out	Uses one validation sample per run	Small datasets where many training runs are practical

For 90 negative and 10 positive examples, five stratified folds each contain approximately 18 negative and two positive examples. Plain K-fold does not enforce this balance.

Five or ten folds are common starting points. More folds require more training runs. Choose a fold count that leaves enough examples of each class in every split.

Shuffling changes the sample assignment. A fixed random seed makes that assignment repeatable. Data with time order or shared authors can require a splitter that preserves time order or keeps authors in separate groups.

What the score means

Cross-validation measures how the chosen training procedure performs across the selected splits. Its mean can be above or below the score from one train-test split.

Keep preprocessing inside each training fold. For text, learn the vocabulary from that fold's training documents, then use that vocabulary to transform its validation documents. A pipeline performs these operations in the correct order. See the scikit-learn data leakage guide.

After model selection, fit the chosen design on all development data. Keep a separate test set for the final evaluation.

---

Code

Read the sample indices

import numpy as np
from sklearn.model_selection import KFold
 
X = np.arange(6).reshape(-1, 1)
kf = KFold(n_splits=3, shuffle=False)
 
for fold, (train_idx, val_idx) in enumerate(kf.split(X), start=1):
    print(fold, train_idx.tolist(), val_idx.tolist())

Output:

1 [2, 3, 4, 5] [0, 1]
2 [0, 1, 4, 5] [2, 3]
3 [0, 1, 2, 3] [4, 5]

Read the code in this order:

• X has shape (6, 1): six samples with one feature each
• n_splits=3 creates three validation folds
• split(X) produces pairs of index arrays; it does not train a model
• train_idx selects training rows, so X[train_idx] has shape (4, 1)
• val_idx selects validation rows, so X[val_idx] has shape (2, 1)
• enumerate(..., start=1) adds the fold number used in the print statement

To shuffle these independent samples, use KFold(n_splits=3, shuffle=True, random_state=42). The seed controls the split, not the model's learned weights. See the KFold API.

Evaluate a complete text pipeline

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
 
texts = [
    "good bright film", "bad dull film",
    "good fun story", "bad slow story",
    "good warm acting", "bad weak acting",
]
labels = [1, 0, 1, 0, 1, 0]
 
pipeline = Pipeline([
    ("vectorizer", CountVectorizer()),
    ("classifier", MultinomialNB()),
])
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
scores = cross_val_score(
    pipeline, texts, labels, cv=cv, scoring="f1_macro"
)
 
print(scores.shape)
print(scores.mean())

The steps have separate jobs:

• texts contains the raw documents; labels contains their matching classes
• CountVectorizer converts documents into word-count rows
• MultinomialNB learns class patterns from those counts
• StratifiedKFold uses the labels to keep each validation fold balanced
• cross_val_score fits a fresh copy of the complete pipeline for each split
• f1_macro gives each class equal weight in the score
• scores has shape (3,), with one score per fold

The vocabulary is learned separately in each fold. A word that appears only in validation is ignored for that run because the training vocabulary has no column for it.