Loss Functions

A loss function measures the error between a model's prediction and the known target. Training uses this error to calculate gradients and update the model's parameters.

A metric reports performance. A loss also supplies the training signal. For example, accuracy reports the fraction of correct class decisions, while cross-entropy measures the probability assigned to the correct answer.

Choose the Loss from the Task

Task	Target	Suitable loss
Binary classification	One of two classes, such as spam or ordinary mail	Binary cross-entropy
Multiclass classification	One class from several choices, such as three news topics	Categorical cross-entropy
Regression	A numerical value, such as a rating score	Mean squared error

Binary cross-entropy penalises confident wrong predictions strongly. For a positive example, a predicted positive probability of 0.9 gives a lower loss than 0.1.

Categorical cross-entropy applies the same principle to several mutually exclusive classes. Its connection to probability and uncertainty is explained in Entropy.

Logits and Probabilities

A logit is a raw score. It can be negative or positive. A probability is between zero and one.

The output layer and loss must agree:

Output supplied to the loss	Required interpretation
Raw scores	Use a loss that performs the required probability conversion internally
Sigmoid probabilities	Use a binary loss configured for probabilities
Softmax probabilities	Use a categorical loss configured for probabilities

Match all three parts

Check the output shape, target format, and loss function together. The number of output values describes what the model predicts.

Labels as IDs or One-Hot Vectors

For three classes, class 2 can be stored as an integer ID or as [0, 0, 1].

Both representations identify the same answer. Libraries use different loss names or target requirements for them. In Keras, sparse categorical cross-entropy uses integer class IDs. Here, sparse refers to the label format.

---

Code

PyTorch: Binary Classification

import torch
from torch import nn
 
logits = torch.tensor([[0.0], [2.0]], requires_grad=True)
targets = torch.tensor([[0.0], [1.0]])
 
criterion = nn.BCEWithLogitsLoss()
loss = criterion(logits, targets)
print(round(loss.item(), 4))  # 0.4100
loss.backward()

Read the inputs:

• Each row is one example with one output score
• Both tensors have shape (2, 1)
• Binary targets are floating-point values
• BCEWithLogitsLoss combines sigmoid and binary cross-entropy; pass raw scores directly
• loss.item() extracts the single loss value for display
• loss.backward() calculates gradients; a separate optimiser step updates model weights

The raw score 0.0 corresponds to probability 0.5. The raw score 2.0 corresponds to approximately 0.881. PyTorch binary loss reference.

PyTorch: Multiclass Classification

logits = torch.tensor([
    [2.0, 0.0, -1.0],
    [0.0, 1.0, 2.0],
], requires_grad=True)
targets = torch.tensor([0, 2], dtype=torch.long)
 
criterion = nn.CrossEntropyLoss()
loss = criterion(logits, targets)
predictions = logits.argmax(dim=1)
 
print(tuple(logits.shape))  # (2, 3)
print(predictions.tolist())  # [0, 2]

Each row contains three raw class scores. targets has shape (2,) and stores one class ID per row. argmax(dim=1) selects the class column with the largest score.

CrossEntropyLoss applies the required log-softmax operation internally. The direct input is the raw model output. PyTorch multiclass loss reference.

PyTorch: One Prediction at Every Token

Suppose a sequence model returns shape (3, 4, 10): three sequences, four positions, ten class scores per position.

output = torch.randn(3, 4, 10, requires_grad=True)
target_ids = torch.tensor([
    [2, 3, 4, 5],
    [3, 4, 5, 6],
    [4, 5, 6, 7],
], dtype=torch.long)
 
scores = output.reshape(-1, 10)
labels = target_ids.reshape(-1)
loss = nn.CrossEntropyLoss()(scores, labels)
 
print(tuple(scores.shape))  # (12, 10)
print(tuple(labels.shape))  # (12,)

-1 asks PyTorch to calculate that dimension. The batch and sequence axes become twelve prediction rows. The matching targets become twelve class IDs, in the same order.

view(-1, 10) performs the same shape change when the tensor's memory layout permits that view. reshape can also handle layouts that require a copy.

TensorFlow/Keras: Match the Target Format

import tensorflow as tf
 
binary_logits = tf.constant([[0.0], [2.0]])
binary_targets = tf.constant([[0.0], [1.0]])
binary_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True)
print(round(float(binary_loss(binary_targets, binary_logits)), 4))
# 0.4100
 
class_logits = tf.constant([[2.0, 0.0, -1.0], [0.0, 1.0, 2.0]])
class_ids = tf.constant([0, 2])
 
sparse_loss = tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=True
)
loss_from_ids = sparse_loss(class_ids, class_logits)
 
one_hot_targets = tf.one_hot(class_ids, depth=3)
categorical_loss = tf.keras.losses.CategoricalCrossentropy(
    from_logits=True
)
loss_from_one_hot = categorical_loss(one_hot_targets, class_logits)
 
print(float(loss_from_ids))
print(float(loss_from_one_hot))  # Same value, within rounding

The call order in Keras is (targets, predictions). The PyTorch examples use (predictions, targets).

Read the three loss choices:

• BinaryCrossentropy receives one binary target per output
• SparseCategoricalCrossentropy receives integer class IDs
• CategoricalCrossentropy receives one-hot targets in this example
• from_logits=True matches the raw scores; for explicit sigmoid or softmax outputs, use from_logits=False

Use the chosen object in model.compile(loss=..., optimizer=...). A final Dense(1) layer supplies one raw binary score. A final Dense(3) layer supplies three raw class scores. Keras loss reference.