Multilayer Perceptrons (MLP)

A multilayer perceptron (MLP), also called a feed-forward network (FFN), passes features through fully connected layers and activation functions. It can predict a class or a numerical value.

General two-layer model

The data path is input -> Linear -> ReLU -> Linear -> prediction. Each fully connected layer contains several Neurons. The first layer creates hidden features. ReLU adds a nonlinear operation. The final layer produces the required output size.

For 50 input features, 100 hidden features, and one regression target, the shapes are (batch, 50) -> (batch, 100) -> (batch, 1). Use mean squared error for the numerical target. For classification, match the output size and target format to the classification loss.

Inside a Transformer, an MLP also processes each token's representation after attention.

Role in a Transformer Block

graph TD
A[Input Tokens] --> B[Multi-Head Self-Attention] --> C[Multilayer Perceptrons, applied independently to each token] --> D[Output Tokens]

Steps

For each token independently:

1. Linear Projection (UP): expand the token embedding into a larger hidden dimension, typically 4 times
2. Activation (ReLU / GELU): introduce non-linearity where ReLU zeros out negatives and GELU is a smoother variant
3. Linear Projection (DOWN): compress back to the original embedding dimension

token (d_model)
   → Linear: d_model     --> d_model * 4
   → ReLU / GELU
   → Linear: 4·d_model   --> d_model

Key Points

Property	Detail
Applied per token	Each token goes through the MLP independently meaning no cross token mixing happening
Same weights	The same MLP is reused for every token position
Complements attention	the tokens that received from MLP already has attention baked in, so this MLP complements attention
Parameter-heavy	The two projection matrices hold ~2/3 of a Transformer's total parameters

Common Transformer order

Many Transformer blocks apply attention and then an MLP. Other architectures can arrange these operations differently.

---

Code

PyTorch: read the model class

import torch
from torch import nn
from torch.nn import functional as F
 
class OwnModel(nn.Module):
    def __init__(self, input_size, output_size, hidden_size):
        super().__init__()
        self.linear1 = nn.Linear(input_size, hidden_size)
        self.linear2 = nn.Linear(hidden_size, output_size)
 
    def forward(self, x):
        x = F.relu(self.linear1(x))
        return self.linear2(x)
 
X = torch.randn(100, 50)
y = X.sum(dim=1, keepdim=True)
model = OwnModel(input_size=50, output_size=1, hidden_size=100)
predictions = model(X)
loss = nn.MSELoss()(predictions, y)
print(tuple(predictions.shape))  # (100, 1)

Read Linear as values in, values out

Think of nn.Linear(12, 4) as 12 values in, 4 values out.
PyTorch stores its weight matrix as (4, 12): 4 output neurons, each with 12 weights. With the default bias=True, the 4 biases are stored separately with shape (4,).
nn.Linear(in_features, out_features) changes only the last input dimension. For example, (3, 5, 12) becomes (3, 5, 4). The other dimensions stay unchanged.

Read the lines:

• X contains 100 samples with 50 features each
• y is the sum of each sample's features, retained as a column
• linear1 changes the last dimension from 50 to 100
• ReLU sets negative hidden values to zero
• linear2 produces one numerical prediction per sample
• MSELoss compares predictions and targets with the same (100, 1) shape

This code calculates a loss. An optimiser and training loop are required to improve the weights.

TensorFlow/Keras: the same layer sizes

import tensorflow as tf
 
X = tf.random.normal((100, 50))
y = tf.reduce_sum(X, axis=1, keepdims=True)
model = tf.keras.Sequential([
    tf.keras.Input(shape=(50,)),
    tf.keras.layers.Dense(100, activation="relu"),
    tf.keras.layers.Dense(1),
])
model.compile(optimizer="adam", loss="mse")
model.fit(X, y, batch_size=20, epochs=2, verbose=0)
print(tuple(model(X).shape))  # (100, 1)

Dense performs the fully connected operation. compile selects the training settings, and fit applies weight updates. Each epoch has five batches of twenty samples.