ReLU and Gated Activations

Modern language models have moved away from the original ReLU non-linearity in their MLP blocks toward gated activations (e.g. SwiGLU). This page explains the structural difference.

ReLU

Formula: output = relu(x @ W)

ReLU simply applies a linear transformation, then threshold negatives to zero.

x  →  Linear (x @ W)  →  ReLU (max(0, z))  →  output

max(0, x): negatives become 0, positives pass through unchanged
because of this, a feature is either killed completely or passed through linearly
No in-between --> no way to say "pass 40% of this feature through"
Used in the original 2017 Transformer paper

Sigmoid and tanh in recurrent models

Sigmoid produces a value between zero and one, so it can control how much information a gate passes. Tanh produces a value between minus one and one, so it can supply signed candidate content. Inputs far from zero can make their derivatives small.

Gated Activations

Formula: output = swish(x @ W1) * (x @ W2)

Gated activations use two branches, multiplied element-wise:

x  →  Linear (x @ W1)  →  Swish  ─┐
                                    × →  output
x  →  Linear (x @ W2)  ────────────┘
         ↑ gate branch       ↑ content branch

Branch	Role
Gate branch `swish(x @ W1)`	A learned, input-dependent valve which controls how much of each feature passes through
Content branch `x @ W2`	The signal being modulated

Both branches use the same input x but with different learned weight matrices (W1 and W2).

The Real Difference

The key difference is structural, not about learnable parameters since both approaches have weight matrices.

	ReLU	Gated (SwiGLU)
Branches	1	2 (multiplied together)
Non-linearity	Fixed (hard threshold)	Continuous, input-dependent
Per-dimension control	Zero negative values; retain positive values	Multiply by a learned input-dependent value
Parameters	`W` (one matrix)	`W1` + `W2` (two matrices)
Compute	One input projection in this comparison	Two input projections in this comparison
Quality	Depends on the model and task	Depends on the model and task

Example: A sigmoid gate of 0.2 passes 20% of a content value. A sigmoid gate of 0.9 passes 90%. SwiGLU uses Swish rather than sigmoid for its gate branch, so its multiplier can be negative or greater than one.

Variants of Gated Activations

Variant	Gate activation	Used in
GLU	Sigmoid	Original gating paper
SwiGLU	Swish	Llama, Mistral, PaLM
GeGLU	GELU	Various modern models

A complete gated feed-forward block usually adds an output projection after the two branches. Hidden width affects its parameter count and computation. Choose the architecture using measured performance for the task. GLU variants paper.

Code

PyTorch: compare the output values

import torch
 
x = torch.tensor([-2.0, 0.0, 2.0])
print(torch.relu(x))                 # [0., 0., 2.]
print(torch.sigmoid(x).round(decimals=3))  # [0.119, 0.500, 0.881]
print(torch.tanh(x).round(decimals=3))     # [-0.964, 0.000, 0.964]
 
content = torch.tensor([10.0, 10.0, 10.0])
gate = torch.sigmoid(x)
gated_content = gate * content
print(gated_content.round(decimals=3))  # [1.192, 5.000, 8.808]

All operations keep the input shape. gate * content multiplies corresponding positions. At the middle position, a gate value of 0.5 passes half of the content value 10.

TensorFlow/Keras

import tensorflow as tf
 
x = tf.constant([-2.0, 0.0, 2.0])
print(tf.nn.relu(x).numpy())
print(tf.math.sigmoid(x).numpy().round(3))
print(tf.math.tanh(x).numpy().round(3))
content = tf.constant([10.0, 10.0, 10.0])
print((tf.math.sigmoid(x) * content).numpy().round(3))

The outputs match the PyTorch example within floating-point rounding. For a Swish gate, use torch.nn.functional.silu(x) or tf.nn.silu(x) in place of sigmoid. The multiplication still occurs element by element.

JJ

Explorer