Bahdanau attention is the attention method used inside an RNN encoder-decoder. It is also called additive attention, because it adds the query and the key before it scores them, rather than taking their dot product.

It lets the decoder look at every encoder output at each step, instead of relying on 1 fixed context vector. This is the fix for the bottleneck described in Sequence-to-Sequence Models and Attention.

Roles

RoleWhat it is hereShape
QueryThe current decoder hidden state(batch, 1, hidden)
KeysAll the encoder outputs(batch, src_len, hidden)
ValuesThe same encoder outputs as the keys(batch, src_len, hidden)

There is no separate value projection. The encoder outputs are used for matching and are also the content that gets mixed into the context.

The score

The 4 steps for 1 decoder step:

  • project the query with and every key with , then add them together
  • apply to the sum
  • project with down to 1 number, so every source word gets 1 score
  • apply softmax over the source words, then take the weighted sum of the encoder outputs to get the context

Code

class BahdanauAttention(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.Wa = nn.Linear(hidden_size, hidden_size)
        self.Ua = nn.Linear(hidden_size, hidden_size)
        self.Va = nn.Linear(hidden_size, 1)
 
    def forward(self, query, keys):
        scores = self.Va(torch.tanh(self.Wa(query) + self.Ua(keys)))
        scores = scores.squeeze(2).unsqueeze(1)
        weights = F.softmax(scores, dim=-1)
        context = torch.bmm(weights, keys)
        return context, weights

Why the layer sizes are what they are:

  • Wa and Ua both keep hidden_size, so Wa(query) and Ua(keys) have the same last dimension and can be added
  • Va maps hidden_size to 1, because each source position needs exactly 1 score
  • a version where Va also outputs hidden_size would give a vector per position, which softmax cannot turn into 1 weight per word

Follow the shapes, with B the batch size, S the source length and H the hidden size:

ExpressionShapeMeaning
Wa(query) + Ua(keys)(B, S, H)The query is broadcast across all S positions
Va(tanh(...))(B, S, 1)1 score for each source position
.squeeze(2).unsqueeze(1)(B, 1, S)Scores laid out as 1 row per decoder step
softmax(..., dim=-1)(B, 1, S)Weights over the source words, summing to 1
torch.bmm(weights, keys)(B, 1, H)The context vector

The decoder then joins its token embedding (H) with this context (H), which is why its GRU is built as nn.GRU(2 * hidden_size, hidden_size, batch_first=True).

Comparison with Transformer attention

PointBahdanau attentionScaled dot-product attention
Score
Combines query and key byaddition, then a small networkdot product
Valuesthe encoder outputs themselvesa separate projection
Model around itRNN (GRU) encoder and decoderno recurrence

Remembering tip

Bahdanau Adds: query plus key, squash with tanh, then shrinks it to 1 score.
Sizes go H, H, 1 for Wa, Ua, Va.