Bahdanau attention is the attention method used inside an RNN encoder-decoder. It is also called additive attention, because it adds the query and the key before it scores them, rather than taking their dot product.
It lets the decoder look at every encoder output at each step, instead of relying on 1 fixed context vector. This is the fix for the bottleneck described in Sequence-to-Sequence Models and Attention.
Roles
| Role | What it is here | Shape |
|---|---|---|
| Query | The current decoder hidden state | (batch, 1, hidden) |
| Keys | All the encoder outputs | (batch, src_len, hidden) |
| Values | The same encoder outputs as the keys | (batch, src_len, hidden) |
There is no separate value projection. The encoder outputs are used for matching and are also the content that gets mixed into the context.
The score
The 4 steps for 1 decoder step:
- project the query with and every key with , then add them together
- apply to the sum
- project with down to 1 number, so every source word gets 1 score
- apply softmax over the source words, then take the weighted sum of the encoder outputs to get the context
Code
class BahdanauAttention(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.Wa = nn.Linear(hidden_size, hidden_size)
self.Ua = nn.Linear(hidden_size, hidden_size)
self.Va = nn.Linear(hidden_size, 1)
def forward(self, query, keys):
scores = self.Va(torch.tanh(self.Wa(query) + self.Ua(keys)))
scores = scores.squeeze(2).unsqueeze(1)
weights = F.softmax(scores, dim=-1)
context = torch.bmm(weights, keys)
return context, weightsWhy the layer sizes are what they are:
WaandUaboth keephidden_size, soWa(query)andUa(keys)have the same last dimension and can be addedVamapshidden_sizeto1, because each source position needs exactly 1 score- a version where
Vaalso outputshidden_sizewould give a vector per position, which softmax cannot turn into 1 weight per word
Follow the shapes, with B the batch size, S the source length and H the hidden size:
| Expression | Shape | Meaning |
|---|---|---|
Wa(query) + Ua(keys) | (B, S, H) | The query is broadcast across all S positions |
Va(tanh(...)) | (B, S, 1) | 1 score for each source position |
.squeeze(2).unsqueeze(1) | (B, 1, S) | Scores laid out as 1 row per decoder step |
softmax(..., dim=-1) | (B, 1, S) | Weights over the source words, summing to 1 |
torch.bmm(weights, keys) | (B, 1, H) | The context vector |
The decoder then joins its token embedding (H) with this context (H), which is why its GRU is built as nn.GRU(2 * hidden_size, hidden_size, batch_first=True).
Comparison with Transformer attention
| Point | Bahdanau attention | Scaled dot-product attention |
|---|---|---|
| Score | ||
| Combines query and key by | addition, then a small network | dot product |
| Values | the encoder outputs themselves | a separate projection |
| Model around it | RNN (GRU) encoder and decoder | no recurrence |
Remembering tip
Bahdanau Adds: query plus key, squash with tanh, then shrinks it to 1 score.
Sizes goH, H, 1forWa, Ua, Va.