Sequence-to-Sequence Models and Attention

A sequence-to-sequence model, or Seq2Seq model, receives one sequence and produces another sequence. Machine translation is a common use.

source sentence -> encoder -> context -> decoder -> target sentence

Encoder-decoder Seq2Seq model

The encoder processes the input one position at a time with an RNN, LSTM, or GRU. Its hidden states record information from the source sequence.

The original Seq2Seq design compresses the complete source sequence into one context vector. The decoder uses this vector to generate the target sequence.

This fixed-size context vector creates a bottleneck. Important information from an early input position can become difficult to preserve when the source sequence is long.

Why attention helps

Attention lets the decoder examine all encoder hidden states instead of relying on only one fixed summary.

At one decoder step, attention follows this process:

1. Use the current decoder state as the query, its like who is the one querying (in this case its the decoder)
2. Use the encoder hidden states as the keys
3. Compare the query with every key to produce attention scores
4. Apply softmax to convert the scores into attention weights
5. Use the weights to combine the encoder hidden states, which act as the values
6. Give the weighted context to the decoder for its next output

decoder query + encoder keys
-> scores
-> softmax weights
-> weighted sum of encoder values
-> attended context

The decoder can focus on different source words at different output steps.

Query, key, and value roles

Part	Practical meaning
Query	The information that the current position is looking for
Key	The information used to decide whether a position is relevant
Value	The information that passes forward when that position receives attention

A large query-key match produces a large attention weight. The final attention output is a weighted combination of the value vectors.

Encoder-decoder attention and self-attention

Mechanism	Query source	Key and value source
Encoder-decoder attention	Decoder	Encoder outputs
Self-attention	The current sequence	The same current sequence

Self-attention lets each input position connect directly with every other input position. This direct connection helps the model represent long-range relationships without recurrent processing.

For example, self-attention can connect it with animal in the sentence The animal did not cross the street because it was too tired.

Connection to Transformers

The Transformer replaces recurrent processing with self-attention. This change permits parallel processing across positions. The decoder still uses encoder-decoder attention when it needs information from the encoded source sequence.

Related notes: Transformers · Self-Attention · Attention Masks · Softmax

Code

PyTorch: one attention decoder step

import torch
from torch import nn
import torch.nn.functional as F
 
class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, dropout_p=0.1):
        super().__init__()
        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
        self.dropout = nn.Dropout(dropout_p)
 
    def forward(self, token_ids):
        embedded = self.dropout(self.embedding(token_ids))
        encoder_outputs, encoder_hidden = self.gru(embedded)
        return encoder_outputs, encoder_hidden
 
class BahdanauAttention(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.Wa = nn.Linear(hidden_size, hidden_size)
        self.Ua = nn.Linear(hidden_size, hidden_size)
        self.Va = nn.Linear(hidden_size, 1)
 
    def forward(self, query, keys):
        scores = self.Va(torch.tanh(self.Wa(query) + self.Ua(keys)))
        scores = scores.squeeze(2).unsqueeze(1)
        weights = F.softmax(scores, dim=-1)
        context = torch.bmm(weights, keys)
        return context, weights
 
class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1):
        super().__init__()
        self.embedding = nn.Embedding(output_size, hidden_size)
        self.attention = BahdanauAttention(hidden_size)
        self.gru = nn.GRU(2 * hidden_size, hidden_size, batch_first=True)
        self.out = nn.Linear(hidden_size, output_size)
        self.dropout = nn.Dropout(dropout_p)
 
    def forward_step(self, token_id, hidden, encoder_outputs):
        embedded = self.dropout(self.embedding(token_id))
        query = hidden.permute(1, 0, 2)
        context, weights = self.attention(query, encoder_outputs)
        gru_input = torch.cat((embedded, context), dim=2)
        output, hidden = self.gru(gru_input, hidden)
        logits = self.out(output)
        return logits, hidden, weights
 
encoder = EncoderRNN(input_size=20, hidden_size=8)
decoder = AttnDecoderRNN(hidden_size=8, output_size=30)
 
source_ids = torch.tensor([
    [1, 2, 3, 4, 5],
    [6, 7, 8, 9, 10],
])
decoder_input = torch.zeros(2, 1, dtype=torch.long)
 
encoder_outputs, encoder_hidden = encoder(source_ids)
logits, decoder_hidden, weights = decoder.forward_step(
    decoder_input,
    encoder_hidden,
    encoder_outputs,
)
 
print(tuple(logits.shape))   # (2, 1, 30)
print(tuple(weights.shape))  # (2, 1, 5)

Follow the shapes for this two-sentence batch:

Value	Shape	Meaning
source_ids	(2, 5)	Two input sequences with five token IDs each
encoder_outputs	(2, 5, 8)	One eight-value encoder state for each input position
encoder_hidden	(1, 2, 8)	Final encoder state for one GRU layer
query	(2, 1, 8)	Current decoder state in batch-first order
context	(2, 1, 8)	Weighted combination of the five encoder states
gru_input	(2, 1, 16)	Eight embedding values joined with eight context values
logits	(2, 1, 30)	Thirty target-vocabulary scores for the next position
weights	(2, 1, 5)	One attention weight for each input position

The decoder GRU receives 2 * hidden_size features because it joins the target-token embedding with the attended context. During training, a generation loop can feed the correct target token into the next step. During inference, it can feed the predicted token instead.

A basic decoder can return (decoder_outputs, decoder_hidden, None). The None keeps its return structure consistent with an attention decoder that returns attention weights.