JJ

Bidirectional Recurrent Neural Networks

4 min read

A bidirectional recurrent neural network reads a sequence in both directions. One network reads from left to right. A second network reads from right to left. Each direction has its own learned weights.

It extends Recurrent Neural Networks (RNN) by combining information from both sides of each position. The recurrent cells can be simple RNN, LSTM, or GRU cells.

Why Use Both Directions?

Consider the sentence She sat by the river bank. To interpret bank, the model can use the words before and after that position.

The two directions supply different context:

  • The forward state contains information from the start through the current token
  • The backward state contains information from the end through the current token
  • Combining the states gives a representation that can use the complete input sequence

Typical uses include entity tagging, part-of-speech tagging, and classification of a complete sentence.

The complete input must be available

A backward network uses later input tokens. Use this model when those tokens are available at prediction time. For next-token prediction, access to the token being predicted would reveal the answer.

Combining the Directions

The common operation is concatenation: place the two state vectors next to each other.

If each direction produces 16 values, the combined state has 32 values. A prediction layer must accept this combined size.

Output neededRepresentation usedExample
One label per tokenCombined state at every positionEntity tagging
One label per sequenceFinal state from each directionSentiment classification

Two directions and two stacked layers describe different structures. A single bidirectional layer already contains a forward network and a backward network.

Code

PyTorch: One Output per Token

This example uses three sequences. Each sequence has four token IDs. The model produces three class scores per token.

import torch
from torch import nn
 
class BiRNNModel(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_size):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.rnn = nn.RNN(
            embedding_dim,
            hidden_dim,
            batch_first=True,
            bidirectional=True,
        )
        self.fc = nn.Linear(hidden_dim * 2, output_size)
 
    def forward(self, x):
        embedded = self.embedding(x)
        output, hidden = self.rnn(embedded)
        return self.fc(output)
 
token_ids = torch.tensor([
    [1, 2, 3, 4],
    [2, 3, 4, 5],
    [3, 4, 5, 6],
], dtype=torch.long)
 
model = BiRNNModel(10, 8, 16, 3)
logits = model(token_ids)
print(tuple(logits.shape))  # (3, 4, 3)

Follow one batch through the model:

ExpressionShapeMeaning
token_ids(3, 4)Three sequences, four IDs each
embedded(3, 4, 8)Eight values per token
output(3, 4, 32)Sixteen values from each direction
hidden(2, 3, 16)Final states of the two directions
logits(3, 4, 3)Three class scores at each position

Key lines:

  • nn.Embedding(10, 8) stores ten rows of eight learned values; input IDs must be integers from 0 through 9
  • batch_first=True makes the input and output order (batch, sequence, features)
  • bidirectional=True creates the second direction
  • nn.Linear(32, 3) acts on the last dimension at every token position
  • hidden is returned by the RNN, but this token-level prediction uses output

For this one-layer RNN, a sequence-level head can use both final states:

embedded = model.embedding(token_ids)
output, hidden = model.rnn(embedded)
summary = torch.cat((hidden[0], hidden[1]), dim=-1)
sequence_logits = model.fc(summary)
print(tuple(sequence_logits.shape))  # (3, 3)

hidden[0] is the final forward state. hidden[1] is the final backward state. In a bidirectional RNN, output[:, -1, :] contains the backward state at the last input position, which has processed only that end of the sequence.

The hidden-state shape keeps the direction/layer axis first, including when batch_first=True. PyTorch RNN reference.

TensorFlow/Keras: The Same Output Structure

import tensorflow as tf
 
token_ids = tf.constant([
    [1, 2, 3, 4],
    [2, 3, 4, 5],
    [3, 4, 5, 6],
], dtype=tf.int32)
 
model = tf.keras.Sequential([
    tf.keras.Input(shape=(4,), dtype="int32"),
    tf.keras.layers.Embedding(input_dim=10, output_dim=8),
    tf.keras.layers.Bidirectional(
        tf.keras.layers.LSTM(16, return_sequences=True),
        merge_mode="concat",
    ),
    tf.keras.layers.Dense(3),
])
 
logits = model(token_ids)
print(tuple(logits.shape))  # (3, 4, 3)

Read the layer settings:

  • Bidirectional(...) wraps an LSTM with separate forward and backward processing
  • merge_mode="concat" joins the two 16-value outputs
  • return_sequences=True keeps all four token positions
  • Set return_sequences=False to obtain one combined representation per sequence; the final output then has shape (3, 3)
  • Replace LSTM with GRU or SimpleRNN to change the recurrent cell

The output values are logits, or raw class scores. These examples show data flow through randomly initialised models. Training is required before the predictions have learned meaning. Keras bidirectional layer reference.

Graph View