Graph Attention Networks (GAT)

A Graph Attention Network (GAT) is a graph neural network that learns how much attention to give to each neighbouring node.

This model is useful when some graph connections give more useful context than other connections. In a text graph, a word can receive a strong signal from an important syntactic neighbour and a weaker signal from a less useful neighbour.

Main idea

A GAT layer follows this flow:

node features + graph edges
        -> score each permitted neighbour
        -> normalise the scores into attention weights
        -> form a weighted sum of neighbour features
        -> produce an updated embedding for each node

The graph edges decide which node pairs can interact. The learned attention weights decide the relative importance of the permitted neighbours.

GCN and GAT

Model	Neighbour aggregation
GCN	Uses a graph-based normalisation rule to combine neighbour information
GAT	Learns an attention weight for each permitted neighbour

Both models keep one output row for each node after message passing. A graph-classification task still needs a pooling operation to combine the node embeddings into one graph embedding.

Multiple attention heads

A GAT can use several attention heads. Each head learns its own neighbour-attention pattern.

With the default concat=True setting, the layer concatenates the head outputs. If one head produces hidden_channels features and the layer has heads heads, the output width is:

hidden_channels * heads

With concat=False, the layer averages the head outputs. The output width stays equal to hidden_channels.

PyTorch Geometric implementation

import torch
from torch_geometric.nn import GATConv
 
class GAT(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GATConv(in_channels, hidden_channels, heads=2)
        self.conv2 = GATConv(hidden_channels * 2, out_channels, heads=1)
 
    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x

For four nodes with 16 input features, 32 hidden features per head, and two heads:

x                              (4, 16)
GATConv(16, 32, heads=2)  ->   (4, 64)
GATConv(64, 8, heads=1)   ->   (4, 8)

The second layer must accept 64 features because the first layer concatenates two 32-feature head outputs.

In PyTorch Geometric, edge_index has shape (2, E). It lists the permitted source and target nodes. Attention does not create unrestricted connections between all nodes unless those edges are present.

Graph classification

The GAT output contains node embeddings. A document-level classifier can use this flow:

node embeddings -> graph pooling -> graph embedding -> classifier

Mean, sum, max, or attention pooling can produce the graph embedding.

Common confusions

Confusion	Correct distinction
Graph edge versus attention weight	The edge permits information flow. The attention weight controls how much the permitted neighbour contributes
Attention head versus graph node	A head is one learned attention mechanism. A node is one item in the graph
Multi-head output width	Concatenation multiplies the width by the number of heads. Averaging keeps the width unchanged
Node output versus graph output	A GAT layer returns one embedding per node. Pooling produces one embedding per graph
GAT versus Transformer self-attention	GAT attention follows graph edges. Transformer self-attention usually compares token positions allowed by its attention mask

JJ

Explorer