Graph Convolutional Networks (GCN)

A Graph Convolutional Network (GCN) learns from graph-structured data. In NLP, the graph can represent words or phrases and the relationships between them.

The central idea is simple: each node receives information from its neighbours and uses that information to update its own representation.

Graph representation in NLP

Graph part	NLP meaning	Example
Node	A word, phrase, sentence, or document	`bank`, `approved`, `loan`
Edge	A linguistic or statistical relationship	Dependency link or co-occurrence link
Node feature	The starting vector for a node	Word2Vec, GloVe, or contextual BERT embedding

Graphs expose relationships that can be distant in the token sequence. A dependency edge can connect two related words even when other words occur between them.

Adjacency matrix

The adjacency matrix records which nodes are connected. For a graph with N nodes, it has shape (N, N).

Entry (i, j) is 1 when node i has an edge to node j
Entry (i, j) is 0 when that edge is absent
An undirected graph where each connections work both ways, has a symmetric adjacency matrix
A directed graph where each connections have a specific direction, can have an asymmetric adjacency matrix
A self-loop connects a node to itself, so the node can keep its own information during an update

The adjacency matrix controls who can send information to whom. The node-feature matrix stores what each node currently knows. These two matrices have different jobs.

Message passing

One message-passing layer follows this flow:

find neighbours -> collect their features -> combine the features -> update each node

The adjacency structure masks unrelated nodes. Multiplying or indexing with the graph connectivity makes each node receive only the permitted neighbour information.

For the sentence "The bank approved the loan," a graph can connect bank to approved and loan. After message passing, the representation of bank receives financial context from these neighbours.

Two meanings of aggregation

Aggregation can refer to two connected operations:

Neighbour aggregation combines messages for one target node. Mean, sum, max, or attention can perform this operation
Graph pooling combines all final node embeddings into one graph vector. A document classifier can use this vector

These operations produce different outputs. Neighbour aggregation keeps one output row for each node. Graph pooling reduces all node rows to one graph row.

Method	Result
Sum	Adds feature values and therefore reflects graph or neighbourhood size
Mean	Averages feature values and reduces the direct effect of size
Max	Keeps the strongest value for each feature
Attention	Learns a different importance weight for each neighbour or node

Code and shapes

A two-layer GCNConv model can use this structure:

import torch
from torch_geometric.nn import GCNConv
 
class GCN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, out_channels)
 
    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x
 
gcn = GCN(16, 32, 8)
node_embeddings = gcn(x, edge_index)

For four nodes with 16 input features each:

x                    (4, 16)
GCNConv(16, 32)  ->  (4, 32)
GCNConv(32, 8)   ->  (4, 8)

The output has four rows, so it contains four node embeddings. A graph-classification task still needs graph pooling to produce one document vector.

In PyTorch Geometric, edge_index has shape (2, E), where E is the number of listed directed edges. Its two rows store source-node and target-node indices.

Common confusions

Confusion	Correct distinction
Graph edge versus token order	An edge records a selected relationship. It does not have to connect adjacent words
Adjacency matrix versus node-feature matrix	Adjacency stores connectivity. Node features store vector values
Node embedding versus graph embedding	A node embedding represents one node. A graph embedding represents the complete sentence or document graph
GCN versus GAT	A GCN uses a defined neighbour aggregation rule. A [[Graph Attention Networks (GAT)	GAT]] learns neighbour importance weights

JJ

Explorer