Low-Rank Adaptation (LoRA)

LoRA adapts a large model by training two small matrices instead of the whole weight matrix. The pretrained weights never move.

How it works

• the pretrained weight matrix $W_0$ is frozen, and only $B$ and $A$ are trained
• LoRA learns a low-rank update, formed as the product $BA$ of two small matrices
• during training, the low-rank branch runs beside the frozen weights, and its output is scaled and added to the output of the frozen weights
• the update is scaled by alpha divided by r, which decouples the learning rate from the choice of rank

Initialisation

• $A$ is drawn from a Gaussian
• $B$ is initialised to 0, so the update contributes nothing at step one and training starts exactly at the pretrained model

Where to inject it

• it is most effective in the attention projections Wq and Wv
• rank, alpha and the choice of weight matrices are the design choices it exposes, and the tokenizer is not one of them

Cost

• the number of trainable parameters grows with the rank $r$, which is chosen far smaller than the model width
• a rank of 1 or 2 can be enough even when the model width is 12,288
• on GPT-3 175B, LoRA reported 10,000 times fewer trainable parameters and 3 times less GPU memory

Deployment

• at deploy time $BA$ is folded into $W_0$, so LoRA adds no extra inference latency
• series adapters are the ones that add latency, because they sit in the path

LoRA is one method inside Parameter-Efficient Fine-Tuning (PEFT).