48GB GPU Fine-Tuning a 65B Model: Where QLoRA Saves GPU Memory

In 2023, a University of Washington team used QLoRA to fine-tune a 65-billion-parameter model on a single GPU with 48GB of memory, and reported task performance close to 16-bit full-parameter fine-tuning. The trained Guanaco ran for 24 hours on one GPU and reached 99.3% of ChatGPT's performance on the Vicuna evaluation. The memory bill is more honest than the terminology; first, see how it saves.

In the original article, Dongxi 东锡 NLP breaks LoRA and QLoRA into three concepts: rank, low-rank approximation, and quantization. The three can be tied together with one word: patterns in matrices. The original link is here, and the main post link is here.

Full-Parameter Fine-Tuning Eats Memory First

Full-parameter fine-tuning updates all parameters. During training, GPU memory holds not only weights but also gradients and the two sets of states Adam maintains for each parameter. Estimated at 32-bit floating point, each parameter takes about 16 bytes. A 7B model needs more than 100GB, a 65B model about 1TB, and the activations produced by forward computation must be counted separately.

There are two directions for reducing memory. LoRA reduces the number of parameters that need training; quantization reduces the number of bits each parameter occupies.

A Pattern Is One Column Times One Row

A pattern is a column vector multiplied by a row vector, producing a rank-1 matrix. Each element in the matrix equals the product of the corresponding column element and row element; each row is the same row vector scaled by a different factor.

The sales table example is intuitive. Three stores' sales over four consecutive days, with each day's rhythm [1, 2, 2, 1], and store scales 10, 20, and 30. The whole 3×4 table is the column vector [10, 20, 30] times the row vector [1, 2, 2, 1]. The 12 numbers are determined by only 3+4=7 numbers, and the rank is 1. Add a promotion, with an extra [0, 0, 5, 5] on days 3 and 4, and store participation levels [1, 0, 2]. The new table is the sum of two patterns, and the rank becomes 2.

A matrix can be viewed as a sum of several patterns, and the rank is the number of mutually independent patterns among them. When the pattern u vᵀ acts on an input vector x, first compute x's component in direction v, then write that value onto direction u. A rank-r matrix has r pairs of read/write directions.

SVD Keeps the Strongest Patterns

Real matrices are usually composed of many patterns of varying strength. Singular value decomposition, SVD, can decompose any matrix into a sum of patterns, ordered from strongest to weakest, with the patterns mutually orthogonal. Keeping only the strongest r patterns gives the approximation with the smallest error among all rank-r matrices.

The grayscale photo example shows the effect. Keep 1 pattern and only horizontal and vertical stripes remain; keep 20 and a face is recognizable; keep 50 and it is close to the original image. A few strong patterns determine the overall structure, while many weak patterns contribute detail.

Low-rank approximation uses only r patterns to represent a large matrix, written as a tall-thin matrix times a short-wide matrix. A d×d matrix needs d² numbers; representing it with rank r needs only 2dr numbers. When d=4096 and r=8, the count drops from about 16.8 million to 65,536, about 0.4% of the original.

LoRA Freezes the Original Weights and Learns Only New Patterns

LoRA keeps the original weights W, freezes them, and adds an update composed of r patterns beside them: W′ = W + B·A. B is a d×r matrix, A is an r×d matrix, and B·A is exactly the sum of r patterns. Training updates only A and B.

Why not directly perform low-rank approximation on W? Pretrained weights spread strength across hundreds of patterns; keeping only a few would lose a lot of information. LoRA's premise is that, when adapting to a new task, the weight change is concentrated in a few patterns. The LoRA paper provides empirical analysis of the low-rank property of weight updates during adaptation.

Training starts with B=0 and A random. At this point B·A=0, so the model output is the same as the original model; A is nonzero, so B can receive gradients in the first step and training unfolds. After training, B·A has the same shape as W and can be added directly to W, so inference has no extra overhead; A and B can also be saved separately for different tasks and switched as needed. The LoRA paper reports that on GPT-3 175B, compared with Adam full-parameter fine-tuning, trainable parameters are reduced to about one ten-thousandth, and GPU memory demand drops to about one third.

LoRA also has limits. The rank r restricts the expressiveness of the update, and changes that require more than r patterns cannot be fully learned. What it reduces is the state that needs training; the original weights are still stored in GPU memory in 16 bits. The weights of a 65B model alone already require about 130GB.

Quantization Keeps the Patterns and Reduces Bits per Number

Quantization reduces the number of bits each number occupies; the number of numbers stays the same. A b-bit number has 2ᵇ possible values. bf16 uses 16 bits and has 65,536 possible values; 4 bits has only 16 possible values. 4-bit quantization means each weight chooses one of 16 preset levels and only stores the level index from 0 to 15.

Real weights are usually very small, roughly in the ±0.05 range. Sixteen levels spread between -1 and 1 would waste capacity if placed uniformly. The approach is to split weights into blocks of 64, record a scaling factor for each block, namely the weight with the largest absolute value in the block. When storing, first divide the weight by the scaling factor, then take the nearest level and save only the index; when using, multiply the level by the scaling factor to restore. After blocking, one unusually large weight affects only its own block.

The 4-bit format QLoRA uses is called NF4. Trained weights roughly follow a bell-shaped distribution, so NF4 places the 16 levels according to normal-distribution quantiles: dense near 0, sparse at the two ends, and including an exact 0. Most weights fall near 0, so rounding error is smaller.

Each weight takes 4 bits, plus a scaling factor shared by every 64 weights. The scaling factor itself is further quantized to 8 bits, called double quantization. After that, each weight occupies about 4.13 bits on average, about one quarter of bf16. During computation, each block of weights is temporarily restored to bf16 before participating in matrix multiplication, then discarded after use. Storage uses 4 bits; computation still uses 16 bits.

Low-Rank and Quantization Have Different Error Structures

Low-rank approximation deletes weaker patterns, and the retained patterns are completely accurate. Quantization keeps all patterns, with only a small change in each pattern's strength, while adding a layer of tiny errors spread across all directions, close in form to random noise.

Low-rank approximation reduces the number of patterns in a matrix; quantization reduces the number of bits per number. This also explains a seemingly contradictory phenomenon. Pretrained weights are hard to compress with low rank because there are many patterns, and deleting weak patterns loses a lot of information; yet they can be quantized relatively well because all patterns are retained and only a small amount of noise is added.

QLoRA Puts the Two Methods in Different Places

QLoRA combines LoRA with 4-bit quantization. The frozen original weights W are stored in 4-bit NF4 and do not participate in updates, retaining all patterns but with lower precision. LoRA's A and B are stored in 16 bits and participate in training, learning a small number of new patterns.

During forward computation, each block of W is temporarily restored to bf16, multiplied by the input, and then added to the LoRA branch output. During backpropagation, gradients pass through the restored W to A and B; W itself is not updated. Paged optimizers temporarily move optimizer states to CPU memory when GPU memory peaks, avoiding out-of-memory.

Roughly estimating a 7B model by bytes, counting only weights, gradients, and optimizer states, excluding activations: full-parameter fine-tuning about 100GB, LoRA about 15GB, QLoRA about 5GB.

QLoRA also has costs. Every computation must first restore the weights, so training is slower than 16-bit LoRA; quantization introduces a small amount of error, so the quantized model starts slightly worse and LoRA must compensate during training; the trained A and B cannot be added directly to the 4-bit weights and must be merged into restored 16-bit weights or saved separately.

QLoRA's trade-off is clear: frozen weights can be coarse, but adapter updates must be fine. It does not change the goal of fine-tuning; it only spends memory where precision is more needed.