bfloat16 has 8 exponent bits (same as float32), giving it a much larger range (±3.4×10³⁸ vs ±65,504). During training, gradients and intermediate activations can spike to large values. float16 would overflow to inf, breaking training, while bfloat16 handles these values safely.

float32: 7B × 4 bytes = 28 GB
int4: 7B × 0.5 bytes = 3.5 GB

That’s an 8× reduction, making the difference between needing a data center GPU and running on a laptop.

Use asymmetric quantization for activations after ReLU, which are always non-negative. Symmetric quantization would waste half the range (the negative side). Asymmetric adds a zero-point offset to shift the range, using all available levels efficiently.

Use symmetric for weights, which are typically centered around zero.

GPU compute cores have tiny caches (a few MB), while models have billions of parameters (tens of GB). For each forward pass, all weights must be loaded from memory to the compute units. The GPU spends most of its time waiting for data transfer, not computing. Smaller weights (via quantization) = faster loading = more tokens per second.

PTQ (Post-Training Quantization): Quantize a pre-trained model using calibration data. Fast (minutes), no training needed, but may lose accuracy.

QAT (Quantization-Aware Training): Train with simulated quantization so weights learn to be quantization-friendly. Takes longer but produces better results.

Choose PTQ when: you need quick results, have limited compute, or are using int8 (mild compression).
Choose QAT when: accuracy is critical, you’re using int4 or lower, or PTQ results are unacceptable.