How a giant AI model gets squeezed onto your phone

A modern AI model is basically billions of numbers, called weights, that it learned during training. By default each weight is stored in high precision — often 16 or 32 bits each. Multiply that by billions and you get a model far too heavy to run on a laptop or phone. Quantization is the trick that shrinks it: store each weight using fewer bits, commonly just 8 or even 4 bits.

Think of it like saving a photo as a smaller file. Instead of recording every weight as 0.7182634..., quantization rounds it to a coarser value drawn from a much smaller set of options. Each number loses a little detail, but there are so many weights working together that the model's overall behavior barely changes. The payoff is huge: a 4-bit model can be roughly a quarter the size of a 16-bit one, using less memory and running faster.

The catch is a precision-versus-quality tradeoff. Round too aggressively and the accumulated errors start to show — the model gets slightly worse at reasoning or produces clumsier answers. Engineers pick a bit-width that keeps quality acceptable while making the model small and cheap enough to actually deploy. This is why the same powerful model can run in a data center at full precision and also, in a quantized form, on the device in your pocket.