Quantization

What Is Quantization?

Model weights are normally stored as 16-bit floats. Quantization maps them to lower precision (8-bit or 4-bit integers), cutting memory footprint by half or more and speeding inference, which makes larger models run on smaller or cheaper hardware. Aggressive quantization can degrade quality, so the practical question is always how low you can go before the eval scores drop below your bar.