Quantization without the accuracy cliff
INT8, FP8, and AWQ — measuring what you actually lose.
CR
Carlos Ruiz
May 20, 2026·1 min read·139
Quantization halves your memory footprint and can double throughput. The catch is accuracy, and the only honest way to know is to measure on your data.
from awq import AutoAWQForCausalLM
model = AutoAWQForCausalLM.from_quantized("model-awq", fuse_layers=True)INT8 is nearly free for most chat workloads.
FP8 needs Hopper but keeps more precision.
Always eval on a task-specific set, never just perplexity.
CR
Written by
Carlos RuizGPU performance nerd
Squeezing every last token/sec out of accelerators. CUDA, MIG, and kernels.
3 followers · 2 stories