Advanced Quantization Algorithm for LLMs
Summary
AutoRound is an advanced quantization toolkit for LLMs and VLMs that enables ultra-low-bit quantization (2–4 bits) with minimal tuning, leveraging sign-gradient descent and broad hardware compatibility. The repository highlights new features like Block-wise FP8 quantization, MTP layer quantization, and AutoScheme for adaptive mixed-precision, and provides installation, usage, and back-end integration guidance along with related publications.