Breaking the 1.58-bit Barrier for Ternary LLMs
Summary
The paper analyzes weight distributions in ternary LLMs and shows zeros can account for up to 51.5% of weights, enabling BITCOS, a distribution-adaptive layout that stores weights more compactly than five-trit packing. It reports substantial gains in unpacking efficiency and end-to-end inference across CPUs and GPUs, with up to ~1.28x speedups in matrix-vector ops and up to 1.18x–1.27x decode throughput on multiple platforms.