Sub-1-Bit LLM Compression via Latent Factorization
Summary
Sub-1-Bit LLM Compression via Latent Factorization introduces LittleBit, a method to compress large language models to sub-1-bit precision by factorizing weight matrices into low-rank latent factors, binarizing those factors, and restoring magnitude with learned scales. It covers LittleBit-2 with Joint ITQ initialization to improve latent geometry, and provides installation, training, evaluation, and supported models. The project is presented as an official implementation with cited papers and an openly available codebase.