I'm making LLMs work on small and old devices
Summary
This article describes a project to run various sizes of LLMs on limited hardware (TV box and iPad), detailing kernel patches, 1-bit/ternary weights, MoE, and streaming from flash. It includes benchmark results, bottlenecks, and a practical quick-start guide for building and deploying on edge devices.