Flash-Moe: Running a 397B Parameter Model on a Mac with 48GB RAM
Summary
The Flash-Moe project presents a pure C/Metal inference engine that runs a 397B Mixture-of-Experts model on a MacBook Pro with 48GB RAM, achieving production-like throughput without Python or frameworks. It details the architecture, techniques like SSD expert streaming, FMA-optimized dequantization, and deferred GPU compute, and compares 4-bit vs 2-bit configurations with emphasis on production readiness (4-bit) and tool-calling limitations (2-bit).