Janas: Run LLMs larger than RAM by streaming experts from SSD
Summary
The Janas project describes a C-based toolkit that lets running large mixture-of-experts LLMs on ordinary hardware by streaming model experts from an NVMe SSD while keeping active components in RAM. It benchmarks multiple Qwen and Gemma models against llama.cpp on a 32 GB RAM laptop, highlighting speedups and the experimental nature of the tool. The article covers hardware requirements, usage via janas-chat and janas-bench, and the intention to push local AI capabilities on SMB-grade systems.