Run a 1T parameter model on a 32gb Mac by streaming tensors from NVMe
Summary
Hypura is a storage-tier-aware LLM inference scheduler for Apple Silicon that streams tensors from NVMe to run models larger than RAM. It demonstrates running Mixtral 8x7B and Llama 70B on 32 GB hardware with multiple inference modes (full-resident, expert-streaming, dense FFN-streaming), auto-tuning, and an Ollama-compatible API. This approach is valuable for AI tool developers, SMBs, and IT teams evaluating local AI inference on commodity hardware.