Ferrox: Building a Rust Inference Engine That Matches llama.cpp
Summary
This article introduces Ferrox, a pure-Rust inference engine designed to run open LLMs locally (including dense models and MoE) on CPU, Apple Metal, or CUDA. It explains architecture decisions (GGUF loading, memory-mapped weights, on-the-fly dequantization), CLI and OpenAI-compatible server usage, and provides benchmarking results against llama.cpp to substantiate performance claims. It also discusses current limitations and ongoing work, such as CUDA support and Metal-specific optimizations.