LLM inference infrastructure for a systems audience
Summary
This article offers a systems-focused overview of LLM inference infrastructure and the serving runtimes powering large-model deployments. It targets infrastructure engineers with limited ML background and emphasizes performance, scalability, and hardware utilization, highlighting techniques like batching, KV caches, model sharding, and I/O-aware kernels. It notes the piece is opinionated and iterative, inviting contributions to keep it living and useful.