Smaller, faster, safer: running Kimi and GLM at scale
Summary
Cloudflare details how they run Moonshot's Kimi K-series and GLM at scale, focusing on memory and latency optimizations: quantizing the KV cache to FP8, compressing weights to INT4, and protecting the shared KV cache with integrity checks. They use an open-source inference framework (SGLang) and provide benchmarks across concurrency levels, highlighting throughput gains with minimal accuracy loss. The article emphasizes deploying scalable, cost-efficient large-model inference without sacrificing model quality.