Why is Speculative Decoding Fast?
Summary
An accessible breakdown of why speculative decoding speeds up LLM decoding, reframing speed as a memory- and compute-bound tradeoff rather than simply reduced work. It explains how speculative decoding shifts work from per-token generation to parallel verification and data loading, and discusses practical implications for batch size, depth versus breadth, and deployment with tools like vLLM.