Width vs. depth: speculating on the margin
Summary
This Doubleword blog post analyzes speculative decoding for LLM inference, comparing width (batched random sequences) versus depth (speculating ahead along a single draft). It presents empirical findings on MoE routing, confidence gating, and the potential throughput gains from uneven depth across a batch, and discusses practical implications for inference engines, including ragged speculation and memory-bound versus compute-bound considerations.