Width vs. Depth: Speculating on the Margin
Summary
A technical blog post analyzing throughput optimization for large language models, comparing width (batching random sequences) versus depth (speculating ahead in a single sequence) under memory-bounded conditions. The author presents empirical observations on MoE routing and per-sequence confidence gating, concluding that spending depth can outperform widening batches in many scenarios, and discusses practical gating and calibration strategies.