Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
Summary
This paper tackles memory pressure in long-context Mixture-of-Experts training by proposing four scheduling/offload strategies that bound GPU memory use while preserving exact losses and gradients. It reports large reductions in peak memory for dispatch, vocabulary projection, and optimizer state, and demonstrates up to 10.4x throughput on models from 120B to 667B parameters.