Attention to Mamba: A Recipe for Cross-Architecture Distillation
Summary
An academic article detailing a two-stage cross-architecture distillation method from Transformer to a State Space Model called Mamba. The work demonstrates that principled initialization enables retaining performance while transitioning to a more memory-efficient architecture, with thorough ablations and scaling analyses at 1B scale and beyond.