Matrix Orthogonalization Improves Memory in Recurrent Models
Summary
The article presents a method to orthogonalize the memory matrix in mLSTM models during reads to improve noisy associative recall (NAR) in long-sequence tasks. It builds on Muon optimization ideas and reports notable accuracy gains on MAD noisy AR benchmarks, especially at higher vocab sizes, while noting the results come from a small synthetic task and require real-world validation.