You could have invented transformers from memory
Summary
Alex Wang's post develops an intuition for memory stores in neural networks, tracing a path from outer-product (linear) memory to optimal linear stores, then to nonlinear MLP-based memory stores and self-attention. It quantifies memory capacity through interference-to-signal ratios and shows how increasing nonlinearity or using self-attention dramatically extends recall capacity, culminating in a non-parametric memory store that scales with context length.