The 90-year-old idea behind JEPA models: Canonical Correlation Analysis
Summary
The article traces Canonical Correlation Analysis (CCA) as the foundational idea behind JEPA models, describing how CCA minimizes embedding prediction error and how JEPA relaxes whitening constraints with non-linear variants. It also discusses the relationship to Deep CCA and the role of isotropic Gaussian regularization (SIGReg) to prevent collapse, with references to foundational papers.