Forward propagation of errors through time
Summary
This post introduces Forward Propagation of Errors Through Time (FPTT) as a forward-gradient alternative to backpropagation through time for training recurrent neural networks. It describes a warm-up phase to initialize error dynamics, a forward update scheme using inverted Jacobians, and a multi-layer extension, along with complexity analyses and experimental results on a sequential MNIST variant. The work highlights significant numerical stability challenges and concludes that, despite theoretical promise, practical use is limited by instability in forgetting regimes.