An SLM trained on $8 ESP32-S3
Summary
The article documents training a 319k-parameter transformer entirely on an ESP32-S3 (with a memory footprint of about 1.3 MB for weights plus additional memory for gradients and activations). It covers the constraints of edge-training, the Klingon language corpus used, backpropagation implemented by hand, and reproducible build steps with PlatformIO, under an Apache 2.0 license. It showcases on-device learning on ultra-low-power hardware and outlines future directions for scaling or applying to other small languages.