Train and run transformers directly on Apple's Neural Engine
Summary
Espresso describes running transformers directly on Apple Silicon Neural Engine by compiling MIL programs to private ANE APIs. It claims significant speedups over CoreML, including 1.08 ms per token and a 4.76x overall improvement, with two dispatches for six transformer layers and zero copy I/O. The project also offers training on ANE and a Swift 6.2 implementation with zero external dependencies, while noting that using private ANE APIs may be rejected by the App Store. The article highlights architecture components and a quick start for integration in local projects.