Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)
Summary
The post covers Qwen 3.8-27B running on the iPhone 18's Apple Neural Engine, noting 24K context with int8 K/V and FP16 delta nets, and decoding up to 70 tok/s for predictable contexts. It also mentions a draft model dflash-2 at about 2.8 bpw, a Bonsai-2 base re-quantized to 1.5 bpw, and decoding speeds of 15-60 tps. The content highlights mobile AI inference and an Open Source ML library ANEMLL.