Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't (and Where They Slow Down)
Summary
The article explains how unified memory enables mini PCs to load 70B models that a high-end GPU cannot fit, but these systems are bandwidth-limited in speed due to memory bandwidth and prompt processing. It contrasts capacity versus bandwidth across several SoCs and GPUs, applies the roofline model to explain performance, and recommends when a mini PC is appropriate versus a discrete GPU, including MoE models and practical caveats about NPUs.