Running Qwen3.8 at 255 tok/s on a single 5090
Summary
Mesh.applied showcases optimized open-source LLMs (DeepSeek-V4.1-Flash-Ternary and Qwen3.8-Flash-Next-NVFP4) using post-training, quantization, and kernel specialization to run fast on local hardware, achieving up to 334 tokens per second on a single RTX6000. The post includes hardware recommendations and the Goethe framework with a GitHub link for cloning. This highlights practical edge/inference improvements for developers and researchers using open-source models.