Jamesob's guide to running SOTA LLMs locally
Summary
Jamesob provides a hands-on guide to running state-of-the-art LLMs locally, detailing hardware choices, costs, and a Docker-based deployment approach. The post covers 4x RTX Pro 6000 configurations, PCIe Gen4 switching, BIOS tweaks, kernel parameters, and PCIe ACS handling to enable GPU peer-to-peer. It also includes practical scripts and references for setting up local STT and model hosting.