Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes
Summary
A practical case study of training a 3.8B LLM for $998, achieving CORE 0.384 on 65B tokens in 43 hours. The article covers the config-driven little-lm framework, hardware tradeoffs (B200 vs H100), FP8 training, and throughput optimizations, plus insights into context length and value embeddings. It offers actionable lessons for solo researchers and SMBs aiming to run small-scale LLM experiments cost-effectively.