DeepSeek V3 in pure JAX
Summary
DeepSeek V3 is implemented in 190 lines of pure JAX (no Flax or PyTorch), showcasing a full architecture inspired by the DeepSeek-V3 technical report. It features MLA, DeepSeekMoE, auxiliary-loss-free balancing, sequence-wise balance loss, and Multi-token prediction, trained from scratch on a MacBook Air. The repository includes code, data preparation, training, sampling, and visualization tools, with the model described as an architecture demo rather than a strong language model.