microgpt
Summary
Karpathy presents microgpt, a tiny, dependency-free GPT-like model implemented in about 200 lines of Python. The article explains the dataset, a simple tokenizer, a hand-rolled autograd engine, a GPT-2–style architecture, and a training loop with Adam, plus how to run inference and generate names. It also contrasts microgpt with production LLMs and discusses data, tokenization, optimization, and the engineering steps needed for larger-scale models.