in progress
LLM from scratch
Building a ~125M parameter Python code LLM in plain PyTorch, one concept per stage, from tokenizer to serving.
~125M params
Problem
Libraries hide the parts that matter most when you optimize models. The goal is understanding, not a leaderboard score.
Approach
A 25-stage curriculum where every component is written by hand before any shortcut is allowed:
- Tokenizer, data pipeline, and a dense GPT.
- Mixture of Experts, then pretraining.
- Post-training: SFT, LoRA, reward models, DPO and GRPO, distillation, quantization.
- Evaluation and serving.
Each stage answers “why does this exist and what breaks without it?” before “how do I code it?”.
Stack
PyTorch, Python.