Zero to GPT-2

Zero to GPT-2 is my way of building language models from scratch, starting at a character-level bigram and ending at a 124M parameter GPT-2 trained from scratch on 10 billion tokens of FineWeb-Edu. Every model was written by hand to understand the mechanics, not just to run something, following Andrej Karpathy's Neural Networks: Zero to Hero course and his build-nanogpt reproduction, rebuilt in my own words.

Why I built it

It is easy to import a transformer and call it done. I wanted the opposite: to write manual backprop, batch normalization, residual connections, self attention, byte pair encoding, mixed precision, gradient accumulation, and multi-GPU data parallelism myself, so I actually understood how each piece works. AI was used sparingly and only on the edges, like the dataset download and tokenization script and part of the demo. The models and the reasoning behind them are my own.

The build log

The project follows the course progression, one stage per folder, each building on the last:

• Bigram: a character-level bigram on a names dataset, done two ways, the counting table and a single-layer neural net trained with gradient descent and hand-written backprop.
• MLP: a Bengio-style multilayer perceptron, grown from a basic version to a deeper net, then a GPU and DataLoader rewrite, and a BatchNorm variant, on Shakespeare characters.
• RNN / GRU: a recurrent net built around a hand-written GRU cell.
• WaveNet: a hierarchical model that grows its receptive field in a tree, in the spirit of WaveNet.
• GPT (nanoGPT): a character-level transformer on tiny Shakespeare, the let's-build-GPT step, with self attention, multi-head attention, and residual blocks.
• GPT-2 124M: a from-scratch GPT-2 trained on 10 billion tokens of FineWeb-Edu, with byte pair encoding, flash attention, distributed data parallel training, gradient accumulation, checkpointing, inference, and a live demo.

What I started with, and where I ended up

At the very first optimizer step, the model is guessing uniformly over a 50,000 token vocabulary: the loss sits around 11.0 and the output is pure noise. Over roughly 19,072 steps, one pass over the data, the training loss falls to about 3.1 and the validation loss settles near 3.10.

By the end, the same architecture that started as noise generates coherent English. Prompted with a recipe, the base model continues:

Here is a simple recipe for bread: 1 cup of water. 4 cups of pure white flour. 1 tsp of sugar. 1/3 cup of unsalted butter. Mix all ingredients into the flour mixture, whisk together, and reduce to a simmer.

It is a base model, not instruction-tuned, so it continues text rather than answering questions, but it has clearly learned real language from scratch.

The GPT-2 run

• Data: FineWeb-Edu, sample-10BT, about 10 billion tokens, tokenized into 100 shards (99 train, 1 validation).
• Model: 12 layers, 12 heads, 768 embedding dimensions, 1024 context length, weight-tied embeddings, flash attention.
• Training: bfloat16 autocast, a fixed effective batch of 524,288 tokens held constant with gradient accumulation, a cosine learning-rate schedule with warmup, and gradient clipping at 1.0.
• Infrastructure: two RTX 5090 GPUs on RunPod, with a babysit script that resumes on failure and a checkpoint every 200 steps, so an interruption never costs the whole run. About 7 hours end to end.

What I learned

• How a loss function and manual backprop actually move weights, before any autograd.
• Why initialization, BatchNorm, and residual connections matter for training deep nets.
• The full transformer: self attention, causal masking, multiple heads, and the block structure.
• Byte pair encoding, and why token-level modeling scales past character level.
• Training at scale: mixed precision, gradient accumulation to hit a large effective batch on small cards, learning-rate schedules, and gradient clipping.
• Distributed data parallel training, how gradients sync across GPUs, and how sharding and checkpointing keep long runs safe.
• The full lifecycle: dataset preparation, cloud training, resuming from checkpoints, inference, and shipping a demo.

Where I stopped, and why

Reproducing GPT-2 124M was the goal, and it is done. I could push the validation loss lower with more tokens or longer schedules, but the returns would be incremental and the architecture is now several years old. Reproducing a 2019 model taught me the fundamentals. The natural next step is to apply them to current designs, things like rotary position embeddings, RMSNorm, SwiGLU, and grouped query attention, rather than to keep polishing GPT-2.

Try it

The trained 124M checkpoint lives on Hugging Face , and all of the code, notebooks, and training scripts are in the repository .