Series

Building a Language Model from Scratch

A tokenizer, a Transformer, a training loop, and a 45-minute leaderboard run, written from PyTorch primitives while working through Stanford’s CS 336. Every post is built around numbers I measured.

Building a Language Model from Scratch - Part 1: The Tokenizer Is a Compression Algorithm

Byte-level BPE from scratch: why UTF-8, what the merge loop actually costs, and why the longest token in your vocabulary tells you how clean your corpus is.

7 min

Building a Language Model from Scratch - Part 2: Where the FLOPs Actually Go

Counting every matmul in GPT-2 XL. Attention is only 10% of the compute at context 1,024 - and 63% at 16,384. The crossover is why long context is its own engineering problem.

7 min

Building a Language Model from Scratch - Part 3: Training Dynamics

An LR sweep, a batch-size sweep, and four architecture ablations on TinyStories - ranked by how much validation loss each component is actually worth.

8 min

Building a Language Model from Scratch - Part 4: 45 Minutes on a B200

A fixed wall-clock budget turns model training into a throughput problem. The lever that mattered most was a data-loading fix, not an architecture choice.

7 min

Parallelism in AI

How training is split across many GPUs, from replicating the whole model to slicing weight matrices. Each post is animated and narrated.

Series

Building a Language Model from Scratch

Building a Language Model from Scratch - Part 1: The Tokenizer Is a Compression Algorithm

Building a Language Model from Scratch - Part 2: Where the FLOPs Actually Go

Building a Language Model from Scratch - Part 3: Training Dynamics

Building a Language Model from Scratch - Part 4: 45 Minutes on a B200

Parallelism in AI

Parallelism in AI - Part 1: Data Parallelism & FSDP

Parallelism in AI - Part 2: Pipeline Parallelism

Parallelism in AI - Part 3: Tensor Parallelism