A fixed wall-clock budget turns model training into a throughput problem. The lever that mattered most was a data-loading fix, not an architecture choice.
An LR sweep, a batch-size sweep, and four architecture ablations on TinyStories - ranked by how much validation loss each component is actually worth.
Counting every matmul in GPT-2 XL. Attention is only 10% of the compute at context 1,024 - and 63% at 16,384. The crossover is why long context is its own engineering problem.
Byte-level BPE from scratch: why UTF-8, what the merge loop actually costs, and why the longest token in your vocabulary tells you how clean your corpus is.
Splitting individual weight matrices across GPUs: column- and row-parallel layers, how Megatron-LM shards attention and MLP blocks, and what the AllReduce costs.
When a model no longer fits on one device: splitting it into stages, the pipeline bubble, and how micro-batching keeps every GPU busy.
Why a single GPU is never enough, how data parallelism splits the batch across devices, and how FSDP shards the model itself to break the memory wall.