AI With Rohan

Visual deep dives into how large AI systems are trained, served, and turned into agents.

I’m Rohan, an ML engineer working on agentic AI. I write from-scratch explanations of the systems behind modern AI, with animations and narration.

Start with a series: Building a Language Model from Scratch · Parallelism in AI

Latest posts

Building a Language Model from Scratch - Part 4: 45 Minutes on a B200

cs336

training

systems

language-models

A fixed wall-clock budget turns model training into a throughput problem. The lever that mattered most was a data-loading fix, not an architecture choice.

Aug 9, 2026

Building a Language Model from Scratch - Part 3: Training Dynamics

cs336

training

ablations

language-models

An LR sweep, a batch-size sweep, and four architecture ablations on TinyStories - ranked by how much validation loss each component is actually worth.

Aug 7, 2026

Building a Language Model from Scratch - Part 2: Where the FLOPs Actually Go

cs336

transformers

performance

language-models

Counting every matmul in GPT-2 XL. Attention is only 10% of the compute at context 1,024 - and 63% at 16,384. The crossover is why long context is its own engineering problem.

Aug 5, 2026

Building a Language Model from Scratch - Part 1: The Tokenizer Is a Compression Algorithm

cs336

tokenization

bpe

language-models

Byte-level BPE from scratch: why UTF-8, what the merge loop actually costs, and why the longest token in your vocabulary tells you how clean your corpus is.

Aug 2, 2026

Parallelism in AI - Part 3: Tensor Parallelism

parallelism

distributed-training

tensor-parallelism

Splitting individual weight matrices across GPUs: column- and row-parallel layers, how Megatron-LM shards attention and MLP blocks, and what the AllReduce costs.

Mar 11, 2026