⚡ Building Transformers from Scratch with PyTorch

Hand-write every component of a GPT-style decoder — tokenizer, attention, blocks, training, KV-cache generation — and train your own tiny GPT.

10-day mini-course — Hand-write every component of a GPT-style decoder — tokenizer, attention, blocks, training, KV-cache generation — and train your own tiny GPT.

Like every course on this site, each day is a deep, code-first lesson: an intuition-first explainer, a staged code walkthrough explaining the methodology line by line, visuals, and a 🧪 Your task exercise with a hidden solution. Theory lives in the AI & ML Encyclopedia; here you build.

▶ Start Day 1   📚 All mini-courses

Syllabus

Day	Lesson
Day 1	The Transformer Map: What We’re Building
Day 2	Tokenization & the Data Pipeline
Day 3	Embeddings & Positional Information
Day 4	Scaled Dot-Product Attention from Scratch
Day 5	Multi-Head Attention: Many Perspectives in Parallel
Day 6	The Transformer Block: Where Everything Clicks Together
Day 7	Assembling the Full GPT
Day 8	Training the Tiny GPT
Day 9	Generation & Decoding: Making Your GPT Speak
Day 10	From Tiny-GPT to the Real Thing