Neural Networks: Zero to Hero

TL;DR Highlight

Andrej Karpathy teaches everything from backprop to GPT by building it in code — hands-on lectures for engineers who learn best by implementing.

Who Should Read

Software engineers and ML practitioners who want deep intuition for how neural networks and LLMs actually work, not just how to use APIs.

Core Mechanics

Karpathy's lecture series (Neural Networks: Zero to Hero) covers the full stack from basic backpropagation through modern GPT architecture, all built from scratch in Python/PyTorch.
The pedagogical approach: implement everything yourself rather than using libraries as black boxes — you understand autograd by building it, not by reading about it.
The series covers: micrograd (backprop from scratch), makemore (bigram → MLP → attention), and nanoGPT (a minimal but complete GPT implementation).
The content is targeted at engineers with Python knowledge but limited ML background — accessible without requiring a deep math background.
nanoGPT became a widely used reference implementation because it's readable, not just functional.
The lectures are freely available on YouTube and have become a standard self-study resource for engineers entering ML.

Evidence

The series has millions of views and is frequently cited as the best free resource for engineers learning ML fundamentals.
HN discussions of the series are consistently positive, with experienced ML engineers recommending it even to practitioners with existing backgrounds.
nanoGPT's GitHub repo has tens of thousands of stars and is regularly forked for research experiments — evidence of practical utility beyond just education.
Several professional ML engineers noted that working through the series filled gaps in their understanding that years of using high-level frameworks hadn't addressed.

How to Apply

Work through the series sequentially — don't skip micrograd, even if you already use autograd. The implementation details matter for debugging mental models.
After each lecture, try to extend the implementation yourself before looking at solutions — the struggle is where the learning happens.
Use nanoGPT as a starting point for research experiments: it's small enough to fit in your head and modify confidently.
After completing the series, you'll have the foundation to read ML papers directly rather than relying on blog post summaries.

Terminology

Backpropagation (backprop)The algorithm for computing gradients in neural networks — how the model learns which parameter changes reduce the loss.

AutogradAutomatic differentiation — a system that automatically computes gradients through arbitrary computational graphs, enabling backprop without manual calculus.

nanoGPTKarpathy's minimal but complete GPT implementation, designed to be readable and hackable — the simplest code that correctly implements the architecture.