I were 17, I'd learn how to build LLMs from scratch

TL;DR Highlight

PG says at 17 he'd skip startups and learn to build LLMs from scratch.

Who Should Read

A backend developer who only uses LLM APIs and wonders whether learning how models work inside is worth the time

Core Mechanics

PG's advice is to build LLMs yourself
- At 17 he'd learn to build LLMs from scratch.
- Then he'd train the most powerful ones he could on whatever hardware he could access.
This isn't a call to start a startup
- PG added he would not try to start a startup at 17.
- He'd build the knowledge foundation for a later startup instead.
Deep LLM understanding breeds better ideas
- He says better startup ideas come from understanding LLMs deeply than from starting companies on what you knew at 17.
Yann LeCun suggested a different path
- He'd work out why LLMs can write essays but not clean a bedroom.
- He'd study topics in college and grad school that help solve that.
- He'd look for methods and architectures beyond LLMs.

Community Reactions

Opinion that building one gives intuition for the next problems
- Commenters said it builds intuition down to the weights and math.
- They said it also shows when "just LLM it" is the wrong choice.
Experience following Karpathy videos and Raschka books
- Someone who already knew PyTorch and neural nets finished in a few evenings and weekend days over a couple of months.
- They said it made frontier concepts like compressed latent attention easier to follow.
Counterpoint on hardware and cost barriers
- One commenter said an 8GB GPU limits you to fine-tuning small models or days per iteration on nanochat.
- They noted $100 buys only about 25 hours on a B200.
Doubts about job market value
- One opinion said there are no jobs for people who understand toy LLMs because so many do.
- Another said someone who can set up an open-weight model and expose an API may be more valuable to companies.

How to Apply

Skim one build-an-LLM tutorial this weekend
- Today, open languagemodelbuilder.com or the how-to-train-your-gpt repo from the comments and read the table of contents.
- Read only the tokenizer and attention parts first.
Map model-swap points in your LLM features
- If training isn't your goal, build experience setting up open-weight models and exposing them via API.
- Write a one-page list of where in your service a model swap could solve a problem.
Write a "should this be an LLM task" checklist
- Draft a checklist for deciding whether new features suit an LLM.
- Fold in the intuition you gain from learning the internals.

Try It

copy & run

# Paste into Claude or ChatGPT as-is (simplified version, since the source is advice rather than a method)

I'm a developer who uses LLM APIs and want to learn to build an LLM from scratch. My level: {{math/PyTorch experience}}. Available hardware: {{GPU/memory}}. Time available: {{hours per week}}.

Make me a 4-week study plan for these constraints. For each week, separate what to read, what to implement, and what to scale down because of hardware limits.

Next step: ask for a code skeleton for the week 1 implementation task.

Terminology

LLM from scratchWriting the tokenizer, attention, and training loop yourself instead of using a ready-made model. Like assembling an engine rather than buying a car.

compressed latent attentionAn attention technique named in a comment as a frontier concept. The source gives no detail.

nanochatA Karpathy LLM training project mentioned in comments. One commenter said running it on a weak GPU takes days per iteration.

B200A high-end GPU that a commenter said research assumes you have. Renting one is expensive for an individual.