AI Engineer Notebooks – free, framework-free RAG/agents/evals on Colab
TL;DR Highlight
Free Colab notebooks teach RAG, agents and evals from raw API calls on Groq, no frameworks
Who Should Read
A backend developer moving into AI Engineer or FDE roles who wants to build RAG, agents and evals from scratch
Core Mechanics
- Built on raw APIs, not LangChain
- You write the agent loop, RAG and evals from raw API calls first.
- The idea is to learn what LangChain/LlamaIndex do before reaching for them.
- Evals are the spine
- 'Measure before you tune' is introduced in section 02 and returns in every later section.
- Section 04 covers golden sets, LLM as judge, and regression evals as CI when prompts or models change.
- RAG failures are usually retrieval failures
- Section 03 says retrieval quality, not generation, is usually the bottleneck.
- It covers embeddings, hybrid search, rerankers and chunking in order.
- Agents cover tools, guardrails, MCP and Skills
- Section 05 covers the agent loop, tool design, stopping conditions and cost/latency budgets.
- It also covers MCP, the SKILL.md pattern, and harness engineering (context assembly and compaction, tool-result shaping, verification loops).
Community Reactions
- Commenters liked evals coming first
- One said people often throw together a RAG pipeline and judge by skimming three responses in the terminal.
- Evals matter when the harness is the product
- One commenter said for something like voice AI, building a test harness is a non-trivial first step.
- Pushback on excluding frameworks and harnesses
- One commenter asked why applied AI intentionally excludes a harness, since harnessing AI is the job.
- Comments suspected the text was written by Claude
- One commenter pointed to the many em-dashes as a sign.
- Another criticized that the author apparently didn't read what was published.
How to Apply
- Build 10 golden questions for your RAG
- In 30 minutes today, write 10 real user questions with their expected source documents into a file.
- Count how often retrieval returns the right document.
- Check retrieval first on bad answers
- Before editing the generation prompt, log the retrieved chunks for the failing query.
- If retrieval is wrong, try hybrid search or a reranker.
- Add regression evals to prompt and model changes
- Run the golden set in CI on every PR that changes a prompt or model and catch score drops.
- Try a pipeline before an agent
- For fixed-step tasks, build both a pipeline and an agent and compare accuracy and token cost.
Try It
# Paste into Claude or ChatGPT as is (a lightweight version of the notebooks' method: building a golden set)
I run a RAG service. Based on the sample documents below, build an evaluation golden set.
Documents:
{{문서}}
Instructions:
1. Write 10 questions users would realistically ask.
2. For each, give the location of the document with the answer and the expected answer in one line.
3. Include 2 questions the documents cannot answer and mark them 'no answer'.
4. Output as a JSON array.
Next step: run these questions through your RAG and count how often the right document appears near the top of retrieval.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.