Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

TL;DR Highlight

RL-training an agent that writes training jobs lifted its reward from ~0.0 to a ~0.63 peak.

Who Should Read

Backend or ML platform developers curious about improving agents with reward signals and what automating small-model training actually costs.

Core Mechanics

The agent writes training jobs itself
- Given a training task, the agent writes a full prime-rl job with environment, reward, dataset and hyperparameters.
- The job trains a small Qwen3-0.6B/1.7B model with GRPO on Runpod GPUs.
There are two nested RL loops
- The outer loop trains a Qwen3.6-35B-A3B (LoRA) agent with Tinker.
- The inner model's hidden-eval improvement becomes the outer loop's reward.
Reward rose in two rungs
- Reward climbed from ~0.0 to a ~0.63 peak over 54 steps.
- The early gain came from cutting validation failures and jobs dying on GPUs.
- Later, job grade rose 0.30 → 0.41 and trained models' hidden-eval scores went from ~0.04 to 0.22–0.48.
It partly transferred to an unseen task family
- Mean reward on the held-out triage family went 0.399 at step 0 → 0.545 at step 34.
- It eased to 0.492 at step 54, a rise then plateau.

Community Reactions

Concern about reward hacking
- A commenter asked how to stop the agent from leaking the eval distribution through training data instead of teaching a general skill.
Criticism of vibe-coded RL systems
- One commenter argued that vibe coding a complex RL system leaves you not knowing what is benchmarked or when it breaks.
- They asked for the limits and failure points to be explained.
Question about recursive degradation
- A commenter wondered whether recursively trained models degrade after a few generations.
Reaction that agents already build training pipelines
- One comment noted agents can already spin up training pipelines and asked whether the advantage is the framework for extra compute.

How to Apply

Log reward components separately
- Today, aggregate validation pass rate, task success and final quality score separately from your agent eval logs.
- Find the bottleneck stage and fix in that order.
Add a baseline-lookup tool to the agent
- If your agent picks models or settings, expose baseline scores as a function-call tool.
- Grade by uplift over baseline rather than absolute score.
Keep a cost ledger per call
- Log tokens and USD per LLM call to JSONL and enforce a per-episode budget.
- Check total cost per batch.
Reserve a held-out task family
- When running agent improvement experiments, set aside one task family from training to check generalization.

Try It

copy & run

# Run in a terminal (needs uv and OPENROUTER_API_KEY; offline tests + one episode from the README)

git clone https://github.com/Danau5tin/ai-trains-ai
cd ai-trains-ai

# Put OPENROUTER_API_KEY=... in .env
uv sync
uv run pytest

# Run one episode with a frontier agent
uv run at-episode --task examples/tasks/calc_chain_v1_fast.json --model qwen3.6-27b

# Next: open the trajectory and workspace saved in runs/<run_id>/ and read the training job the agent wrote

Terminology

GRPOAn RL method that samples several answers to the same problem and reinforces those that did better relative to the group, like grading on a curve.

LoRATrains a small adapter instead of changing the whole model, like adding sticky notes instead of rewriting the book.

TinkerThinking Machines' managed RL training API, so you don't manage training infrastructure yourself.

prime-rlA framework for GRPO training. Here it trains the small model from the agent-written config.

Held-out task familyA set of tasks excluded from agent training and used only to test generalization, like an exam on unseen question types.