GPT-6 Astra on robot arms
TL;DR Highlight
GPT-6 Astra on robot arms placed the block 95% of the time at about half the cost
Who Should Read
Developers wiring LLM agents to physical tools who need to compare success rate and cost across models
Core Mechanics
- Astra placed the block 95% of the time
- It succeeded in 19 of 20 trials.
- Fable 5.1 got 8/20 (40%) and Fable 5 got 1/20 (5%).
- Puzzle insertion was only 10% for Astra
- It completed 2 of 20, the same as Fable 5.1 (2/20).
- It reached the groove but stalled at the final insertion step.
- Astra was faster and cheaper
- The bowl task took 2.5 min and about $0.94 per run versus 6.8 min and $2.12 for Fable 5.1.
- Output tokens on the bowl task were 2.1k for Astra versus 12.9k for Fable 5.1.
- Same agent policy for all models
- Bimanual YAM arms got three camera views plus state each turn and moved via absolute end-effector poses (move_to).
- Settings were medium thinking effort, a 20-LLM-call budget and a 25% speed cap.
- Scoring used a 0-4 stage rubric by a human
Community Reactions
- Critique: small experiment leaning on an IK pipeline
- One commenter said it is a single block-picking test relying on an IK controller with no comparison to VLA models.
- They added that hierarchical designs with a VLM planner plus a VLA controller are the recent trend.
- Concern that cost is too high
- One commenter said $2 to put away a block is very expensive labor.
- Experience: Astra wrote poor robot-arm code
- One user asked it to build a skill for a toy tank's arm and said it got servo directions and gripper range wrong and mostly wrote tests.
- They said it cost more than Sol 5.6 and burned 1000-1250 credits in about 2 hours.
- Experience: Astra computer use impressed
- One user left it running overnight in Codex with computer use and woke up to a finished game with assets.
How to Apply
- Build a stage rubric for your agent today
- Read your failure logs and define stages like contact, lift, position and done.
- Record where each run stops, not just pass or fail.
- Compare models by cost per run
- Run the same task 20 times per model and tabulate success rate, time, output tokens and cost per run.
- Use list price and note whether caching applied.
- Match conditions and write down limits
- Use the same equipment, time window and reasoning effort.
- Hide the model name from the grader and document any conditions you could not match.
Try It
# Paste into Claude or ChatGPT (simplified version of the article's method: stage-based scoring)
I'm automating {{task description}} with an LLM agent. Below are logs of failed runs.
{{run logs}}
Please do the following.
1. Write a 0-4 stage rubric for this task (0 no attempt, 1 started, 2 midway, 3 nearly done, 4 completed).
2. Score each log by the highest stage reached and give a one-line reason.
3. Find the stage where runs stall most and suggest fixes.
Next: have two models score the same logs and compare the results.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.