Gemini 4 Argon for Robotics

TL;DR Highlight

Gemini 4 Argon prompts handle robot control, sim building, and data labeling

Who Should Read

Developers new to robotics who want to try multimodal LLMs for robot control or automated MuJoCo simulation setup

Core Mechanics

ERQA 87%, top among frontier models
- On the Embodied Reasoning (ERQA) benchmark of 3D physical and spatial problems, Argon scores 87% with pure reasoning.
- The post claims this beats every other frontier model.
Robot control via prompts
- A Spot quadruped practiced in a MuJoCo digital twin before stepping on a real foot-pedal trash bin.
- A simple arm used camera images and obs.json, issuing curl commands over several turns to set a mug upright.
Code-as-Policies style control
- The prompt lists primitives like robot.grasp, robot.place, robot.goto plus coordinate conventions.
- After each turn the model checks wrist_cam.png and calls look_at again on grasp_failed.
Real2Sim from video and sketches
- From teleoperation video it built a MuJoCo scene and iterated until camera views and contact forces matched.
- Egocentric human video was retargeted to a 76-DoF humanoid, and a hand-drawn sketch became a mujoco_warp scene.

Community Reactions

Experience: Argon avoided obstacles better
- A newcomer who wired vision LLMs into a Unitree humanoid said Opus and Astra crashed into tables and beds.
- Argon 4 walked slowly to the table and described its objects, built in a few hours without LIDAR.
Opinion: Argon won't drive robots directly
- A snow-robot builder thinks such models will mainly generate data to distill into smaller models.
- They also noted that LLMs commanding predefined actions was seen in Stanford's Homebody.
Pushback: on-device is far away
- One commenter argues that even with steady efficiency gains, battery-powered Argon-class robots are about 10 years out.
- They guessed cheap home robots in 5-7 years.
Opinion: underwhelming performance
- One comment says it is not impressive given Google's resources and not as good as Figure 3.
- Another criticism is that Unitree robots do parkour while this demo robot is hung from a hook.

How to Apply

Paste the Odd Ball prompt into an LLM today
- Feed the original 12-ball Prompt 1 to a coding-capable LLM to generate a MuJoCo environment.
- Check that the script runs and the reward logic is correct.
Put primitives and coordinates in the prompt
- When an LLM controls a robot or agent, list callable primitives and the coordinate system in the system prompt.
- Send an observation image and state JSON each turn, with explicit failure handling.
Add no-cheating rules to sim prompts
- List forbidden actions in auto-generated simulation prompts.
- The post bans external forces, welds, disabled collisions, and copying logged readings.
Request hindsight labels for demo videos
- Ask for 4-level JSON labels on your demonstration videos: goal, subtasks, atomic primitives, quality weights.
- Downweight failed segments to filter training data.

Try It

copy & run

# Paste into a multimodal LLM with video input such as Gemini (simplified version of the original prompt)

[Attached: {{robot demo video.mp4}}]
Produce a time-aligned multi-granularity hindsight annotation JSON with four levels:
1. G1_episode_instruction: Concise imperative sentence describing the overall manipulation goal.
2. G2_subtasks: Contiguous semantic subtask phases specifying which arm(s) anchor vs. fold/rotate the object.
3. G3_atomic_primitives: Fine-grained contact-level primitives (3-15s) aligned with grasp closure, release, or retry attempts.
4. quality_control_and_reweighting: Hold task_progress_pct flat during non-progressing slip/retry loops and downweight them (0.15x-0.25x) for Behavior Cloning.

Next step: check that the timestamps in the output JSON match the actual scenes in the video.

Terminology

MuJoCoA robot physics simulator. Like a game engine that computes contacts and forces, a sandbox for robots.

Real2SimBuilding a simulation from real video or photos. It's like making a 3D copy of reality.

digital twinA copy of a real object or environment recreated inside a simulator.

VLAVision-Language-Action model. It takes images and instructions and outputs robot actions.

Behavior CloningLearning by imitating human demonstrations. Bad demos lead to bad imitation.

value functionA function scoring how far a task has progressed, like a game's progress bar.