Training AI to Paint with Code

Who Should Read

A developer whose LLM-judge scores or RL tuning have plateaued and who wants to fix the reward design.

Core Mechanics

• Painting with code makes the output editable
  • Image models can only be steered by re-prompting.
  • The authors trained Qwen 3.5 35B with reinforcement learning (RL) to write p5.brush JavaScript.
  • The code is the artefact, so it can be edited directly without a prompt.
• Nine signals were saying the same thing
  • The first reward had nine signals, including a compile gate, code length, HPSv3, four quality judges and prompt adherence.
  • The four quality judges and prompt adherence correlated at 0.85 to 0.95.
  • Reward plateaued at 0.65 and every output was a flat clip-art flower with five petals.
• The code length reward saturated by step 30
  • Code length made up roughly a third of the total reward.
  • It produced zero gradient (no learning signal) after about step thirty.
  • HPSv3, the one signal with real variance, was weighted only 0.10.
• Pairwise judging replaced absolute scores
  • Scores from 0 to 10 came back compressed near zero.

Community Reactions

• A view that SVG is a better target
  • One commenter argued SVG is already image as code, so teaching models to emit SVG beats JavaScript that renders pixels.
• Experience that RL is weak at new styles
  • One commenter who trained a model to draw SVG pelicans said most of the behavior came from SFT on the base model.
  • They said RL is very inefficient at style or at novelty outside the distribution.
• A quick capability test for LLM art
  • One commenter asks models to programmatically assign color and alpha values to an 800x800 PNG and gets generative art.
• A similar attempt with voxels
  • One commenter built a similar setup for voxels using Python calling a Rust native module.
  • They said it worked better than hoped but is not perfect.

How to Apply

• Switch your LLM judge to pairwise comparison
  • Plot the score histogram of your current 0 to 10 LLM judge first.
  • If scores cluster, compare each output against two reference samples and ask which is better.
  • Use the win rate as the score.
• Check correlation between reward signals
  • Log each eval criterion's score separately.
  • If two criteria correlate above 0.85, keep only one.
  • Lower the weight of saturated signals or turn them into gates.
• Shrink API docs in the system prompt to an allowlist
  • If the model hallucinates APIs, remove the long reference and keep an allowlist of around eight functions.
  • Run it three times and check for calls to nonexistent functions.
• Build a pool of hand-picked good examples
  • Hand-sort 100 to 200 outputs into love, okay and nope.
  • Use only the love tier as comparison references.

Try It

# Paste into Claude or ChatGPT as is (simplified version of the paper's method: pairwise judging)

You are a strict judge. Below are reference answers A and B to the same question, and a candidate answer C.

Question: {{question}}

[Reference A]
{{good example 1}}

[Reference B]
{{good example 2}}

[Candidate C]
{{answer to evaluate}}

Instruction: Compare C with A and pick which is better, then compare C with B and pick which is better. Give a one-line reason for each. Finish with JSON only: {"vs_A": "C"|"A", "vs_B": "C"|"B", "win_rate": 0|0.5|1}.

Next step: compare the results with your old 0 to 10 scoring and check whether the score distribution spreads out more.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://surya.website/rling-qwen-to-paint-with-code
• https://proceedings.mlr.press/v80/ganin18a.html