Choosing an AI model: one prompt, 11 models, different results

Who Should Read

Web developers choosing coding-agent models who need to balance cost against quality

Core Mechanics

• Cost differed by over 200x
  • Average credits ranged from 519 for Claude Opus 5 to 2.4 for DeepSeek V4 Flash (0731).
  • The prompt was identical and each model ran three times.
• One Opus run used 1,055 credits
  • Opus's three runs were 253, 249 and 1,055 credits.
  • The 1,055-credit run was about 4x any other run.
  • The author wrote that Opus runs away with credit usage more often than other models.
• GPT 5.6 Sol averaged 141 credits on low effort
  • Netlify runs Sol on low effort by default, an Opus-class model asked to think a bit less.
  • The author judged Sol's basic design intuition better than Claude Sonnet 5 (143 credits average) in this scenario.
• Same model, very different results
  • GLM 5.2 used 15, 42 and 24 credits, and the author said the runs looked like they came from different models.
  • The author advised running cheap models several times before judging them.
  • DeepSeek V4 Flash 0731 also gave mixed results.

Community Reactions

• Three runs is too few, some argued
  • One commenter said a sample size of 1 is worthless because models are probabilistic.
  • That commenter requires at least 5 runs in their own spam-filtering benchmarks.
  • Another commenter also said three runs is highly sensitive to randomness.
• A two-sentence prompt is unrealistic, some said
  • Several commenters said real work uses detailed instructions, so one-shot evals are not representative.
  • One said real constraints like opening hours and addresses would reveal where models fall short.
• Cheaper models looked better to some
  • One commenter said Kimi, GLM and DeepSeek win at quickly finding the menu and address on a phone.
  • Another found Gemini 3.1 and DeepSeek V4 text-first outputs looked less AI-generated.
• Mobile was not tested, one commenter noted
  • They checked with Firefox dev tools and saw large differences between models.
  • They said the Gemini 3.6 site was too heavy to load on a 3G connection.

How to Apply

• Run your own prompt 3-5 times
  • Run one of your production prompts 3-5 times on each of 2-3 candidate models.
  • Tabulate the outputs and the cost to see the variance.
• Rerun cheap models before choosing
  • Run models like GLM 5.2 or DeepSeek V4 Flash several times on the same prompt.
  • Shortlist them if the share of usable outputs is high.
• Split drafting from iteration
  • Use an expensive model only once or twice for the initial draft.
  • Use a cheaper model for follow-up edits to save budget.
• Evaluate with real constraints
  • Put real requirements like opening hours, address and mobile load time into the prompt and checklist.
  • Count each model's misses and compare them alongside cost.

Try It

# Paste into Claude or ChatGPT: a simplified version of the post's comparison (check variance across runs of one prompt)

You are a code reviewer. Compare three outputs that the same model produced from the prompt below.

[Original prompt]
{{original prompt: e.g. Build a one-page site for a neighbourhood coffee shop: opening hours, address, short menu, photo. Nothing changes unless I edit it}}

[Output 1]
{{result 1}}

[Output 2]
{{result 2}}

[Output 3]
{{result 3}}

For each output, make a table covering:
1. Missing requested elements (hours, address, menu, photo)
2. Broken images or bad references
3. Whether the menu and address are easy to find on mobile
4. Over-engineering (database, CMS, etc.)
End with a one-line summary of the items where the three outputs differ most.

Next step: build the same table for three outputs from another model and put the credit or token cost next to each.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://www.netlify.com/blog/one-prompt-11-models-very-different-results/
• https://wiki.roshangeorge.dev/w/images/4/4e/Screenshot_-_Eva...
• https://nehmeailabs.com/right-size