Are AI labs pelicanmaxxing?

TL;DR Highlight

1,008 SVGs over a 48-cell animal×vehicle grid show no pelican-on-bicycle-specific training.

Who Should Read

An LLM API developer who picks models based on informal benchmark results shown at each new release

Core Mechanics

A 48-cell grid checks whether pelicans stand out
- 8 animals × 6 vehicles gives 48 prompts, and pelican+bicycle is just one cell.
- Seven models drew 3 samples per prompt for 1,008 SVGs, and an LLM judge scored them 1-5.
Neither pelicans nor bicycles ranked high
- Pelican placed 6th of 8 animals, behind cat, whale, raccoon, heron and antelope.
- Bicycle placed 5th of 6 vehicles, and pelican+bicycle ranked 42nd of 48 combinations.
Adjusting for difficulty shows no special boost
- A regression absorbing per-combination difficulty gave per-lab pelican effects of -0.11 to +0.14 points, smallest p = 0.25.
- Only Gemini 3.5 Flash's bicycle effect was significant (p=0.022), which is about what chance gives over 21 tests and fails the Bonferroni threshold (0.05/21 ≈ 0.002).
21/21 right-facing pelican bicycles isn't unusual
- 60% of all 1,008 images face right, including 81% for bicycles and 78% for pelicans.
- Antelope+scooter and pelican+scooter hit 20/21 and heron+bicycle 19/21.

Community Reactions

Right-facing bikes come from photo conventions, one commenter says
- The drivetrain is on the right side, so bike sales photos are shot from the right, and training data likely reflects that.
- Another observed that bikes always face right and riders' legs sit only on the right, suggesting weak structural understanding.
A counterargument says the tables do show training traces
- One commenter argues averaging 100% right-facing into a 60% overall figure is statistically unsound.
- They also say a single-model judge without a clear rubric makes the method unreliable.
Otter-on-a-plane looks specifically trained, one commenter notes
- GLM 5.2 and DeepSeek V4 put animals standing on top of planes, except otters, which sit in seats.
- The commenter links this to Ethan Mollick's 'Otter On A Plane Using WiFi' benchmark.
Labs may be strengthening SVG generally, one commenter suggests
- They guess labs overtrain on an SVG RL environment (a reinforcement-learning training setup).
- They cite the Gemini 3.1 Pro announcement, where only animated SVG improved a lot.

How to Apply

Build a mini grid benchmark for your service
- Today, vary your frequent prompt along 3-4 topics and 3-4 conditions to get a grid such as 3×4=12.
- Run each candidate model 3 times and check whether one cell is unusually good.
Validate with variant prompts instead of famous examples
- When choosing a model, run 10 or more variants of the same format, not just one popular example.
- Suspect overfitting if the famous example scores far above the variant average.
Split LLM judge scores by criterion
Score subject, tool and action coherence separately 1-5 in the grading prompt and average them.
- Use two judge models and score the same image twice to check consistency.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the article's method: grid benchmark design)

My service's core task is {{task description}}. Design a grid evaluation to check whether a model was tuned only to a famous example for this task.

Conditions:
1. Propose two axes. One has 6-8 values of {{axis 1: e.g. topic}}, the other 4-6 values of {{axis 2: e.g. format}}. Mix values similar to the famous example, easy ones, hard ones, and completely different ones.
2. Write a prompt for each combination using the same sentence template.
3. Give a 1-5 rubric split into 3 criteria, with a one-line scoring rule for each.
4. Explain step by step how to judge whether the famous combination scores above average after adjusting for difficulty.

Next step: run the prompts 3 times per model, paste the results as a table, and ask for analysis.

Terminology

benchmaxxingOptimizing a model to score high on popular benchmarks. Like studying the exam questions in advance.

pelicanmaxxingA joking term for tuning a model specifically to the 'pelican riding a bicycle' SVG benchmark.

Fixed-effects regressionA statistical method that absorbs each combination's inherent difficulty so you can compare after removing it. Like adjusting scores from tests of different difficulty.

Bonferroni correctionDivides the p-value threshold by the number of tests, because running many tests produces chance significant results.

LLM judgeUsing an LLM to score outputs instead of humans. With one judge, its taste leaks into the results.