A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

Who Should Read

A backend developer deciding between a general LLM API and a trained specialist model for a medical or image classification feature

Core Mechanics

• CNN got 125 of 200, LLMs 84 and 85
  • Inception v3 named the right diagnosis out of 6 for 125 lesions (62%).
  • GPT-6 Astra got 84 and Claude Fable 5.1 got 85 (both 42%).
• The CNN also led with a second guess
  • Correct in first or second choice: CNN 173, GPT-6 Astra 134, Claude Fable 5.1 138.
  • First-choice melanoma out of 30: CNN 13, GPT-6 Astra 8, Claude Fable 5.1 11.
• The CNN is an old model fine-tuned in about 20 minutes
  • Inception v3, published by Google in 2015, was fine-tuned on all layers using 68,011 ISIC Archive images on one H100.
  • Test lesions and other photos of the same lesion or patient were removed from the training and validation data.
• Each LLM had a favourite answer
  • GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas.
  • Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were.
• The biggest gap was on harmless lesions
  • Moles out of 50: CNN 37, GPT-6 Astra 16, Claude Fable 5.1 33.
  • Seborrhoeic keratoses out of 50: CNN 26, GPT-6 Astra 13, Claude Fable 5.1 6.
• The CNN did not win everywhere
  • On basal cell carcinoma GPT-6 Astra got 25/25 and the CNN 24/25.
  • On actinic keratosis both LLMs (10 and 12) beat the CNN (9).
• The author lists clear caveats
  • Only 200 lesions and 30 melanomas, and each LLM case was run once.
  • All lesions were biopsied, making them harder: CNN AUC was 0.96 on ordinary archive images versus 0.86 here.

Community Reactions

• Opinion that a dedicated classifier beats a general LLM
  • Commenters said that with enough samples and good labels, xgboost or a deep model with a classification head is the way to go.
  • They suggested using the LLM for reasoning and explanation, or to build the classifier.
• Objection that it is benchmaxxed and may not generalize
  • One commenter said fine-tuning on a dataset lets a small model win but it may fail on out-of-domain wild images.
  • Another said the test misses LLM potential with notes, patient questions or odd content like tattoos.
• Suggestion to compare with training data in context
  • A commenter wanted the LLM tested with the CNN's training data, or a subset, in the prompt.
  • They noted TabPFN works this way for tabular tasks.
• Experience from a skin leaderboard top model builder
  • They said a much simpler model on a narrow ~11-label task beating an LLM is no surprise.
  • They noted open-weight medical image-language embeddings like MedSigLIP did not reach the top 10.

How to Apply

• Score your current LLM classification against labels
  • Take 100-200 labeled samples, run the same prompt, and tabulate accuracy and per-class errors.
  • Check a confusion matrix for answers piling onto one class.
• Add a specialist classifier as a baseline when you have labels
  • Compare a fine-tuned pretrained model or xgboost on the same test set.
  • Remove samples of the same subject from training before testing.
• Use the LLM to build and explain the classifier
  • Have it write training code, preprocessing and error analysis.
  • Let the specialist model decide and the LLM add the explanation.

Try It

# Paste into Claude or ChatGPT (simplified version of the article's method: diagnose LLM classification results)

I am using an LLM for {{classification task description}}. Below is a CSV with true labels and LLM predictions.
{{paste label,prediction CSV}}

Please do the following.
1. Show overall accuracy and per-class accuracy in a table.
2. Build a confusion matrix.
3. Check whether the LLM over-predicts any class by comparing predicted and actual counts.
4. Point out the riskiest type of misclassification.

Next step: ask for Python code for an xgboost or logistic regression baseline on the same data and compare.

Terminology

Related Papers

• America has lost 200k of jobs in media, film and the arts. How much is AI?

  US creative-industry jobs fell by over 200k in four years, but AI's share of the blame is hard to prove

• No signs of AI payoff outside the tech sector

  Revenue per worker and margins rise at the Mag 7 but stay flat across the S&P 493

• LLMs are making formal software verification feasible at a much larger scale

  Claude proved no-crash properties for 123 open-source libraries and found bugs along the way.

• PHP2Shell: AI uncovers critical RCE in PHP

  Wiz used AI to turn a decade-old PHP core flaw into a working RCE exploit

• Using LLM for confidence scoring is a bad idea

  An LLM's self-reported confidence is plausible text, not a measurement; build scores from outside rules.

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

Related Resources

• https://lynchlab.net/llm-skin-cancer