Show HN: Training a model to identify AI web content from structure alone

Who Should Read

A content or backend developer who needs to filter AI-written marketing content or evaluate the quality of their own generated posts

Core Mechanics

• Word-level detectors are brittle under rewording
  • They catch unedited AI text almost perfectly.
  • The literature documents that they break when text is reworded.
  • A word-level score can't characterize a text or tell which model wrote it.
• Structure alone can tell them apart
  • Structural signatures cover how information is presented, in what order, with what evidence, and in what voice.
  • The paper replicates StoryScope, originally on fiction, on commercial blog content.
• The data: 2,250 human posts vs 11,250 AI posts
  • Human posts are pre-ChatGPT blogs from 268 company domains.
  • AI mirrors come from five frontier models.
• 176 structural features reach 97.0 macro-F1
  • An LLM applies a 203-feature instrument, and only the 176 structural features are used.
  • The score is on held-out companies.

Community Reactions

• Structural features double as an eval, one commenter says
  • You could measure whether a fine-tuned brand model got less sloppy than the live one.
  • The commenter also notes the repeat-three-times pattern was taught in content-marketing courses long before LLMs.
• Were the 19 misclassifications SEO-template posts?
  • A commenter asks whether listicles or "ultimate guides" were involved.
  • If so, the classifier may detect "written to a template" rather than "written by a model".
• Relying on LLMs to set thresholds is questionable, one commenter says
  • It ties reproduction to token spend on a single third-party model.
  • They would prefer deterministic code.
• Several commenters report false positives
  • A fully handwritten blog was scored 50% "likely human written with AI assistance".
  • Another commenter saw false positives on pre-ChatGPT content and objected to treating a restated thesis as a signal.

How to Apply

• Run structure analysis on 3 of your own posts
  • Today, paste 3 human posts and 3 AI posts into the prompt below.
  • Note which items differ between them.
• Add a structure-check step to your generation pipeline
  • If you auto-generate marketing posts, score drafts with an LLM checklist before publishing.
  • Send high-scoring drafts back to a human for rewriting.
• Compare fine-tuned versions by structure score
  • Generate 20 same-topic posts with the new and current model.
  • Don't ship the version that scores more AI-shaped.
• Don't use the score as a sole verdict
  • False positives were reported, so use scores only to pick items for review.
  • Leave the final call to a human.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: a few items instead of 203 features)

Analyze only the STRUCTURE of the text below, not wording. Look at how information is presented.
Score each item 0 (absent) to 2 (clear) with a one-line reason.
1. Does the intro announce what will be covered?
2. Does the conclusion repeat the same points?
3. Are headings and lists uniformly tidy?
4. Is there concrete evidence (numbers, cases, sources)?
5. Is the voice consistent throughout?
6. Are there unexpected structural turns?

[Text]
{{문서}}

At the end, give a total and rate "how AI-shaped" it is: high/medium/low.

Next step: feed in a human post and an AI post on the same topic and compare the scores.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://arxiv.org/abs/2609.15369
• https://github.com/KatyarAILabs/Ookami