Detecting LLM-Generated Texts with “Classical” Machine Learning

TL;DR Highlight

Seven TF-IDF+LinearSVC classifiers voting flagged LLM text at about 85% per-sentence accuracy.

Who Should Read

A backend or applied-ML developer who wants to build an LLM-text detector or understand how 'AI checkers' work.

Core Mechanics

Classical ML can catch AI text
- As of early 2026, mainstream LLM text shows strong statistical patterns that traditional ML models can separate from human writing.
- The author suspects many commercial 'AI plagiarism checkers' work this way.
Data came from summarize-then-regenerate
- Nearly 10,000 human texts from 2010–2022 (pre-ChatGPT) were sampled.
- An LLM summarized chapters and rewrote full articles from the summaries, giving seven sets of AI samples from seven models.
TF-IDF + LinearSVC hits about 85% per sentence
- Texts were split into sentences and trained with TF-IDF → LinearSVC.
- Per-model sentence accuracy ranged 0.8436–0.8940, and the pruned web version lost about 1 point on average.
Seven binary classifiers beat one 8-class model
- A single human + 7 AI classifier got only about 50% accuracy.
- Seven per-model binary classifiers flag a sentence as AI when 2 or more agree.

Community Reactions

Opinion that detection is a lost cause
- One commenter said text is too information-poor to carry a reliable provenance signal and called detection tarot-card reading.
- They argued text is undetectable today unlike images.
Opinion that detectors become training signals
- Commenters said releasing a detector gives LLMs a signal to optimize, and cheap patterns will be used for retraining.
- Another theory was that labs RL models to output easily classifiable text to avoid model collapse.
Firsthand note from someone at Pangram
- A commenter said the synthetic-mirror data method matches what Pangram uses.
- They said perplexity looked SOTA in 2022 because people tested on open or near-open models, and many papers used an undertrained RoBERTa baseline.
Small encoders and English use cases
- One commenter built an on-device iOS app with a small encoder-only model reaching AUROC 99.81 on RAID-bench.
- Another wanted an English version as a browser extension scanning every paragraph.

How to Apply

Build a baseline binary classifier on your own domain text
- Collect a few hundred human texts and pair them with LLM summarize-then-regenerate samples.
- Split into sentences, run TF-IDF → LinearSVC, and check accuracy and F1 first.
Vote across per-model classifiers instead of one
- Train a separate binary classifier on samples from each LLM you care about.
- Flag a sentence as AI when 2 or more agree and use the share of flagged characters as the score.
Set the threshold from false positive rate on lots of human text
Export to JSON to run in the browser without a server
- Store the vocabulary weights as JSON and implement TF-IDF+SVM in JS.
- When trimming features, check the shift in human-text detection rates, not just accuracy.

Try It

copy & run

# Run in a shell (pip install scikit-learn). Simplified version of the article's method: sentence-level TF-IDF→LinearSVC

# human.txt, ai.txt: prepare files with one sentence per line
import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.metrics import accuracy_score, f1_score

def load(p):
    return [l.strip() for l in open(p, encoding='utf-8') if len(l.strip()) > 5]

h, a = load('human.txt'), load('ai.txt')
X = h + a
y = [0]*len(h) + [1]*len(a)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)

clf = make_pipeline(TfidfVectorizer(analyzer='char_wb', ngram_range=(1, 3), max_features=500000), LinearSVC(C=1.0))
clf.fit(Xtr, ytr)
pred = clf.predict(Xte)
print('acc', accuracy_score(yte, pred), 'f1', f1_score(yte, pred))

# Next: train one classifier per LLM and flag text when 2+ vote AI

Terminology

TF-IDFA way to turn text into numbers by boosting distinctive words and discounting common ones. Think of it as a per-word importance scoreboard.

LinearSVCscikit-learn's linear SVM classifier that finds a boundary separating two classes. It draws the widest possible margin between the groups.

PerplexityA measure of how well an LLM predicts the next word of a text. Easy-to-predict text is suspected to be AI-written.

F1A classification score that combines precision and recall. It reflects both false positives and misses.

False positive rateHow often human text is wrongly labeled as AI. This is the weak spot of any detector.