Language models for text classification: From bag-of-words to Jev

Who Should Read

A backend or ML engineer deciding whether to use an LLM or build a dedicated classifier for text classification.

Core Mechanics

• Jev is a classifier, but not just a classifier
  • The author says GPT and open-weight LLMs can do the same classification, but Jev is faster and cheaper.
  • For a narrow, well-defined problem, he expects it not to beat a special-purpose classifier.
  • Its selling point is being far more general than task-specific models.
• Bag-of-words plus logistic regression is still the baseline
  • It counts each vocabulary word into a fixed-size vector that classic classifiers can consume.
  • It loses word order, so "the dog bites the man" and "the man bites the dog" look identical.
  • On IMDb (balanced dataset) it reached 89.9% accuracy.
• The LSTM RNN actually lost on the same data
  • An LSTM trained from scratch got 85.66% on IMDb, below the 89.9% of bag-of-words logistic regression.
  • Its higher training accuracy indicated substantial overfitting.
  • ULMFiT (2018), which pre-trains then fine-tunes, reached 95.4% test accuracy on IMDb.
• Text CNNs run in parallel and do fine

Community Reactions

• Opinion that hype outran reality
  • Some commenters said people uncritically cited AGI, system 1, and "no hallucinating" claims.
  • Game demos (Doom, Minecraft) rarely explained they used game state as input, which misled people.
• Experience: Jev looked promising for email classification
  • One commenter is comparing email classification strategies and says Jev looks promising.
  • They wrote up the comparison at housecat.com/blog/classifying-email.
• View that this is the ChatGPT moment for classification
  • Commenters said the excitement is cheap classification of many text types without fine-tuning a custom classifier per task.
• Addition: continuous bag-of-words bridges the gap
  • A commenter pointed to averaging or summing word embeddings per document as a middle step.
  • They said pretrained embeddings help on small datasets, and fastText works this way.

How to Apply

• Build a logistic regression baseline first
  • Run TF-IDF plus logistic regression on your labeled data within 30 minutes and record accuracy.
  • Use that number as the yardstick for LLM prompting or fine-tuned models.
• Shortlist ModernBERT fine-tuning for fixed-label tasks
  • If your labels are fixed, compare prompt classification and ModernBERT fine-tuning on the same test set for accuracy, latency, and cost.
  • Start with minimal tuning and check the default results first.
• Decide on Jev by the scope of your task
  • Keep a dedicated classifier for a narrow, single, well-defined task.
  • For multi-purpose classification with changing labels, add a general classification API like Jev to the candidates and measure cost and speed.

Try It

# Paste into Claude or ChatGPT (simplified version of the article's idea: choosing a classification approach)

I'm building a text classification feature. Compare the options given these conditions.
- Task: {{task description}}
- Label list: {{labels}}
- Number of labeled examples: {{count}}
- Daily request volume and latency target: {{volume and latency}}

Compare four options: (1) TF-IDF + logistic regression, (2) ModernBERT fine-tuning, (3) LLM prompt classification, (4) a general classification API.
Put expected accuracy, speed, cost, and maintenance burden when labels change in a table, and suggest the order of baselines to try first.

Next step: ask it to write scikit-learn code for option 1 matching {{my data format}}.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.

Related Resources

• https://magazine.sebastianraschka.com/p/classifier-history-and-jev
• https://housecat.com/blog/classifying-email
• https://sebastianraschka.com/llms-from-scratch/
• https://sebastianraschka.com/reasoning-from-scratch/