90% of US working hours are spent on tasks not covered well by AI benchmarks

TL;DR Highlight

Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

Who Should Read

An LLM app developer who picks models by benchmark scores without knowing whether they cover their own domain's tasks

Core Mechanics

90% of work hours fall outside benchmarks
- Using 29 benchmarks, 90% of US working hours go to tasks with little or no coverage.
- That makes it hard to know how good AI is at real work.
Only 208 of 923 occupations covered
- Just 208 occupations have a task with at least 5 benchmark prompts.
- 6 of 22 job families have no such occupation.
Coverage follows AI exposure and wage bill
- AI-exposed tasks are about 6x more likely to appear in benchmarks than unexposed ones.
- Even controlling for exposure, higher-wage-bill occupations are overrepresented.
Skew exists inside job families too
- The most important tasks are 6–10x more likely to be benchmarked than the least important.
- Lawyer tasks are heavily covered but legal secretary and court clerk tasks barely are.
Task-weighted scores barely changed results

Community Reactions

Benchmarks miss error detection, one view says
- A commenter said real work hinges on catching mistakes while benchmarks focus on making or solving.
'Well covered' really means 'covered at all'
- A commenter noted social worker counseling counts as well covered with 13 prompts.
- They argued a strict definition would push the number higher.
Building AI solutions, one found 90% may overstate
- A commenter in a well-covered family said public benchmarks cover low-value tasks.
- They said automating the easy-to-measure parts sometimes made things worse overall.
It's about the measuring tool, not AI, one view says
- A commenter said it just means there are more tasks than benchmarks.
- Another argued O*NET misses many modern white-collar occupations.

How to Apply

Compare your task list against benchmarks
- Today, list your service's 10 core tasks and mark whether your reference benchmarks cover each.
- Label uncovered ones as unverified.
Build your own eval set with 5+ cases per task
Add error-detection cases to your evals
Check score skew when choosing a model

Try It

copy & run

# Paste into Claude or ChatGPT as-is (simplified version of the article's method: benchmark coverage audit)

I'm a developer building {{service description}}. The core tasks we give the LLM are:
{{task list, one per line}}

The benchmarks I use to choose models are:
{{benchmark names}}

For each task, build this table.
| Task | Related benchmarks | Coverage (none/low/enough) | 5 test cases we should build ourselves |
If you don't know a benchmark, write "unknown" instead of guessing.

Next step: replace the test cases for "none" tasks with real data and build your eval set.

Terminology

O*NETA US Department of Labor taxonomy that breaks 923 occupations into task lists, like job descriptions for each role.

Benchmark coverageHow many benchmark prompts map to a given task. Here, 5 or more counts as covered.

APEXMercor's own benchmark set. It holds private prompts that can be mapped to O*NET tasks.

AI exposureA binary tag for how much a task is affected by AI automation, taken from Eloundou et al. (2023).

SaturatedA benchmark where frontier models score near the maximum, so it no longer separates models.