All stories
Mercor Benchmark Map Debates Developing

Mercor says AI benchmarks miss 90% of American working hours

A map of 29 selected benchmarks finds sparse coverage across US jobs, exposing a gap between leaderboard scores and the work people actually do.

By AITrending2 min read
Sources (3)

Why now Mercor's Benchmark Map reached an English Hacker News discussion on October 16 at 17:08 UTC. This is newly observed discussion of the research, not an independently established publication timestamp for the undated map.

A leaderboard blind spot, mapped to real jobs

Mercor Research estimates that 90% of American working hours fall on tasks with little or no coverage in its selected AI benchmark index. Its Benchmark Map connects 29 benchmarks to the US Department of Labor’s O*NET taxonomy, covering 923 occupations and 18,797 tasks.

The map counts an occupational task as covered when at least five benchmark prompts map to it. Under that threshold, Mercor reports coverage for 924 tasks and at least one covered task in 208 occupations. Six of O*NET’s 22 job families have no occupation meeting that test.

The research drew a public English Hacker News discussion on October 16. Participants debated how much workplace activity, including coordination and stakeholder management, escapes benchmark tasks. Those comments are individual observations, not a representative labor survey.

What the 90% figure actually measures

The finding is about measurement gaps, not proof that AI cannot perform 90% of work, nor a forecast that a particular share of jobs will disappear. Mercor combines benchmark-to-task mappings with LLM-derived estimates of time spent on each task to calculate working-hours coverage.

Its index prioritizes public, economically realistic, agentic tasks on benchmarks that frontier models have not saturated. It also caps benchmarks per job family to broaden representation. Mercor explicitly says this initial list is incomplete and its results provisional: omitted benchmarks and the selection criteria can push coverage estimates down.

The mapping itself uses a model judge. Human experts reviewed 1,013 APEX prompts and accepted the model’s first-choice task in 76.7% of 1,763 judgments. The map counts additional task matches that were not all independently reviewed; APEX-SWE was tagged after that review.

For teams comparing models, the practical implication is concrete: a high aggregate score can leave important parts of a workflow unmeasured. Mercor’s map offers a way to inspect that gap, while its own limitations argue against turning one coverage percentage into an economy-wide capability verdict.

Related reading