AI agents top out at 10.8% on Surge's full-job tax benchmark

Knowing the tax rule is only part of finishing the job. Surge AI’s DAYJOB: Tax benchmark asks agents to investigate a client’s records, discover consequential issues the client never raised and deliver usable professional work. In Surge’s October 27 report, the leading system, Claude Sonnet 5.5, passes 10.8% of attempts under the benchmark’s strict rubric.

Claude Fable 5.1 follows at 9.6% and Claude Opus 5.5 at 9.2%, according to Surge. These are author-reported pass@1 results: an attempt passes when it satisfies at least 95% of the task’s criteria. The figures are not the percentage of ordinary tax questions answered correctly, nor a measurement of every model’s usefulness for supervised tax work.

The problem can be hidden in the files

The full benchmark contains 50 assignments across eight areas of tax practice and 1,789 workspace files. Agents may need to prepare returns, assemble workpapers, review a disputed position or write client advice. Supporting records include PDFs, spreadsheets, correspondence and less tidy evidence such as handwritten receipts and field dispatch logs.

In one example, a client’s proposed condo-sale dates both qualify for an exclusion, but a later deadline could change eligibility. Surge says Astra missed that deadline across five attempts while Opus identified it in all five. Another example reverses the pattern: Opus repeatedly built retirement advice around an unverified marital-status assumption, while Astra recognized the need to verify it in four attempts. The examples illustrate different failure modes rather than a single universal model ranking.

Two public samples, not the whole benchmark

The English dataset card says the public repository contains two sample tasks. Evaluating the complete 50-task set requires contacting Surge. The materials include synthetic documents modeled on professional records; they should not be described as a public dump of real client tax files.

Tasks use Harbor format in a Linux environment. During execution, network access is limited to explicitly allowed model API hosts, while browsing and package downloads are blocked. The documented setup requires model and grading credentials, and results on the two samples alone are not a full benchmark score.

The report drew fresh English Hacker News discussion on October 28 about review workflows, mistakes and accountability. That discussion provides a current attention signal, not independent validation of Surge’s results. AITrending has read the original report and dataset card but has not reproduced the evaluation.