Dayjob: Tax – evaluating AI agents on 50 professional tax assignments
TL;DR Highlight
On 50 professional tax tasks, the best AI agent passed only 10.8%
Who Should Read
A backend developer deciding how far to trust agents on messy professional document work
Core Mechanics
- 50 environments, 1,789 files
- Agents get client requests and records, then produce returns, workpapers, reviews, or advice.
- Median is 30 files per task, up to 123.
- Best pass rate is 10.8%
- Claude Sonnet 5.5 led at 10.8%, Claude Fable 5.1 at 9.6%, Claude Opus 5.5 at 9.2%.
- It is pass@1, and an attempt must meet at least 95% of the rubric criteria.
- It tests finding the problem
- Existing tax benchmarks give a defined problem to solve.
- DAYJOB checks whether agents surface issues the client never raised.
- The condo case missed a deadline
- Anna moved out in August 2024 and loses the exclusion eligibility around the end of August 2027.
- Astra never flagged it in five attempts, while Opus flagged it in all five.
- An unverified assumption drove the answer
- In a retirement account split, Opus 5.5 assumed the couple was unmarried in all five attempts.
- The records showed Texas cohabitation and a Colorado ceremony, and Astra advised confirming marital status first in four of five attempts.
- Cost versus performance was compared
- Sonnet 5.5 and Opus 5.5 sit at the high-performance end of the measured Pareto frontier.
Community Reactions
- Opinion that a tax-specific harness would change results
- Commenters said it was a general agent given documents, and a well-built tax harness would show the real limit.
- Another said scaffolding that compiles requirements first would easily fix the condo miss.
- Experience reviewing a draft return with Claude
- It caught an exchange-rate error, but its estimated tax calculations were badly off and it could not fix them.
- The takeaway was that it needs checking and is not a dump-and-file tool.
- Experience checking an accountant's work with Codex
- Codex found the accountant-prepared return riddled with errors, with a refund nearly $2k too low.
- It suggests value as a verifier of human work.
- Experience with Gemini on RMD calculations
- Advice was mostly good, but withholding was wrong and deductions were missed.
- It worked better for preparation than for planning or advice.
How to Apply
- Run a hidden-issues-first prompt on your documents
- Try the prompt below today on a contract or policy document.
- Make the model list unasked issues and unverified assumptions before answering.
- Add an assumption check step to agent output
- Require a list of assumptions with verified or unverified status before the final answer.
- Make the agent ask a clarifying question when an unverified assumption could change the conclusion.
- Add unasked-issue cases to your eval set
- Build about 10 cases where the answer is in the records but not in the question.
- Run each five times to measure consistency.
Try It
# Paste into Claude or ChatGPT (simplified version of the article's method: find unasked issues and unverified assumptions)
You are a professional practitioner. Read the materials and answer the client request, but follow these steps before answering.
[Client request]
{{request}}
[Materials]
{{document}}
Instructions:
1. List issues and deadlines from the materials that the client did not ask about but that affect the outcome.
2. List the assumptions your conclusion depends on, and separate those confirmed by the materials from those that are not.
3. If an unconfirmed assumption could change the conclusion, ask a clarifying question before recommending.
4. Mark conflicting evidence and missing supporting documents separately.
Next step: run the same input five times and compare whether any issue gets missed.Terminology
Related Papers
Gemini 4 Carbon
Google unveils Gemini 4 Carbon, a cybersecurity-focused model, but access is limited to select cyber defenders.
Anthropic S-1
Anthropic filed an S-1 to list Class A shares on Nasdaq as 'ANT'.
A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
A fine-tuned Inception v3 beat GPT-6 Astra and Claude Fable 5.1 on 200 skin lesions
America has lost 200k of jobs in media, film and the arts. How much is AI?
US creative-industry jobs fell by over 200k in four years, but AI's share of the blame is hard to prove
No signs of AI payoff outside the tech sector
Revenue per worker and margins rise at the Mag 7 but stay flat across the S&P 493
LLMs are making formal software verification feasible at a much larger scale
Claude proved no-crash properties for 123 open-source libraries and found bugs along the way.
PHP2Shell: AI uncovers critical RCE in PHP
Wiz used AI to turn a decade-old PHP core flaw into a working RCE exploit
Using LLM for confidence scoring is a bad idea
An LLM's self-reported confidence is plausible text, not a measurement; build scores from outside rules.