Dayjob: Tax – evaluating AI agents on 50 professional tax assignments
Surge AI's DAYJOB: Tax benchmark challenges AI agents with real-world tax scenarios, emphasizing professional judgment over rote calculations. Its findings reveal frontier models still struggle with nuanced tasks despite excelling at data processing. This sparks a lively Hacker News debate on AI's disruptive potential in an industry known for complexity and human fallibility.
The Lowdown
Surge AI has introduced DAYJOB: Tax, the latest in its benchmark series designed to evaluate AI agents on professional-level assignments. Unlike other benchmarks that focus on specific questions or calculations, DAYJOB: Tax assesses an agent's ability to handle end-to-end responsibility in complex, real-world tax situations.
- Comprehensive Evaluation: The benchmark comprises 50 assignments across eight diverse areas of tax practice, from individual taxes to corporate transactions and international tax. Each assignment involves intricate scenarios and a wealth of unstructured data.
- Real-World Data: Agents must navigate messy workspaces containing up to 123 files, including PDFs, Excel sheets, emails, and even handwritten receipts or dispatch logs, mimicking the varied evidence tax professionals encounter.
- Beyond Calculation: The core challenge lies in testing professional judgment, requiring agents to identify unstated but critical issues (like a looming tax exclusion deadline), reconcile conflicting evidence, recognize insufficient information, and carry these decisions through to a final deliverable.
- Current Performance: Even the leading models, like Claude Sonnet 5.5, achieve only a modest 10.8% pass rate, indicating significant room for improvement in handling the complexity and nuanced judgment required.
- Expert-Designed: All tasks are developed and rigorously reviewed by tax experts with diverse backgrounds in corporate, partnership, and individual taxation, ensuring real-world relevance and accuracy.
Ultimately, DAYJOB: Tax demonstrates that while AI can process information, consistently exercising professional judgment and managing the full scope of real-world tax workflows remains a significant hurdle for current frontier models. They haven't quite passed the audit yet.
The Gossip
AI as the Accountant's Auditor
Many users recounted surprisingly positive experiences using LLMs (like Claude and GPT) to review their own or their human accountant's tax returns. These AI checks frequently uncovered costly errors or omissions that saved users significant money, leading to the sentiment that AI is already a valuable 'second pair of eyes' or even a primary tool for preparing complex returns, despite the benchmark's lower scores. There's a strong undercurrent of users questioning why professional accountants aren't leveraging these tools more proactively.
Human Hazard & Hand-Wringing
A recurring theme was the profound dissatisfaction with traditional tax accountants. Numerous commenters shared anecdotes of accountants making severe, sometimes basic, errors that cost them thousands of dollars, despite the high fees. This highlighted the inherent fallibility of human professionals and the maddening reality that, in most jurisdictions, the taxpayer ultimately bears the liability for these mistakes. This user frustration fueled the interest in AI as a potentially more reliable or at least an equally fallible, but cheaper, alternative.
Benchmark Blinders & Botches
Some commenters critically assessed the benchmark itself, questioning its design and the reported AI performance. Points were raised about whether the benchmark truly tests raw model intelligence or the model's ability to 'self-scaffold' problem-solving without explicit tool integration or prompt engineering. Skeptics also pointed out AI's known weaknesses in basic mathematical reasoning and physical world understanding, suggesting that specialized 'harnesses' or deterministic tool-use are crucial for accurate tax calculations, rather than relying solely on the raw model. Additionally, concerns were voiced that publicly documenting missed edge cases might inadvertently 'train' future models, making benchmarks susceptible to gaming.
Byzantine Bureaucracy & Burdens
The discussion often veered into the broader issues of tax code complexity and the political landscape of tax preparation. Commenters lamented the intricate nature of the US tax system, which forces many to seek professional help. There was significant criticism of the lobbying efforts by the tax preparation industry that reportedly led to the demise of initiatives like the IRS Direct File program, thereby perpetuating the need for paid services despite the government already possessing much of the necessary data for simplified filing.