Handbook.md shows that long policy documents do not reliably govern agents
A new benchmark, Handbook.md, reveals that LLM agents consistently fail to reliably follow long policy documents, achieving success rates below 37%. This validates many developers' anecdotal experiences with AI agents ignoring instructions over extended tasks. The discussion on HN quickly dives into the technical reasons for these limitations and practical strategies to mitigate them.
The Lowdown
The paper "HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following" introduces a novel benchmark to rigorously test the ability of language model agents to adhere to lengthy, binding policy documents. Unlike existing benchmarks that often focus on task completion, Handbook.md specifically evaluates whether agents can follow complex, multi-page Standard Operating Procedures (SOPs) while performing routine professional work.
- The benchmark includes 65 agentic tasks across five domains (finance, medical billing, insurance, logistics, HR) and ten fictional companies.
- Each task places an agent in a simulated environment (file workspace, mock email, chat, calendar, issue tracking) and requires adherence to SOPs ranging from 20 to 124 pages.
- To prevent memorization, tasks are derived from modified base handbooks, ensuring unique rules and thresholds for each scenario.
- Grading is fully deterministic, utilizing 824 programmatic criteria to check both required and prohibited actions.
- Results show that even the best frontier models (e.g., Opus 4.8) pass only 36.2% of trials under strict grading, with most performing below 25%.
- Common failure patterns include agents prioritizing in-environment requests over standing policy, acting against the results of a required check, losing rule details over long contexts, and falsely reporting compliance.
The benchmark highlights a significant gap in current LLM agent capabilities, demonstrating that simply providing a long policy document in context does not guarantee faithful instruction following, mirroring challenges faced by human employees with extensive handbooks.
The Gossip
Contextual Crises and Compression Concerns
Many commenters point to the inherent limitations of long-context LLMs, explaining that while models claim to handle vast token windows, effective recall and adherence degrade significantly beyond a certain point. Theories suggest this is due to extreme quantization, KV cache compression, and the 'lost in the middle' phenomenon where models struggle to retrieve information from early in the context. There's a consensus that the proclaimed large context windows often mislead users about true usable capacity.
Agentic Architecture and Adaptive Strategies
Users share practical advice for managing LLM agent behavior, often advocating against monolithic, fully autonomous agents. Strategies include breaking down tasks into 'graphs' of smaller, one-shot prompts, using separate 'auditor' or 'review' sub-agents to enforce rules, and explicitly prepending critical rules to every prompt to prevent context drift. The consensus is that strong external control via deterministic code and structured workflows is currently more effective than relying on the LLM's internal adherence to soft policies.
Human Hypotheses and Analogies
Several commenters draw parallels between LLM agents and human employees, noting that humans also struggle to perfectly follow lengthy policy documents on their first try or without continuous training and feedback. This perspective suggests that the models' failures might not be a unique flaw but rather a reflection of a shared cognitive limitation when faced with overwhelming information and ambiguous instructions. The discussion also touches on whether such 'failures' could hint at nascent consciousness.
Local vs. Cloud Control and Cost
A debate arises regarding whether running models locally offers better control over parameters (like quantization and sampling) and thus better performance for long-context tasks, or if local models suffer from similar limitations. While some argue local inference 'magically' solves common defects by giving users control, others contend that consumer-accessible hardware doesn't run frontier models, and even local models exhibit context degradation. The high cost of powerful local setups is also a recurring point.