Can a MUD evaluate LLMs? A $99 proof of concept

The CrucibleBench project introduces a novel proof-of-concept benchmark for evaluating Large Language Model (LLM) agents within a persistent Multi-User Dungeon (MUD) environment. Developed with less than $100 in API credits, this experiment aims to move beyond static benchmarks by assessing how LLMs navigate complex social objectives, manage trust, and interact within a dynamic, persistent text-based world.

• MUD-Based Methodology: LLMs are placed in a simple MUD, featuring 7 command types, 12 rooms, 14 items, and Non-Player Characters (NPCs) with evolving trust and suspicion states. Each model undergoes 50 turns to achieve hidden social objectives.
• Unreliable LLM Judges: The most significant finding was the instability of LLM judges. A single LLM-based component in the scoring system drastically reordered the model leaderboard by up to six positions, with per-model agreement between independent judges ranging from a mere 21.7% to 84.8%. This highlights a critical need for per-subject audits of LLM judges, not just aggregate reliability statistics.
• Identified Behavioral Failures: The benchmark surfaced distinct behavioral failure modes in LLMs, including 'dialogue looping' (repeating failed conversational attempts), 'wrong-room interaction' (attempting actions in invalid locations), and 'exploration paralysis' (failure to convert information gathering into goal-directed actions). Dialogue looping was the most prevalent failure.
• 'Lateral Thinking': The project embraces the 'lateral thinking with withered technology' philosophy, leveraging the inherent constraints of MUDs (limited command space, explicit social feedback, persistence) to create a measurable environment for behavioral AI evaluation.
• Transparency and Openness: The team has made all project artifacts publicly available, including the research paper, 650 full transcripts, source code (MIT license), and complete API billing export.
• Future Development: Phase 2 plans include developing a more robust benchmark with human baselines, multiple judges, expanded objectives, and a larger environment.

CrucibleBench provides a creative, low-cost method for probing LLM behavioral capabilities and offers crucial insights into the vulnerabilities of current LLM-based evaluation strategies. Its transparent methodology and focus on measurable, persistent interactions mark a valuable step toward more dependable AI benchmarks.