Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard
A new multi-agent AI system, Memdoor, has demonstrated superior performance on challenging coding problems, besting established models like Fable 5 on the LiveCodeBench Hard benchmark. By orchestrating five Qwen3.8-27B models in distinct roles, Memdoor achieves a 9.8 out of 10 score, highlighting the power of structured AI collaboration. This technical deep-dive resonates on HN as it showcases advanced AI capabilities in automated programming and offers insights into effective multi-agent architectures.
The Lowdown
The Memdoor project introduces a sophisticated multi-agent harness designed to tackle complex programming problems, showcasing a significant leap in AI's ability to generate functional code. Operating as a Memdoor workflow, this system leverages five specialized Qwen3.8-27B language models, each with a distinct role, to collectively solve coding challenges. This novel approach has not only matched but surpassed the performance of leading code generation models and existing harnesses on the LiveCodeBench Hard benchmark.
- Multi-Agent Architecture: Memdoor employs a team of five Qwen3.8-27B models, assigned roles as a planner, two ideators, a solver, and a manager, along with a repair worker. These agents communicate exclusively through files on disk, ensuring a 'fresh context' for each interaction and contributing to the system's effectiveness.
- Benchmark Performance: The Memdoor workflow achieved an average score of 9.8 out of 10 over five passes on the first ten LiveCodeBench Hard problems. This compares favorably to the original GVS5H harness (9.6) and Fable 5 alone (8.0), demonstrating a clear advantage.
- Methodology and Cost: The system operates with the authors' exact prompts and loop, consuming approximately $35.79 for 50 runs, with each successful run typically costing around $0.39 and taking a median of 40 minutes of model time on OpenRouter.
- Workflow Design: The process follows a structured pipeline:
plan→ideate-a/b→solve→test→review→repair→verify. Each step is executed by a specific agent or command, withrequires:defining the dependencies, akin to a Directed Acyclic Graph (DAG). - Identified Strengths and Weaknesses: The system's success is attributed to its file-based communication, ensuring agents operate from a clean slate. It also identified and documented two specific areas where the underlying GVS5H loop could be improved, such as validating manager-invented counterexamples and ensuring the best valid program is retained.
Memdoor's success underscores the potential of breaking down complex problems into manageable, agent-specific tasks, and highlights how structured communication through a shared context (files) can dramatically enhance the collaborative problem-solving capabilities of AI models. This work provides a compelling case study for future advancements in AI-driven software engineering.