Small models also found the vulnerabilities that Mythos found
This article evaluates small, open-weight AI models against Anthropic's Mythos in cybersecurity vulnerability detection, claiming these smaller models can perform similarly when given isolated code. It argues that the true "moat" is the system and human expertise, not the model's sheer size. This challenges the narrative that only large, frontier models can achieve advanced AI cybersecurity, sparking considerable debate on methodological fairness.
The Lowdown
The article, "AI Cybersecurity After Mythos: The Jagged Frontier," directly challenges the narrative set by Anthropic's Mythos Preview, which showcased its advanced AI model autonomously finding and exploiting critical software vulnerabilities. The authors from AISLE contend that while Mythos validates the concept of AI in cybersecurity, the true power lies not in large, exclusive models but in the sophisticated systems built around more accessible AI.
- AISLE tested several small, cheap, open-weight AI models against the same flagship vulnerabilities Anthropic's Mythos identified, including the FreeBSD NFS and OpenBSD SACK bugs.
- These smaller models, some with as few as 3.6 billion active parameters costing just $0.11 per million tokens, were able to recover much of the same analysis, detecting vulnerabilities and reasoning about exploitation.
- The article demonstrates that AI cybersecurity capability is "jagged" across tasks, meaning performance doesn't scale smoothly with model size or cost, and no single model is consistently "best."
- A key finding was "inverse scaling" for some tasks, such as distinguishing real vulnerabilities from false positives on an OWASP benchmark, where smaller models sometimes outperformed larger, more expensive frontier models.
- The authors emphasize that the "moat" in AI cybersecurity is the surrounding system—the scaffold, pipeline, iterative deepening, validation, triage, and human security expertise—rather than the specific large language model itself.
- They also highlight that many models struggled with "specificity," incorrectly flagging patched code as vulnerable, reinforcing the need for a robust system to manage false positives.
The piece concludes that discovery-grade AI cybersecurity capabilities are broadly accessible with current models, including open-weights, and that defenders should prioritize building the necessary systems and pipelines around these models. It praises Anthropic for validating the field but critiques the overstatement of exclusive capabilities in large models.
The Gossip
The Code Context Conundrum
A significant portion of the discussion revolves around the article's methodology, specifically the decision to test small models on "isolated relevant code" rather than entire codebases. Critics argue this fundamentally changes the problem, as discovering the relevant code in a vast project is often the most challenging aspect of vulnerability research. They contend that Anthropic's Mythos demonstrated end-to-end discovery, making the comparison unfair. Conversely, some commenters highlight the article's own caveat and central thesis: that the "moat" is the system (scaffold, agents, context-handling) that feeds models targeted code, not the model's raw size or intelligence. They argue the article's point is precisely that if the system does the heavy lifting of context isolation, cheaper models can then perform the analysis.
Exploit Execution Exclusivity
Commenters debate whether the small models truly match Mythos's capability in exploit *generation*, not just detection. While the article notes that smaller models can reason about exploitability, it concedes they didn't independently conceive novel constrained-delivery mechanisms. One comment specifically recalls Anthropic's claim that Mythos significantly outperformed other models in autonomous exploit development, suggesting this remains a key differentiator for frontier models.
Specificity Scrutiny
The article highlights that while small models demonstrate high "sensitivity" (finding bugs), they often suffer from poor "specificity" (incorrectly flagging patched code as vulnerable). This leads to a high false positive rate, which commenters implicitly acknowledge as a critical practical concern for any security tool. The authors argue this further emphasizes that a robust system and human expertise are essential to filter noise and ensure trust in AI-driven security.