Training a model to identify AI-generated web content from structure alone
Facing the deluge of 'AI slop,' researchers developed a novel model that distinguishes AI-generated web content from human writing with 98% accuracy, focusing purely on structural patterns. This technical deep dive reveals AI's predictable writing 'shape,' which remains detectable even after extensive rewording. The project offers a clever solution to a growing problem for marketers and content creators, highlighting the subtle, yet pervasive, characteristics of artificial prose.
The Lowdown
The creators of Sitefire, a YC-backed company assisting marketing teams with AI search optimization, recognized the pervasive problem of 'AI slop'—AI-generated content that, while effective for search, is widely disliked by humans. They set out to identify what defines this slop and if it could be detected, even as content creators increasingly leverage AI.
- Inspired by previous research (StoryScope) that identified AI-generated fiction by structure, they adapted the methodology for commercial web content.
- They compiled a dataset of 2,250 pre-ChatGPT human blog posts and generated 11,250 AI mirror versions using five frontier AI models (GPT-5.4, Claude Sonnet 4.6, Gemini 3 Flash, DeepSeek V3.2, Kimi K2.5).
- Instead of analyzing words, they used an LLM to answer 214 structural questions about each post (e.g., product promotion, source citation, expert quotes).
- A classifier trained on these structural answers achieved 98% accuracy in differentiating human from AI-generated posts on unseen data.
- The study found that AI models exhibit a consistent 'tidy, self-announcing' structure, often repeating main points multiple times, a trait seen in 77% of AI posts versus 12% of human posts.
- A second classifier could identify which of the five AI models (or a human) authored a post 79% of the time, with most errors occurring between different AI models.
- Crucially, this structural detection method proved robust against rewording; even when AI posts were 73% reworded, the 'slop classifier' maintained its accuracy.
- The authors acknowledge limitations, including untested 'humanizer' tools and a date mismatch between human (2020-2022) and AI (2026) content.
This research provides a powerful tool for understanding and detecting the inherent patterns of AI-generated content, moving beyond word-level analysis to structural signatures that are more difficult to obfuscate. The findings underscore the current predictability of AI writing and offer insights for creating more human-like, varied content.