Detecting LLM-Generated Texts with “Classical” Machine Learning

Driven by personal frustration with the proliferation of low-quality, AI-generated fanfiction, an author set out to build a tool to detect such content. While initial attempts with perplexity-based methods proved unreliable, a return to 'classical' machine learning techniques yielded surprisingly effective results.

• The author developed a binary classifier using TF-IDF and LinearSVC, trained on a diverse dataset of human-written and LLM-generated Chinese texts.
• Data generation involved leveraging various low-cost or free LLM APIs (Gemini, Qwen, GLM-5, Kimi, Deepseek, Doubao) to create AI samples, matching human content genres.
• The system achieved impressive sentence-level accuracy of over 85% for individual models, which improved further by using a majority-voting ensemble of seven classifiers.
• A client-side JavaScript implementation allows for browser-based detection, sacrificing minimal accuracy for serverless deployment.
• Tests showed strong generalization, detecting text from unseen LLMs (like Claude Sonnet and GPT 5.2) with high confidence, and exhibiting very low false positive rates on human-written works.
• The tool revealed that over 32% of articles from trending tags on a Chinese platform scored above 50% AI, with no disclosure from authors.
• Common bypass methods, such as roundtrip translation or 'magic prompts' designed to reduce 'AI-flavor,' proved largely ineffective against the detector.

The author expresses genuine surprise at the ease with which current LLM-generated text patterns can be identified, suggesting these models are fundamentally different from human creativity. The work highlights the growing challenge of distinguishing AI-generated content and laments the perceived decline in quality across online platforms.