99% of My Website Traffic Is Bots
A web developer discovered 99% of his website's traffic comes from bots, sending hosting costs soaring and highlighting the precarious economics of the open web. This issue resonated deeply on Hacker News, sparking a debate about who bears the cost of vast data scraping, particularly by AI, and the increasingly blurry line between legitimate web users and resource-draining automated agents. Commenters shared personal anecdotes and technical solutions, wrestling with the implications for site owners and the future accessibility of online information.
The Lowdown
Peter Cooper's blog post describes a common and increasingly problematic scenario for independent web developers: his website, which curates information about charitable organizations, is inundated with bot traffic. This overwhelming bot activity led to a 500% spike in his monthly hosting bill, making it financially unsustainable.
- Overwhelming Bot Presence: Cooper reports that 99% of his site's traffic consists of bots, with spikes leading to significant cost increases on his Cloudflare D1-based infrastructure.
- Targeted Scraping: These bots specifically target more obscure 'fund pages' that only 10% of human visitors access, indicating a focused data extraction effort rather than general indexing.
- Ineffective Defenses: Traditional bot mitigation strategies, including Cloudflare's non-interactive challenges, proved insufficient. Bots often lacked referrers, making them harder to identify.
- Acknowledged Irony: Cooper, who himself scrapes public documents to build his site's database, acknowledges the irony of complaining about scrapers.
The story underscores a growing tension between the ideal of an open, accessible internet and the very real financial burden placed on content creators by increasingly sophisticated and pervasive automated scraping, especially in the age of large language models.
The Gossip
The Scraper's Ethical Scrutiny
Many commenters honed in on the author's admission of being a 'scraper complaining about scrapers,' questioning the moral high ground. The discussion quickly pivoted to the broader philosophical debate around the 'open web' versus site owners' rights to control access and costs. Some argued that blocking bots, even 'good' ones like research scripts, harms the open nature of the internet, while others vehemently defended a site owner's right to protect their resources from uninvited and costly automated access, especially when it provides no reciprocal benefit.
Costly Cloud Conundrums
A significant portion of the discussion revolved around the author's choice of Cloudflare D1 for hosting, with many suggesting it might be an 'extremely expensive' option for his use case. Commenters offered a plethora of alternative, often cheaper, hosting solutions like static site generators on GitHub Pages, GitLab Pages, Cloudflare Pages, or traditional VPS setups. There was a consensus that proper infrastructure optimization, caching, and careful database queries could drastically reduce costs, often negating the need for aggressive bot blocking on lower-traffic sites.
AI's Insatiable Appetite
The rise of AI-driven scrapers, specifically those attributed to companies like Anthropic (Claude-searchbot) and OpenAI (GPTBot), dominated the conversation. Commenters shared logs demonstrating aggressive and wasteful crawling behavior, sometimes even attempting to access sensitive files. This led to calls for AI companies to either pay for content access or adhere to new protocols, with some highlighting how the increase in `robots.txt` directives blocking AI crawlers indicates growing site owner frustration and a push for AI to pay its 'fair share' for the data it consumes.
Bot Blocking Battles
Practical strategies for identifying and mitigating bot traffic were a hot topic. Suggestions ranged from simple geo-blocking and IP blacklisting (though with caveats for legitimate users on VPNs or in specific regions) to more sophisticated proof-of-work systems like Anubis. While Anubis was praised for its concept, some critics pointed out its potential for being bypassed by determined (and AI-assisted) scrapers or inconveniencing real users. The effectiveness of detecting bots based on their lack of CSS requests or JavaScript execution was also debated.
The AI-Generated Allegation Amusement
A humorous, meta-discussion emerged when a commenter jokingly (or seriously) suggested the article itself was AI-generated. This sparked a mini-debate about the limitations of AI detectors and the common stylistic patterns that often trigger such accusations. The author, Peter Cooper, directly engaged with this, acknowledging that his Twitter-influenced writing style might resemble AI output and even updating his blog post based on the feedback, demonstrating a rare and refreshing openness to criticism.