How the FSF sysadmins block botnets with reaction

The FSF's sysadmins offer a detailed account of their battle against AI-driven botnet scrapers, showcasing their technical evolution from fail2ban to a customized reaction setup. This deep dive into scalable IP blocking techniques resonated with the Hacker News crowd, sparking discussions on network defense and the ethics of web scraping. The community also debated the FSF's foundational principles in light of modern challenges and controversial software manifestos.

100

Score

Comments

#11

Highest Rank

on Front Page

First Seen

Jul 14, 3:00 PM

Last Seen

Jul 14, 8:00 PM

Rank Over Time

The Lowdown

The Free Software Foundation (FSF) has been battling persistent DDoS attacks, primarily from aggressive LLM scrapers leveraging residential botnets. This article details their technical evolution in defending their infrastructure, culminating in the adoption and customization of a robust open-source tool called reaction.

Initial defense involved fail2ban with UFW, but quickly hit a scalability wall at 65,000 firewall rules.
The team migrated to using ipset in conjunction with fail2ban, which allowed for blocking millions of IPs without performance degradation.
Limitations in fail2ban's architecture (Python/SQLite) led to temporary custom scripts and a search for a more modern alternative.
They discovered reaction, a fail2ban successor, and invested time to configure it for their specific needs, including ipset integration.
Further optimizations involved developing mechanisms for reaction to export and restore IP sets for rapid service restarts, enhancing rule application.
The FSF has contributed their ipset configurations back to the reaction project, making their improvements available to the wider community.

This practical account underscores the ongoing challenge of protecting online services from malicious scraping and highlights the FSF's commitment to open-source solutions and sharing their operational improvements.

The Gossip

Scaling Firewall Fortifications

Commenters largely validated the FSF's experience with `UFW` and `fail2ban` hitting scalability limits, noting similar issues with `firewalld`. The discussion pivoted to more performant and scalable blocking alternatives, including `nftables` (as a modern successor to `iptables + ipset`) and direct `ip route add blackhole`. The latter was highlighted for its low CPU overhead and ability to block hundreds of thousands of CIDR blocks, offering a highly efficient method for denying malicious traffic.

Scrapers, Ethics, and 'The Torment Nexus'

A significant portion of the discussion questioned the nature of "attackers," particularly when referring to AI-driven scrapers. Some mused if these were "AI monied interests" that coincidentally funded the FSF, a claim which was subsequently challenged due to lack of current evidence. Commenters debated whether simply accessing public data constitutes an attack or if the resource drain makes it so, suggesting that legitimate scrapers should opt for paid data dumps or API access. The concept of a "torment nexus" was even invoked to describe the complex, often contentious, relationship between data, AI, and content providers.

FSF's Principles and Pragmatism

The FSF's long-standing stance against JavaScript was highlighted, explaining why certain common anti-bot solutions (which often rely on JavaScript challenges) are incompatible with their philosophy. This led to a brief discussion about Anubis's ability to support non-JS proof-of-work. Separately, a controversial message in the `reaction` project's README, declaring the software as "gay, trans and anticolonialist" and advising uncomfortable users not to use it, sparked debate on its alignment with the AGPLv3's universal freedoms, particularly concerning ideological statements in software distribution.