Bwbach, My Guardian Goblin
Faced with aggressive web crawlers overwhelming his Traefik instance, an engineer built "Bwbach," a custom Elixir-based reverse proxy. This clever goblin-themed system traps bots in a slow-drip purgatory of junk data, dramatically reducing server load. HN appreciates the elegant, resource-efficient, and slightly mischievous technical solution to a common internet nuisance.
The Lowdown
Robert May recounts how his small photography site was hammered by millions of requests from badly configured web crawlers, notably Meta's, causing his Traefik proxy to choke despite the backend Elixir application being fine. He decided to build his own, more resilient, and vindictive reverse proxy dubbed "Bwbach," after the Welsh household spirit.
- The Problem: Meta and other crawlers generated millions of requests, with each request taking 30 seconds due to an earlier 'infinite blackhole' trap, pushing Traefik's CPU usage to 85%.
- The Solution (Bwbach): A custom reverse proxy written in Elixir, leveraging libraries like Bandit, Thousand Island, Finch, and Mint, storing data in ETS and Mnesia.
- Core Principle: Waste the bots' time by trickling content at an agonizingly slow rate, just inside their configured timeout, serving endless junk pages and blackhole links.
- Advanced Defenses: Includes a splash redirect, reverse-verification for legitimate bots, and exploit countermeasures like fake
.envfiles, expanding gzip archives for database requests, and PNG bombs for image requests. - Results: Within a week, concurrent trapped bot requests dropped from ~500 to ~5, and Bwbach now uses only 5-10% CPU, running efficiently on cheap ARM instances.
- Technical Implementation: Operates as a Level 7 proxy with ACME for SSL, handles SSL redirects, and uses Mnesia for redundant data sharing across cluster instances.
Bwbach exemplifies a robust, Elixir-powered approach to bot management, combining folklore with sophisticated technical trickery to turn unwanted traffic into a resource-consuming quagmire for the aggressor.
The Gossip
Elixir's Edge: Proxying Prowess
The discussion touches on the suitability of Elixir and the BEAM for high-concurrency tasks like reverse proxying. Commenters inquire about the existence of other open-source proxies in this stack, to which the author confirms he hadn't found any readily available, suggesting that Elixir's strengths are likely utilized more in closed-source, large-scale systems.
Open Sourcing Speculation
The author floats the idea of open-sourcing Bwbach, acknowledging his limited time for contributions but expressing willingness if there's enough community interest. A commenter enthusiastically indicates their interest in trying the proxy if it were made open source, especially given their existing use of Coolify.
Defensive Dexterity & Tarpit Tactics
Comments highlight the aggressive, defensive nature of Bwbach's strategy, with one user noting it's "one step away from an attack barrier." The author also references `iocaine`, a tool from a previous article, drawing parallels and distinguishing Bwbach's more active tarpitting approach from other passive bot-thwarting methods.