Flawed Routers Flood University of Wisconsin Internet Time Server (2003)
The University of Wisconsin-Madison's public NTP server was inundated by a massive, continuous flood of traffic in 2003. This detailed report meticulously uncovers that the root cause was an accidental denial-of-service, stemming from a design flaw in hundreds of thousands of Netgear residential routers hard-coded to query the university's time server repeatedly. It's a classic HN tale of meticulous network forensics meeting the real-world implications of widespread, poorly implemented IoT (or rather, pre-IoT) devices.
The Lowdown
In May 2003, the University of Wisconsin-Madison's public Network Time Protocol (NTP) server became the unwitting target of a massive, sustained traffic flood, reaching hundreds of thousands of packets per second. Initially suspected as a malicious distributed denial-of-service (DDoS) attack, the extensive investigation revealed a far more pervasive and accidental culprit: a critical design flaw embedded within hundreds of thousands of Netgear residential internet routers. This foundational 2003 report by Dave Plonka meticulously documents the entire incident, from initial detection to proposed long-term solutions, illustrating the profound impact of seemingly minor software errors in widely deployed consumer hardware.
- The Initial Deluge: The UW-Madison network observed an sudden, dramatic increase in inbound UDP traffic to its NTP server, peaking at 40,000 packets-per-second. Network operators initially blocked traffic from a suspicious source UDP port (23457), presuming a transient 'script kiddie' attack.
- Unrelenting Influx: A month later, the flood continued and intensified, exceeding 250,000 packets-per-second (150 megabits-per-second). Further deep packet inspection revealed these were legitimate-looking Simple Network Time Protocol (SNTP) version 1 queries, but from many real hosts, often querying at an abnormally high rate of one packet per second per device.
- The Forensics Trail: By contacting other universities identified as sources, the investigation pinpointed Netgear routers (e.g., MR814) as the origin. Analysis of Netgear firmware binaries confirmed a hard-coded IP address for
ntp1.cs.wisc.edu(128.105.39.11) and the use of the fixed source UDP port 23457. - The Flaw's Mechanics: The Netgear routers' SNTP clients were designed to poll the hard-coded UW-Madison server at one-second intervals until a response was received. Even after receiving a reply, some models continued polling at overly frequent rates. With Netgear reporting over 700,000 affected products, the theoretical maximum flood could reach 700,000 packets-per-second, or 426 Mbps.
- Vendor Engagement and Resolution: After initial difficulties in establishing contact, Netgear cooperated, forming a review team with UW-Madison and independent experts. While Netgear developed firmware updates to correct the flaw, a full product recall or reliance on customer upgrades was deemed impractical due to the nature and widespread deployment of the devices.
- Strategic 'Endgames': UW-Madison explored two primary long-term solutions: (A) deploying a BGP anycast NTP service within WiscNet to absorb the flawed Netgear requests, or (B) attempting to suppress the requests globally by sacrificing a block of its IP address space (e.g., a /20 block) to announce an unreachable route, with the latter incurring significant costs in IP resource allocation.
- Broader Community Impact: The report stresses the need to inform the internet community about such flaws, clarify best practices, and improve protocol standards. It notes similar issues with SMC routers flooding CSIRO in Australia and initiates discussions on Internet Drafts, such as "Embedding Globally Routable Internet Addresses Considered Harmful."
By August 2003, Netgear was actively collaborating on a solution, but the university continued to face significant operational challenges from the ongoing, accidental denial-of-service. This incident served as a stark example of how pervasive, low-cost internet devices could unintentionally destabilize critical public services and raised fundamental questions about network infrastructure resilience and manufacturer responsibility.