A false smoke alarm almost drowned Jane Street's only NYC datacenter
A false smoke alarm at Jane Street's NYC datacenter triggered a cascade of failures, shutting down cooling and nearly drenching their critical servers in sprinkler water. The author, a systems engineer, heroically sprinted back to manually yank every power cord just in the nick of time, preventing a catastrophic meltdown. This gripping tale of operational peril, quick thinking, and post-incident OCaml monitoring resonated with HN for its blend of high-stakes finance, practical engineering, and a dash of 'what could possibly go wrong?'
The Lowdown
This is a "Star Trek: Lower Decks" style war story from a former Jane Street systems engineer. He recounts how the firm's unique risk culture, which prioritized having their datacenter physically close to the trading floor for manual intervention, almost led to its demise.
- The datacenter, a significant operation with many rows of servers, CRAC units, and a diesel generator, was crammed into a normal office building in downtown Manhattan.
- A false smoke alarm was triggered by dust from contractors working in an adjacent room.
- Crucially, the smoke alarm was wired to shut down the CRAC (cooling) units, causing the datacenter temperature to skyrocket.
- While the sprinklers were dry-pipe, the alarm pressurized them, and they were set to activate if the temperature reached a critical threshold.
- The author, living minutes away, received the urgent call and raced back, entering a sauna-like datacenter.
- He manually pulled every power cord from every server rack, stopping the temperature climb just below the sprinkler activation point (143.6°F).
- Due to the timing (NYC markets closed), downtime was minimal.
- In the aftermath, the author developed a new, highly reliable OCaml-based monitoring system that directly polled every server for its temperature, deeming it more trustworthy than dedicated CRAC or room sensors.
The story highlights the unforeseen interactions between seemingly unrelated safety systems and the value of robust, direct telemetry in preventing catastrophic failure, even in an environment as sophisticated as Jane Street.
The Gossip
Firefighting Follies & Flawed Facilities
Commenters extensively debated the datacenter's fire suppression setup. Many questioned the decision to shut down CRAC units upon smoke detection, pointing out that industry codes often allow cooling to continue in specific datacenter configurations. The surprising absence of a single Emergency Power Off (EPO) button, which could have instantly de-energized all equipment, drew considerable disbelief and discussion about safety standards. The conversation also touched upon the use of inert gas fire suppression systems (like Halon substitutes) versus water sprinklers, with pros and cons for each in an office building context.
Sandy's Shadow & Systemic Shortcomings
The story's NYC setting and tale of operational crisis immediately brought up parallels to Hurricane Sandy. Commenters shared their own harrowing experiences from that event, recounting struggles like manually carrying diesel fuel up many flights of stairs for generators, or generators failing due to flooded fuel inlets. A strong undercurrent in this discussion was the critique of "heroism" in these situations, arguing that such efforts often mask severe underlying failures in planning and resilience that should be addressed rather than celebrated.
On-Call Obligations & Personal Proximity Perils
The author's account of being the designated emergency contact due to living close to the office resonated deeply with many readers. Numerous commenters shared similar anecdotes of being exploited for their convenience or dedication, leading to undesirable or unpaid after-hours work. This sparked a broader discussion about establishing firm boundaries with employers and colleagues, being careful about how much personal information is shared at work, and the importance of learning to say 'no' to requests that fall outside one's job description, even if it means sacrificing perceived helpfulness.