Incident Report: May 19, 2026 – GCP Account Suspension
Railway's production account on Google Cloud was incorrectly suspended by an automated process, leading to an 8-hour platform-wide outage. This incident report details the cascade of failures, from core services to network caches, and Railway's architectural introspection. The Hacker News discussion heavily criticizes Google's automated account management and perceived decline in customer service.
The Lowdown
Railway, a platform-as-a-service provider, experienced an 8-hour platform-wide outage on May 19-20, 2026, after Google Cloud incorrectly suspended their production account. This automated action disabled core infrastructure, including their dashboard, API, and databases hosted on GCP, and cascaded to affect all Railway workloads, even those on AWS and Railway Metal, as network control plane caches expired.
- The incident began with API health check failures, quickly revealing Google Cloud's suspension as the root cause.
- Even after GCP account access was restored, services required a layer-by-layer recovery, extending the outage due to separate restoration needs for persistent disks, compute instances, and networking.
- The outage prompted GitHub to rate-limit Railway's OAuth and webhook integrations due to a surge in retried requests, further disrupting logins and builds.
- Railway acknowledges its architectural decisions, specifically a hard dependency on a GCP-hosted network control plane API for workload discoverability, allowed a single provider's action to cause a full platform outage.
- Immediate preventative measures include removing this single dependency to create a true mesh network, extending high-availability database shards across multiple clouds, and ultimately removing Google Cloud services from their data plane's hot path.
Railway committed to taking full responsibility for its vendor choices and ensuring future resilience, understanding that customers ultimately hold them accountable for uptime, regardless of upstream provider issues.
The Gossip
Google's Gripes and Gridlock
Commenters expressed significant frustration and a lack of surprise regarding Google Cloud's automated account suspensions. The discussion highlighted a pattern of Google banning accounts, even large enterprise customers, with little to no prior warning or clear justification, drawing parallels to personal Gmail account bans. Many felt this prioritization of automation over human verification was a calculated business decision where "it's cheaper to say sorry than do it right."
A Google Gambit: The New Microsoft?
A recurring sentiment was that Google is exhibiting behaviors reminiscent of Microsoft's past, particularly in becoming an "opinionated Walled Garden" that stifles open ecosystems. Some commenters speculated that Google's focus on advertising and finance, coupled with an increasing reliance on barely functional AI in core products like search, suggests a broader decline in service quality and customer focus across the company.