How Uber Protects Against Retry Storms
Uber dissects the insidious problem of retry storms in microservices, where well-intentioned retries amplify failures into cascading outages. They've engineered a sophisticated 'error ownership' model that intelligently decides when to retry, preventing a localized hiccup from becoming a global meltdown. This deep dive into distributed systems reliability offers a pragmatic solution for a common, complex scaling challenge.
The Lowdown
In the sprawling landscape of microservices, a common foe lurks: the retry storm. Uber's recent blog post details how aggressive, context-unaware retries can transform a minor service degradation into a full-blown, stack-wide incident. Standard retry mechanisms, while useful for transient issues, notoriously amplify load on struggling services during severe degradation, leading to a catastrophic domino effect.
Uber's solution, dubbed 'error ownership,' redefines how services interact with failures in a deep dependency chain. Their approach is designed to prevent requests from fan-out exponentially, which can prolong outages and cripple user experience. Here's how they tackled it:
- The Problem Defined: Traditional retries, even with budgets, lack visibility into cross-service amplification. A service might retry for an error it didn't cause, merely passing along a downstream issue, leading to unproductive request amplification.
- Error Ownership Principle: A service 'claims' an error if it's the true source. If the error originated from a downstream dependency, the service 'unclaims' it, acting as a symptom, not the cause.
- Service Dependency Analysis Solution (SDAS): This internal tool correlates inbound failures with outbound failures, allowing services to accurately determine error ownership. An
x-uber-error-claimheader propagates this critical context. - Intelligent Retry Logic: Upstream services only retry if the downstream service claims the error or if the claim header is missing (in uncooperative environments). If the error is unclaimed, no retry is performed, effectively cutting off the storm before it propagates.
- Edge Case Handling: Mechanisms are in place to guarantee 'at-least-once' retries when appropriate (e.g., if a service's outbound doesn't have retries configured) and to manage 'context drops' within services, ensuring the system remains robust.
- Real-World Validation: The system is fully operational across Uber's service mesh. A real outage in November 2025 demonstrated its efficacy, preventing an estimated 9.5 million spurious requests and dramatically reducing the 'max retry storm radius' across their user-facing APIs.
By ensuring retries are targeted only at services genuinely capable of resolving the issue, Uber has built a formidable defense against cascading failures, preserving system stability and user experience even during significant degradations.