Monzo Stand-In
Monzo unveils its "Stand-in" system, an innovative, minimalist backup banking infrastructure designed to ensure continuous customer service during full cloud outages. This deep dive details their clever approach to disaster recovery, leveraging independent architecture and cost-effective data synchronization to maintain critical operations. It's an exemplary case study in practical resilience engineering that resonates with the HN crowd.
The Lowdown
Monzo, the digital bank, has engineered a robust solution to combat platform outages: Monzo Stand-in. This completely separate, independent backup banking infrastructure is designed to ensure customers can always access essential services like spending and transfers, even during major incidents affecting their primary cloud platform.
- Independent Architecture: Monzo Stand-in operates on Google Cloud Platform (GCP) with a minimalist set of 18 microservices, distinct from the 3,000 microservices on their Amazon Web Services (AWS) primary platform. This deliberate architectural divergence mitigates the risk of common-mode failures.
- Cost-Effective Resilience: Building new, separate services instead of replicating the entire primary stack drastically reduces operational costs, with Stand-in running at approximately 1% of the primary platform's expense.
- Data Synchronization Strategy: Data is synced from the primary to Stand-in in an eventually consistent, non-blocking manner, ensuring Stand-in has the minimal state required for critical functions. Tokenized data uses a similar, secure exchange.
- Recovery and Reconciliation: When Stand-in is active, it generates "Monzo Advices" - a log of effects like approved payments - for the primary platform to consume and apply verbatim upon recovery, maintaining the primary as the ultimate system of record. Correlation IDs are used to avoid data duplication during reconciliation.
- Flexible Activation: Stand-in can be selectively or fully enabled via a configuration service, allowing controlled failover. Payment traffic can be routed via the primary platform or directly to Stand-in during a total primary outage.
- Proven in Practice: The system successfully demonstrated its capability during a significant 2024 outage, ensuring continuous service for customers despite primary system issues.
Monzo Stand-in exemplifies a proactive and innovative approach to operational resilience, particularly relevant in a regulated financial sector. It offers a practical, cost-effective alternative to traditional disaster recovery methods, proving their commitment to uninterrupted customer access.