A Couple Million Lines of Haskell: Production Engineering at Mercury

This article from the Haskell.org blog, authored by Ian Duncan of Mercury, delves into the practicalities and philosophies behind running a massive 2 million line Haskell codebase in a high-stakes fintech environment. Mercury, a company processing billions in transactions, defies the common perception that Haskell is unsuitable for such scale, instead leveraging its strengths for robust production engineering.

Key takeaways from Mercury's experience include:

• Reliability Philosophy: Rather than solely preventing failures, Mercury focuses on building systems that absorb variation, degrade gracefully, and are adaptable by operators, emphasizing "adaptive capacity."
• Purity as a Boundary: Haskell's purity is viewed as an interface design principle that encapsulates dangerous operations (like mutable state) safely, ensuring external immutability while allowing internal efficiency.
• Making the Right Thing Easy: The type system is used to encode operational procedures and institutional knowledge directly into code, making incorrect actions impossible and preserving critical invariants even as teams change.
• Durable Execution with Temporal: Complex, multi-step financial workflows are managed using Temporal, which aligns well with Haskell's determinism, providing built-in retry logic and crash recovery.
• Domain-Centric Design: Emphasizes separating domain errors from transport-specific errors (e.g., HTTP status codes) to ensure code flexibility and reusability across different execution contexts.
• Type Encoding Tradeoffs: Advocates for strategically encoding invariants into types, prioritizing protections against silent corruption, while acknowledging the cognitive and rigidity costs of over-engineering the type system.
• Designing for Introspection: Due to Haskell's lack of monkey patching, the importance of designing for observability is highlighted, recommending "records of functions" and effect systems as mechanisms for instrumenting and adapting behavior.
• Ecosystem Challenges & Practicalities: Discusses the reality of working with a smaller ecosystem, the need for library authors to provide escape hatches for instrumentation, and the compromises (like unsafePerformIO) necessary in production.
• Why Haskell?: Addresses hiring concerns by noting Haskell attracts high-quality generalists, emphasizes the need for a culture of pragmatism over idealism, and highlights Haskell's long-term benefits in refactoring, clarity, and incident prevention, especially in financial services where data integrity is paramount.

Ultimately, Mercury's journey with Haskell illustrates that while it's not a silver bullet, its powerful tools, when applied pragmatically, provide significant engineering leverage. It enables organizations to build resilient, understandable systems that encode hard-won operational knowledge, helping them navigate rapid growth and changing teams effectively.