pg_durable: Microsoft open sources in-database durable execution

Microsoft has open-sourced pg_durable, a PostgreSQL extension enabling durable, fault-tolerant execution of SQL workflows directly within the database. This technical dive is popular on HN for its promise to simplify background job management by eliminating external orchestrators and consolidating state within Postgres. However, much of the discussion revolves around the architectural implications of moving complex control flow directly into the database.

Score

Comments

Highest Rank

on Front Page

First Seen

Jun 5, 4:00 PM

Last Seen

Jun 5, 5:00 PM

Rank Over Time

The Lowdown

pg_durable is a new PostgreSQL extension from Microsoft designed to streamline long-running, fault-tolerant SQL functions by bringing durable execution directly into the database. It aims to reduce the complexity of managing background work that typically involves stitching together multiple external services like cron jobs, workers, queues, and status tables.

Core Functionality: Allows users to define workflows as graphs of SQL steps that are checkpointed by PostgreSQL. In case of database crashes, restarts, or step failures, execution resumes from the last successful checkpoint, eliminating manual state reconstruction.
Target Audience & Use Cases: Primarily for backend and data engineers, DBAs, and SREs who need workflows close to data. Applicable to tasks like vector embedding pipelines, data ingestion, scheduled maintenance, parallel aggregations, and orchestrating external API calls from SQL.
Architectural Shift: Moves workflow definition, retry logic, and progress tracking into SQL, potentially eliminating the need for some app-tier workers and queue consumers. Operational visibility is gained through standard Postgres tables.
Limitations: Not suitable for very simple, single SQL statements, sub-millisecond synchronous operations, environments where extensions cannot be installed, highly heterogeneous multi-system workflows, or complex application logic that doesn't map well to SQL.
Underlying Technology: Built using pgrx in Rust, it leverages duroxide (a durable task framework) and duroxide-pg (its PostgreSQL-backed state provider), with everything running inside the PostgreSQL server.
Status: The project is currently in a preview state.

By integrating durable execution directly into PostgreSQL, pg_durable offers a compelling alternative for teams looking to simplify their data-intensive background processing, provided their workflows align with its SQL-native paradigm.

The Gossip

Database Durability or Distributed Discrepancy?

A central debate revolves around the fundamental architectural choice of embedding workflow orchestration directly within PostgreSQL. Many commenters question the rationale, arguing that complex control flow is better managed by external DAG schedulers like Airflow or Temporal, or kept version-controlled in application code. They express concerns about SQL's readability for intricate logic and the potential for database bloat. Conversely, proponents highlight the benefits of architectural simplicity, avoiding additional infrastructure, achieving app-restart-proof resumable workflows, and having a single, consolidated stateful component.

Competing Queue & Workflow Constructs

Commenters quickly draw parallels and seek comparisons with existing and emerging PostgreSQL-based queuing and workflow solutions. Tools like `pgmq`, `pgflow`, DBOS, and pgQue are brought up as alternatives or potential integrations. One commenter specifically inquired why `pg_durable` wasn't built atop `pgmq`, to which a `pg_durable` committer responded, indicating that the state provider is an extensible component and welcoming contributions for a `pgmq`-based implementation.