Postgres data stored in Parquet on S3: LTAP architecture explained
Databricks unveils Lakebase and LTAP, an innovative architecture that rethinks Postgres from the storage layer up, leveraging cloud object storage like S3 and open formats such as Parquet. By decoupling compute and storage, it solves traditional database monolith challenges, enabling unlimited scalability, durable writes, and instant branching. This approach culminates in LTAP, unifying transactional and analytical workloads on a single, fresh data copy without the performance isolation issues of traditional HTAP systems, a feature likely to resonate deeply with developers on HN.
The Lowdown
Databricks introduces its Lakebase and LTAP (Lake Transactional/Analytical Processing) architecture, a fundamental reimagining of database storage built on Postgres principles but designed for the modern cloud. This approach addresses long-standing limitations of monolithic databases, such as scalability bottlenecks, durability concerns, and the contention between transactional and analytical workloads.
- Monolithic Database Challenges: Traditional databases like Postgres tightly couple the Write-Ahead Log (WAL) and data files on a single machine, leading to issues with data loss, difficult scaling for reads/HA (requiring full physical clones), and performance degradation when OLAP queries interfere with OLTP.
- Lakebase Architecture: The core innovation is to make Postgres compute instances stateless by externalizing the WAL and data files into independent, purpose-built cloud services.
- SafeKeeper (WAL Externalization): The WAL is moved to a distributed service called SafeKeeper, which uses Paxos-based replication for durability. This eliminates dependence on local disk flushes, enhancing write throughput and ensuring zero data loss even if individual nodes fail.
- PageServer (Data File Externalization): Data files are managed by PageServer, another distributed service that streams changes from SafeKeeper and asynchronously materializes pages into low-cost cloud object storage (the 'lake'). It acts as a write-through cache, maintaining high read performance through multi-layered caching.
- Key Benefits of Lakebase: This separation unlocks unlimited storage, serverless elastic compute (scaling to zero), durable writes, simpler high availability, and instant branching/cloning/point-in-time recovery—transforming database operations into a process as agile as code development.
- LTAP - Unifying Transactions and Analytics: LTAP extends Lakebase by storing the durable data in the object store using open columnar formats (like Parquet via Delta/Iceberg). This allows Postgres (for OLTP) and Lakehouse engines (for OLAP) to operate on a single, unified copy of data.
- Columnar Materialization with Semantics: PageServer transcodes Postgres row data into columnar Parquet format while preserving full Postgres type system and MVCC (multi-version concurrency control) semantics. This offers significant compression and analytical performance benefits without sacrificing transactional integrity.
- Freshness for Analytics: Analytical queries achieve real-time freshness by reading the bulk of data from the materialized object store and dynamically fetching the most recent, un-materialized changes from PageServer, all without burdening the transactional Postgres compute.
- Distinction from HTAP: Unlike traditional HTAP systems that attempt to combine OLTP and OLAP into a single engine, LTAP maintains separate, optimized compute engines for each workload, unifying at the storage layer. This avoids common HTAP pitfalls like incomplete feature sets, lack of ecosystem, and performance contention.
This architecture represents a strategic shift from monolithic database designs, offering a robust, scalable, and cost-effective solution that leverages cloud-native principles to unify operational and analytical data processing within a single, open data layer.