HN
Today

Neki

PlanetScale introduces Neki, a new platform preview for sharded PostgreSQL, leveraging their extensive experience with large-scale MySQL to tackle Postgres's scaling limitations. It aims to provide robust distributed capabilities without compromising native Postgres features or forcing complex application-level sharding. This launch addresses a critical pain point for developers facing rapidly growing databases, offering a "stick to Postgres" approach for scalable infrastructure.

26
Score
4
Comments
#7
Highest Rank
19h
on Front Page
First Seen
Sep 10, 4:00 PM
Last Seen
Sep 11, 10:00 AM
Rank Over Time
9777101317131115151717182422191819

The Lowdown

PlanetScale, well-known for its expertise in sharding MySQL, has officially launched "Neki" as a platform preview, aiming to solve the persistent scaling challenges faced by PostgreSQL users. Drawing upon eight years of running some of the world's largest sharded MySQL clusters, Neki is designed to help organizations move beyond the limitations of single-machine Postgres databases.

  • Addressing Scaling Bottlenecks: Neki targets common Postgres pain points, including the difficulty of managing large tables, time-consuming backups, connection limits, and complex schema changes that often require maintenance windows. Existing distributed solutions typically force compromises, such as application-level sharding or sacrificing native Postgres features.
  • Core Principle: "Stick to Postgres": Neki's foundational principle is to maintain true Postgres functionality. Each shard runs a real Postgres instance, preserving full SQL support, extensions, and expected performance characteristics, unlike other solutions that might abstract or modify the underlying database.
  • Architectural Components:
    • Neki Routers: These act as the entry point for applications, speaking the Postgres wire protocol. They parse queries, develop distributed execution plans, send work to relevant shards, and aggregate results seamlessly.
    • Sharding & Shard Groups: Data is distributed across real Postgres instances (primary + replicas across availability zones). Shards are organized into configurable groups, allowing different workloads or tables to have tailored instance sizes, replica counts, and Postgres parameters.
    • Connection Pooling: Neki integrates sidecars with each Postgres instance for advanced connection management, offering more intelligent pooling than external tools like PgBouncer by controlling both connection ends.
    • Control Plane: This central component orchestrates critical operations, including health monitoring, planned switchovers, unplanned failovers, and coordinating workflows for resharding, schema changes, and upgrades.
    • Data Topology: A JSON configuration defines how logical tables map to physical shards, specifying shard indexes for routing and shard groups for distribution.
  • Online Operations: A key feature is Neki's ability to perform operations that typically require downtime (like schema changes, version upgrades, and resharding) as built-in, zero-downtime workflows, accessible via standard psql connections.
  • Flexible Deployment: Neki can initially be deployed unsharded, providing benefits like improved connection pooling and online DDL, with the option to seamlessly reshard the cluster later as demand grows.
  • Platform Preview Status: Neki is currently in a platform preview phase, meaning it's not yet recommended for production workloads due to potential breaking changes. PlanetScale encourages users to provide feedback during this period.

In essence, Neki aims to deliver a robust, scalable, and operationally streamlined Postgres solution that allows users to leverage the power of distributed databases without deviating from the native PostgreSQL experience, leveraging PlanetScale's battle-tested sharding expertise.