I've operated petabyte-scale ClickHouse clusters for 5 years
A seasoned operator shares battle-scarred wisdom from five years of running petabyte-scale ClickHouse, detailing architectural trade-offs, painful upgrades, and the dark arts of ingestion. This practical, no-nonsense guide resonates deeply with HN's data infrastructure crowd, highlighting the brutal realities behind seemingly simple solutions.
The Lowdown
Javi Santana's article, penned from his experience at Tinybird, distills five years of operating petabyte-scale ClickHouse clusters. Targeting those already deep in the ClickHouse trenches, the piece offers a candid look at the challenges and hard-won lessons since version 18.4, aiming to help readers navigate common pitfalls and manage the beast beyond initial setup.
- Architecture & Scaling: Santana discusses standard replica/shard models, questioning the industry's shift to compute-storage separation for high-performance needs, and details Tinybird's custom load-balancing for diverse workloads.
- Storage Strategies: The author highlights ClickHouse's open-source limitations with cloud storage, advocating for zero-copy replication despite its known bugs and the maintainers' reservations, and outlines a hot/cold storage strategy using local SSDs for latency-sensitive data.
- Painful Upgrades & Testing: A rigorous, multi-stage upgrade process, refined over four years for zero-downtime, is described. Santana details potential issues like data format changes, SQL behavior shifts, and performance regressions, stressing comprehensive CI/CD testing on multi-node clusters.
- Cost & Personnel: An overview of cluster cost components (replicas, ZooKeeper, storage, load balancers, backups) is provided, alongside rules of thumb for hardware sizing and the critical need for dedicated personnel to manage the complexity.
- Ingestion Challenges: Dubbed the "fun" part, ingestion is presented as a delicate balance of merges, inserts, reads, mutations, and table design. Santana details common struggles—data loss, duplication, OOMs, backpressure—and offers strategies like batching and careful partitioning.
Ultimately, the article frames Tinybird's mission not as ClickHouse hosting, but as solving analytics problems by abstracting these very operational complexities, subtly pitching it as an antidote to the DIY pain described.
The Gossip
Trademark Tribulations & Corporate Caution
The extensive use of the `®` trademark symbol throughout the article caught many readers' attention, sparking discussion about its necessity and the legal landscape surrounding open-source projects managed by commercial entities. Commenters speculated that this was a defensive move by Tinybird, given its own commercial offerings built on ClickHouse, highlighting concerns about potential intellectual property conflicts similar to past high-profile cases.
ClickHouse Controversies & Architectural Quandaries
Several commentators challenged specific architectural and operational choices mentioned in the article. Points of contention included the author's preference for HTTP over TCP for ClickHouse connections, given TCP's latency advantages; the debate over separating compute and storage (with some arguing ClickHouse's strength lies in local NVMe performance); and the practicality of testing all analytics queries in an ad-hoc OLAP environment. The discussion also touched upon the inherent trade-offs of column-store databases, particularly regarding ingestion latency and the role of Kafka for backpressure.
The Return of the DBA (and the Modern Dev)
A significant thread lamented the perceived loss of the traditional DBA role, which previously acted as a 'gatekeeper' for query quality and schema design. This led to a broader discussion about the increasingly multi-faceted responsibilities of modern 'full-stack' or 'DevOps' engineers, who are now expected to be experts in databases, cloud infrastructure, and more, often without commensurate compensation or dedicated time for deep specialization. While some argued developers *should* understand their environment, others expressed frustration at the unrealistic expectations and the resulting 'jack of all trades, master of none' scenario.