Indexed summary. This entry is an agent-written synopsis of an article first published at tinybird.co. Read the original for the full text.
Santana frames the post around a central observation: setting up ClickHouse is easy; keeping it running at petabyte scale is genuinely hard. The essay moves through architecture, storage, and upgrade concerns in roughly chronological order, reflecting lessons accumulated since running ClickHouse version 18.4.
Key points
- The standard shards-and-replicas architecture becomes expensive quickly: a 300 TB table replicated across 10 nodes for query capacity requires 3 PB of storage.
- Open-source ClickHouse lacks mature cloud storage support; Tinybird uses a modified zero-copy replication in a private fork, but notes that the upstream feature has been buggy and nearly removed by ClickHouse Inc.
- For latency-sensitive customers, a hot/cold architecture with local SSDs and S3 outperforms pure S3 storage; Tinybird also keeps a dedicated write-only replica to isolate ingestion from query traffic.
- Upgrades were originally three hours with two weeks of preparation; the team eventually built a backward-compatible rolling upgrade process integrated into CI/CD.
- Safe upgrade checklist: add a replica on the new version, monitor logs (many alarming messages are harmless), avoid DDL changes during the process, send read traffic first before shifting writes.
- Large part sizes improve read performance but make merges expensive; balancing merge aggressiveness against query performance is an ongoing operational concern.