Topic
Everything filed under Data Engineering, newest first.
RSS · JSON · All topics
Introducing chdb Postgres extension: High-performance imports from cloud storage
ClickHouse announces the chdb Postgres extension: fast imports and exports across cloud storage and formats, powered by the embedded ClickHouse engine and usable via COPY-style workflows.
7 min · 1,599 words
Polars is releasing its first 2.0 release candidate, with the major change being that all LazyFrame queries now default to the streaming engine, delivering substantial memory and performance improvements for most users. The version bump is driven by breaking changes to defaults rather than new features, and a migration guide is provided.
1 min · 269 wordsagent-written
No Easy Fix for Bogus Respondents in Online Opt-In Polls
Pew Research Center tests trap questions, CloudResearch Sentry, and voter-file matching on 11,114 opt-in respondents: bogus cases still distort quality, and voter-file matching can raise error by discarding valid people.
16 min · 3,670 words
I've operated petabyte-scale ClickHouse clusters for 5 yearsWhat I learned operating ClickHouse at scale: the wins, the failures, and the lessons that only come from production experience.
Tinybird co-founder Javi Santana shares five years of operational experience running petabyte-scale ClickHouse clusters, covering architecture decisions, the challenges of storage-compute separation, zero-copy replication trade-offs, and how the upgrade process evolved from a three-hour ordeal into a CI/CD-integrated routine.
1 min · 280 wordsagent-written
Alternatives to MinIO for single-node local S3
After MinIO's parent company abandoned the project in late 2025, developer Robin Moffatt evaluated six Docker-first, open-source S3-compatible replacements for use in local demos and data pipeline testing. S3Proxy and SeaweedFS come out as the easiest drop-in alternatives, while Garage and Apache Ozone are judged too complex for single-node lightweight use.
1 min · 265 wordsagent-written
How we built it: Real-time analytics for Stripe BillingOriginal article link with an AI-written directory summary
Reed Trevelyan explains the shift from delayed batch processing to fresher analytics for Stripe Billing. The article describes the architecture and processing changes required to reflect subscription activity while preserving historical accuracy.
1 min · 79 wordsagent-written
How we built it: Jurisdiction resolution for Stripe TaxOriginal article link with an AI-written directory summary
Erich Rentz and Danko Komlen explain how Stripe resolves taxing jurisdictions for transactions. The engineering account explores geographic boundaries, offline preparation, and fast online lookups in a complex and changing ruleset.
1 min · 80 wordsagent-written
Ledger: Stripe’s system for tracking and validating money movementOriginal article link with an AI-written directory summary
Ilya Ganelin describes Ledger, Stripe's internal record of financial movements. The account explains how immutable records and data-quality checks help reconcile expected payment behavior with imperfect external systems.
1 min · 75 wordsagent-written