Topic
Everything filed under Data Engineering, newest first.
RSS · JSON · All topics
Rows vs Columns: How different are they, really?
Connor Tsui of Spiral explains how storing structured data in columns rather than rows changes how a database system works, covering row versus column stores, vectorized execution and lightweight compression, using a bike-share rides dataset as a running example.
22 min · 5,116 words
Why we're moving our warehouse off DuckDB
PostHog explains why, five months after rebuilding its data warehouse on DuckDB, it re-architected it again: single-node sizing in the cloud, DuckLake catalog limits at ~1 trillion rows, and the pain of scaling Postgres wire connections led it to rebuild on Trino with its own open table format, HogLake.
4 min · 962 words
FastBCP 1.6: Export Tables from SQL Server Backups
ARPEIO's FastBCP 1.6 exports a table straight from a SQL Server full backup (.bak) to Parquet with no restore or SQL Server instance, with benchmarks on a 320 GB backup and notes on loading results via FastTransfer.
12 min · 2,669 words
Announcing DuckDB 2.0.0 LTS “Cinnamon Teal”
The DuckDB team announces DuckDB 2.0.0 LTS, its biggest feature release so far: a polished Quack protocol for DuckDB as a server, asynchronous I/O, a first-class VARIANT type, agent mode, triggers, an extensible PEG-based SQL parser, plus breaking changes and migration notes.
12 min · 2,753 words
Can we replace a $150k Looker with a bucket of HTML files?
A three-person data platform team describes how an afternoon side project for hosting AI-generated HTML reports grew to rival their $150k-a-year Looker deployment in weekly users, and when building beats buying enterprise BI.
12 min · 2,796 words
How We Made Our MMDB Files a Third Smaller Without Changing a Byte of Data
IPinfo's Tiago Martins and Silvano Cerza explain how hash-consing the MMDB search tree, exploiting how repetitive internet address data is, shrank IPinfo's 89 daily databases from 91.2 GB to 62.8 GB (31%, up to 80% for ipinfo_privacy) with no compression and no reader changes, how they proved the output is the same database, and why the mmdbshrink tool and mmdbctl update are open source.
7 min · 1,588 words
The Polars team ships Polars 2.0: initial out-of-core spill-to-disk support on by default, core engine and optimizer speedups, first-class SQL that leads DuckDB and DataFusion on TPC-H and TPC-DS style benchmarks, a new Map dtype, and stricter dtype handling, plus a detailed benchmark appendix.
6 min · 1,287 words
Accelerated Out of Core Shuffling
Benjamin Zaitlen explains RapidsMPF’s reusable out-of-core shuffler for distributed analytics—how spilling turns shuffle OOM headaches into a budgetable resource, and what it takes to push shuffle bandwidth toward terabytes per second.
15 min · 3,466 words
What happens when you analyze college football like the CIA?
What happens when you analyze college football like the CIA? A couple weekends ago, the Illinois football team lost to Duke at home, 31–27.
11 min · 2,518 words
Inline vs. Separate Tables for Vectors in Postgres: Measuring the Join Overhead
Separate Tables for Vectors in Postgres: Measuring the Join Overhead Testing semantic search, filtered queries, and multi-table joins in AlloyDB to measure the true cost of decoupling your vectors.
10 min · 2,296 words
Ghost Jobs Report, September 2026
Unlisted finds 28.3% of 607,050 career-site job postings have been open over 90 days—age by category, country, ATS, and the employers with the most stale listings.
7 min · 1,617 words
Writing Parquet Files Using Haskell
A practical walkthrough of generating Apache Parquet from Haskell: schema encoding, column chunks, and the tradeoffs of building interoperable analytical data files outside the JVM ecosystem.
11 min · 2,537 words
Aleksandar Filipovski, 2026-09-16 See also: John Salvatier’s excellent blog, Reality has a surprising amount of detail I read a comment somewhere that stuck with me, that went something like this:
8 min · 1,784 words
The query finished… Why is my Fabric SQL database still consuming CUs?
Two minutes of SQL database activity in Microsoft Fabric can mean ~17 minutes of compute billing. Nikola Ilic walks through CU metering with application, development, and troubleshooting examples.
8 min · 1,796 words
Beyond the model: Engineering AI infra with scientific judgementHow Airbnb's agent harness encodes scientific methodology for unstructured data exploration.
Ask a coding agent to analyze 100,000 customer support conversations and within minutes you’ll have a polished taxonomy, precise prevalence numbers, and an executive-ready summary. What you can’t see is the investigation that produced them: the methods it chose, the evidence it weighed, how much to trust it, or whether a second request would agree. All that reaches you is the polish. The model is undeniably intelligent, but intelligence without methodology is not science.
5 min · 1,106 words
Same Job, Two Engines: InnoDB and Iceberg Internals
Marc Paul compares running the same stock-analysis workload on MySQL InnoDB (OLTP) and an Iceberg lakehouse (OLAP): what InnoDB optimizes for, and the two mechanisms that make a lakehouse work — read-side pruning and the atomic pointer-swap commit.
14 min · 3,128 words
LLM Classification Is Feature Engineering
Taylor Pospisil argues LLMs work better as feature generators than as end-to-end classifiers, covering calibration, thresholding, cost, and how to treat model outputs as engineered features.
13 min · 3,039 words
Eddie argues that Pandas forces data practitioners to adopt distributed compute infrastructure long before their data sizes justify it, and that DuckDB and Polars can fill the gap for workloads up to around 100 GB on a single machine. Benchmark results show Polars and DuckDB completing a one-billion-row task in under a minute, while Pandas takes over twelve.
1 min · 268 wordsagent-written
MotherDuck's Mehdi Ouazza benchmarks the DuckDB 2.0 alpha on a laptop and S3, covering async I/O read-ahead that makes S3 reads 2-3x faster with no query changes, faster recursive CTEs for deep hierarchies, shredded VARIANT columns that beat JSON parsing about sixfold, and smaller wins from the commit log.
10 min · 2,220 words
Planet Labs' Open Satellite Feed
Mark Litwintschik surveys Planet Labs' satellite fleet and walks through practical use of the company's freely available Disaster Data feed, which publishes pre- and post-event imagery from natural disasters as DuckDB-friendly Parquet files hosted on Cloudflare.
1 min · 260 wordsagent-written