Topic

Data Engineering

Pipelines, storage formats and moving data without losing it.

2articles

Data Engineering
Incremental Pipelines That Can Be Run Twice
Idempotency is the property that turns a fragile nightly job into something you can retry without thinking. Here is how to build it into a pipeline from the start.
Sep 5, 20244 min read
Data Engineering
How Parquet File Layout Decides Your Query Speed
Two datasets with identical rows and identical schemas can differ by an order of magnitude in scan time. The difference is layout - row groups, sort order and file size.
Jun 27, 20244 min read