7:50 · Music video
The Iceberg Open Lakehouse
Follow the stack from durable files through snapshots, catalogs, engines, governance, and agent-ready semantics.
Open videoOpen lakehouse architecture
Your data, in formats
anything can read.
An open lakehouse keeps analytical data in open formats on storage you control, and adds enough metadata that those files behave like tables. Any compliant engine can read them. No vendor sits between you and your own data.
This site is where I collect the reference material: the vocabulary, the layers, and the projects that carry them.
Featured music video
The open lakehouse in motion
A visual and musical walkthrough of the Iceberg architecture I write, teach, and build around.
Start with the vocabulary
Most disagreements about the lakehouse turn out to be two people using one word for different things. These entries fix the terms first.
- 01What is an open lakehouseA data architecture that keeps analytical data in open formats on object storage you control, readable by any compliant engine.
- 02Object storageThe durable, cheap byte store at the bottom of the stack, and the constraints it imposes on everything above it.
- 03File formatHow records are arranged inside a single file, which decides how much of it a query has to read.
- 04Table formatThe metadata layer that turns a collection of files into a table with atomic commits, schema evolution, and history.
- 05CatalogThe service that resolves table names, performs the atomic commit, enforces access, and hands out scoped credentials.
- 06Semantic layerWritten definitions of what the data means, so people and machines compute the same metric the same way.
The projects that carry the weight
Five open projects do most of the work in a modern lakehouse. One page each, covering what it is, where it fits, and where it stops.
Apache ParquetThe columnar file format most lakehouse data sits in, built to be small on disk and cheap to scan selectively.Apache ArrowA standard way to lay out tabular data in memory so processes and languages can share it without converting it first.Apache IcebergThe table format that turns files in object storage into a real table, with atomic commits, schema evolution, and history.Apache PolarisAn open catalog for Iceberg tables that resolves names, enforces access, and vends scoped credentials.Apache OssieAn incubating project defining a vendor-neutral standard for semantic metadata, so metrics are defined once and used everywhere.
Where to start reading
A few pieces from the Dremio blog that cover the ground this site is about.
- ArchitectureOpen Source and the Data LakehouseWhy the openness of each layer has to be assessed separately, rather than as one claim about a platform.
- Table formatsWhat Are Table Formats and Why Were They Needed?The path from raw files to table formats, and the specific problems that made them necessary.
- CatalogsApache Polaris: The Catalog Standard for Lakehouses and AIHow an open catalog standard keeps tables interoperable across engines and clouds.