Open lakehouse architecture
Your data, in formats
anything can read.
An open lakehouse keeps analytical data in open formats on storage you control, and adds enough metadata that those files behave like tables. Any compliant engine can read them. No vendor sits between you and your own data.
This site is where I collect the reference material: the vocabulary, the layers, and the projects that carry them.
Start with the vocabulary
Most disagreements about the lakehouse turn out to be two people using one word for different things. These entries fix the terms first.
- 01 What is an open lakehouse A data architecture that keeps analytical data in open formats on object storage you control, readable by any compliant engine.
- 02 Object storage The durable, cheap byte store at the bottom of the stack, and the constraints it imposes on everything above it.
- 03 File format How records are arranged inside a single file, which decides how much of it a query has to read.
- 04 Table format The metadata layer that turns a collection of files into a table with atomic commits, schema evolution, and history.
- 05 Catalog The service that resolves table names, performs the atomic commit, enforces access, and hands out scoped credentials.
- 06 Semantic layer Written definitions of what the data means, so people and machines compute the same metric the same way.
The projects that carry the weight
Five open projects do most of the work in a modern lakehouse. One page each, covering what it is, where it fits, and where it stops.
Where to start reading
A few pieces from the Dremio blog that cover the ground this site is about.
- Architecture Open Source and the Data Lakehouse Why the openness of each layer has to be assessed separately, rather than as one claim about a platform.
- Table formats What Are Table Formats and Why Were They Needed? The path from raw files to table formats, and the specific problems that made them necessary.
- Catalogs Apache Polaris: The Catalog Standard for Lakehouses and AI How an open catalog standard keeps tables interoperable across engines and clouds.