Knowledge base
The vocabulary of the open lakehouse.
Each entry covers one idea: what it is, the problem it solves, where it stops, and which primary sources to read next. Start at the top if the whole architecture is new to you, or go straight to a technology if you already know where you are.
Concepts
The layers and ideas that make up the architecture, in the order they build on each other.
- 01 What is an open lakehouse A data architecture that keeps analytical data in open formats on object storage you control, readable by any compliant engine.
- 02 Object storage The durable, cheap byte store at the bottom of the stack, and the constraints it imposes on everything above it.
- 03 File format How records are arranged inside a single file, which decides how much of it a query has to read.
- 04 Table format The metadata layer that turns a collection of files into a table with atomic commits, schema evolution, and history.
- 05 Catalog The service that resolves table names, performs the atomic commit, enforces access, and hands out scoped credentials.
- 06 Semantic layer Written definitions of what the data means, so people and machines compute the same metric the same way.
- 07 Query engines The compute that reads the stack, kept separate from storage so you can run several and replace any of them.
- 08 Partitioning How table data is physically grouped so a query can skip whole sections of it, and why hiding the scheme matters.
- 09 ACID and time travel How a lakehouse gets safe concurrent writes and a readable past, using an atomic pointer swap rather than a lock.
- 10 Table maintenance Compaction, snapshot expiration, and orphan cleanup, and what happens to a table when nobody owns them.
- 11 The four kinds of open Open source, open formats, open interfaces, and open governance are separate claims, and conflating them is how lock-in survives.
- 12 Portability and lock-in What it would actually cost to change each component, and why the answer is usually discovered too late.
- 13 AI and the lakehouse Why AI workloads make the same architectural properties matter more, and what they add on top.
Key technologies
The five open projects that carry most of the weight in a modern lakehouse, one page each.
Apache Parquet The columnar file format most lakehouse data sits in, built to be small on disk and cheap to scan selectively. Read the entry Apache Arrow A standard way to lay out tabular data in memory so processes and languages can share it without converting it first. Read the entry Apache Iceberg The table format that turns files in object storage into a real table, with atomic commits, schema evolution, and history. Read the entry Apache Polaris An open catalog for Iceberg tables that resolves names, enforces access, and vends scoped credentials. Read the entry Apache Ossie An incubating project defining a vendor-neutral standard for semantic metadata, so metrics are defined once and used everywhere. Read the entry