Knowledge base
The vocabulary of the open lakehouse.
Each entry covers one idea: what it is, the problem it solves, where it stops, and which primary sources to read next. Start at the top if the whole architecture is new to you, or go straight to a technology if you already know where you are.
Concepts
The layers and ideas that make up the architecture, in the order they build on each other.
- 01What is an open lakehouseA data architecture that keeps analytical data in open formats on object storage you control, readable by any compliant engine.
- 02Object storageThe durable, cheap byte store at the bottom of the stack, and the constraints it imposes on everything above it.
- 03File formatHow records are arranged inside a single file, which decides how much of it a query has to read.
- 04Table formatThe metadata layer that turns a collection of files into a table with atomic commits, schema evolution, and history.
- 05CatalogThe service that resolves table names, performs the atomic commit, enforces access, and hands out scoped credentials.
- 06Semantic layerWritten definitions of what the data means, so people and machines compute the same metric the same way.
- 07Query enginesThe compute that reads the stack, kept separate from storage so you can run several and replace any of them.
- 08PartitioningHow table data is physically grouped so a query can skip whole sections of it, and why hiding the scheme matters.
- 09ACID and time travelHow a lakehouse gets safe concurrent writes and a readable past, using an atomic pointer swap rather than a lock.
- 10Table maintenanceCompaction, snapshot expiration, and orphan cleanup, and what happens to a table when nobody owns them.
- 11The four kinds of openOpen source, open formats, open interfaces, and open governance are separate claims, and conflating them is how lock-in survives.
- 12Portability and lock-inWhat it would actually cost to change each component, and why the answer is usually discovered too late.
- 13AI and the lakehouseWhy AI workloads make the same architectural properties matter more, and what they add on top.
Key technologies
The five open projects that carry most of the weight in a modern lakehouse, one page each.
Apache ParquetThe columnar file format most lakehouse data sits in, built to be small on disk and cheap to scan selectively.Read the entryApache ArrowA standard way to lay out tabular data in memory so processes and languages can share it without converting it first.Read the entryApache IcebergThe table format that turns files in object storage into a real table, with atomic commits, schema evolution, and history.Read the entryApache PolarisAn open catalog for Iceberg tables that resolves names, enforces access, and vends scoped credentials.Read the entryApache OssieAn incubating project defining a vendor-neutral standard for semantic metadata, so metrics are defined once and used everywhere.Read the entry