Managing Open Table Formats: Apache Iceberg

This study guide focuses on the management of open table formats, with a primary emphasis on Apache Iceberg. In the AWS ecosystem, these formats are essential for building transactional data lakes that provide database-like features on top of Amazon S3 storage.

Learning Objectives

Define the role of open table formats (Iceberg, Hudi, Delta Lake) in a modern data architecture.
Explain core features including ACID transactions, schema evolution, and time-travel queries.
Implement Apache Iceberg tables using Amazon Athena and AWS Glue Data Catalog.
Optimize storage costs through snapshot expiration and data maintenance practices.

Key Terms & Glossary

ACID Transactions: A set of properties (Atomicity, Consistency, Isolation, Durability) that guarantee data validity despite errors or power failures.
Schema Evolution: The ability to change a table's schema (adding, dropping, or renaming columns) without rewriting the entire dataset.
Snapshot: A state of a table at a specific point in time, represented by a set of metadata and data files.
Time Travel: The capability to query a table at a previous state by referencing a specific snapshot ID or timestamp.
Compaction: The process of merging many small files into fewer, larger files to improve query performance.

The "Big Idea"

Traditionally, data lakes were just collections of files (Parquet, CSV) in folders. This made it difficult to handle concurrent writes, updates, or changes to the data structure. Apache Iceberg acts as an abstraction layer. It brings the reliability and simplicity of SQL tables to the scale of big data, allowing data engineers to treat S3 objects as a high-performance relational database.

Formula / Concept Box

Athena Iceberg Table Creation

When creating an Iceberg table in Athena, specific TBLPROPERTIES must be defined to tell the engine how to handle the format.

Property	Description
`'table_type'='ICEBERG'`	Mandatory property to identify the table format.
`location`	Must point to an S3 path (not a specific file).
`partitioning`	Can use advanced transforms like `bucket(n, col)` or `day(col)`.

Hierarchical Outline

Introduction to Open Table Formats
- Abstraction Layer: Provides database-like functionality over S3.
- Major Formats: Apache Iceberg, Apache Hudi, and Delta Lake.
Core Capabilities of Apache Iceberg
- Transactional Consistency: Multiple writers can update the table safely.
- Schema Evolution: Changes are metadata-only; no data rewriting required.
- Hidden Partitioning: Users don't need to know the partitioning columns to get efficient queries.
AWS Integration & Implementation
- Amazon Athena: Native support for Read, Write, and DDL queries.
- AWS Glue: Acts as the Metastore for Iceberg tables.
- Lake Formation: Provides fine-grained access control (Row/Column level).
Management and Maintenance
- Snapshot Management: Vital for preventing S3 cost bloat.
- Lifecycle Management: Using native expiration commands vs. S3 bucket rules.

Visual Anchors

Iceberg Architecture Flow

Loading Diagram...

Time Travel Mechanism

This diagram illustrates how snapshots allow access to different "versions" of the table data.

\begin{tikzpicture}[node distance=2cm, every node/.style={draw, rectangle, rounded corners, minimum width=2.5cm}] \node (S1) {Snapshot 1$T=10:00)}; \node (S2) [right of=S1, xshift=2cm] {Snapshot 2$T=11:00)}; \node (S3) [right of=S2, xshift=2cm] {Snapshot 3$T=12:00)};

code

\draw[->, thick] (S1) -- (S2) node[midway, above] {\small INSERT};
\draw[->, thick] (S2) -- (S3) node[midway, above] {\small UPDATE};

\draw[dashed, ->, red] (4, -1.5) node[below] {Query @ T=10:30} -- (1, -0.5);
\draw[dashed, ->, blue] (8, -1.5) node[below] {Query Current} -- (8.5, -0.5);

\end{tikzpicture}

Definition-Example Pairs

Partition Evolution
- Definition: Changing the partitioning strategy of a table without rewriting existing data.
- Example: A table originally partitioned by year(order_date) is changed to month(order_date). New data is written monthly, while old data remains yearly, and the engine handles the logic.
UPSERT Operations
- Definition: A combination of UPDATE and INSERT; if a record exists, update it; if not, insert it.
- Example: Merging daily sales updates into a master customer table where existing customer records are refreshed and new customers are added in one atomic operation.

Worked Examples

1. Creating an Iceberg Table in Athena

sql

CREATE TABLE iceberg_sales (
    order_id int,
    amount double,
    category string
)
PARTITIONED BY (category, bucket(16, order_id))
LOCATION 's3://my-data-lake-bucket/sales_iceberg/'
TBLPROPERTIES (
    'table_type'='ICEBERG'
);

2. Time Travel Query

To see what the data looked like before a mistaken deletion:

sql

SELECT * FROM iceberg_sales 
FOR TIMESTAMP AS OF (current_timestamp - interval '1' hour);

Checkpoint Questions

Which AWS service is typically used as the metastore for Apache Iceberg tables?
What is the main difference between S3 Lifecycle rules and Iceberg Snapshot Expiration?
Why are columnar formats like Parquet preferred for the underlying data files in Iceberg?
True or False: Changing a column name in an Iceberg table requires rewriting all the data files.

Comparison Tables

Feature	Traditional S3 Folders	Apache Iceberg
Transactions	No (Single file atomic only)	Yes (Full ACID support)
Updates	Rewrite entire partition/table	Record-level (Merge-on-Read/Copy-on-Write)
Schema Evolution	Brittle (Manual Glue Schema updates)	Robust (Metadata-only changes)
Small File Problem	Requires manual glue jobs	Can use built-in maintenance/compaction

Muddy Points & Cross-Refs

S3 Lifecycle vs. Snapshot Expiration:

[!WARNING] Do NOT use standard S3 Lifecycle expiration rules to delete Iceberg files based on age alone. Iceberg manages complex relationships between metadata and data files. Deleting an object via S3 lifecycle might break the table's integrity. Always use the Iceberg Expire Snapshot procedure within your processing engine (Athena/Spark).
Small Files: Even though Iceberg handles updates, frequent small writes can lead to many small files. Periodic compaction is necessary to maintain query speed.
Cross-Reference: For security, see the AWS Lake Formation guide to understand how to apply permissions to these tables.

Managing Open Table Formats: Apache Iceberg for Data Engineering