Lab: Building a Serverless Data Lake with AWS Glue and Amazon Athena

This lab provides a hands-on introduction to the AWS serverless analytics ecosystem. You will learn how to store raw data in Amazon S3, use AWS Glue to automatically discover the schema, and perform SQL-based analysis using Amazon Athena—all without managing a single server.

[!WARNING] Remember to run the teardown commands at the end of this lab to avoid ongoing charges for stored data or metadata. This lab is mostly Free Tier eligible, but Athena queries cost $5 per TB scanned (prorated).

Prerequisites

Before starting, ensure you have:

• An AWS Account with administrative access.
• AWS CLI installed and configured (aws configure).
• Basic knowledge of SQL (SELECT, FROM, WHERE).
• A sample dataset (save a few lines of CSV data as customer_data.csv).

Learning Objectives

By the end of this lab, you will be able to:

1. Configure S3 as a landing zone for raw structured data.
2. Deploy an AWS Glue Crawler to populate the Glue Data Catalog.
3. Execute SQL queries against S3 data using the Athena serverless engine.
4. Understand the cost and performance benefits of serverless analytics.

Architecture Overview

The following diagram illustrates the data flow: Raw data is uploaded to S3, indexed by Glue, and queried via Athena.

Step-by-Step Instructions

Step 1: Create the Data Lake Storage

First, we need an S3 bucket to act as our data lake storage and another for Athena query results.

▶Console alternative

1. Navigate to S3 in the AWS Console.
2. Click Create bucket.
3. Bucket name: brainybee-lab-data-<YOUR_NAME>.
4. Click Create bucket at the bottom. Repeat for the results bucket.

📸 Screenshot: S3 Bucket list showing both your data and results buckets.

Step 2: Upload Sample Data

Create a file named customer_data.csv with these contents:

Upload it to a "folder" (prefix) in your bucket:

Step 3: Create an IAM Role for Glue

AWS Glue needs permission to read your S3 bucket and write to the Data Catalog.

Step 4: Catalog Data with Glue Crawler

We will create a crawler to scan the S3 bucket and create a table in the Glue Data Catalog.

▶Console alternative

1. Navigate to AWS Glue > Crawlers.
2. Click Create crawler.
3. Name: brainybee-crawler. Click Next.
4. Add a data source: Select S3, choose your customers/ path. Click Next.
5. Choose the IAM role BrainyBeeGlueRole. Click Next.
6. Target database: Create brainybee_db. Click Finish.
7. Select the crawler and click Run crawler.

Step 5: Querying Data with Athena

Once the crawler finishes, Athena can query the new table using standard SQL.

Checkpoints

1. S3 Verification: Run aws s3 ls s3://brainybee-lab-data-<UNIQUE_ID>/customers/. You should see customer_data.csv.
2. Glue Verification: Check the Glue Console. Under Databases > Tables, you should see a table named customers with columns id, name, signup_date, and plan correctly identified.
3. Athena Verification: In the Athena Console, run SELECT count(*) FROM customers. The result should be 3.

Concept Review

Visualizing Concepts: The Serverless Data Lake

\begin{tikzpicture}[node distance=2cm] \draw[thick, fill=blue!10] (0,0) rectangle (3,2) node[pos=.5] {S3 (Storage)}; \draw[thick, fill=green!10] (4,0) rectangle (7,2) node[pos=.5] {Glue (Metadata)}; \draw[thick, fill=orange!10] (8,0) rectangle (11,2) node[pos=.5] {Athena (Query)}; \draw[->, >=stealth, thick] (3,1) -- (4,1) node[midway, above] {Index}; \draw[->, >=stealth, thick] (7,1) -- (8,1) node[midway, above] {SQL}; \node at (5.5, -1) {No Servers to Manage (Serverless)}; \end{tikzpicture}

Troubleshooting

Cost Estimate

• S3 Storage: ~0.023 per GB/mo. (For this lab: < $0.01)
• Glue Crawler: $0.44 per DPU-Hour. (Minimum 0.25 DPUs for 1 min: < $0.10)
• Amazon Athena: $5.00 per TB scanned. (This lab scans bytes: < $0.01)
• Total Estimated Cost: <$0.20

Challenge

Can you convert the data to Apache Parquet format? Parquet is a columnar format that makes Athena queries faster and cheaper because it reduces the amount of data scanned.

▶Hint

Use an AWS Glue Studio visual job to read from the customers table and write to a new S3 folder with the format set to "Parquet". Then, update your Athena query to point to the Parquet table.

Clean-Up / Teardown

To avoid ongoing charges, delete the resources created: