AWS Lake Formation: Centralized Governance and Fine-Grained Access Control

This study guide focuses on managing permissions through AWS Lake Formation to secure data across Amazon S3, Amazon Athena, Amazon Redshift Spectrum, and Amazon EMR. It covers the shift from coarse-grained IAM policies to fine-grained, cell-level security.

Learning Objectives

After studying this guide, you should be able to:

• Explain the process of registering Amazon S3 locations with Lake Formation.
• Define and apply Fine-Grained Access Control (FGAC) at the database, table, column, and row levels.
• Differentiate between Tag-Based Access Control (TBAC) and Attribute-Based Access Control (ABAC).
• Implement best practices for Data Lake administration and cross-account data sharing.
• Understand the integration points between Lake Formation and compute engines like Athena and Redshift Spectrum.

Key Terms & Glossary

• Principal: An IAM user, role, or group to which Lake Formation permissions are granted.
• Fine-Grained Access Control (FGAC): The ability to restrict access to specific rows or columns within a table, rather than just the whole table or bucket.
• LF-Tags: Key-value pairs used in Lake Formation to manage permissions at scale through Tag-Based Access Control.
• Data Lake Administrator: A designated IAM user/role with full access to Lake Formation settings and the ability to grant permissions to others.
• Service-Linked Role (SLR): A predefined IAM role that allows an AWS service to call other services on your behalf; not recommended for production Lake Formation data access.
• Credential Vending: The process where Lake Formation provides temporary credentials to a compute engine to access S3 data on behalf of a user.

The "Big Idea"

Traditionally, securing a data lake required complex Amazon S3 bucket policies and IAM roles, which are often "all-or-nothing." AWS Lake Formation centralizes security by acting as a governance layer over the AWS Glue Data Catalog. Instead of managing access at the storage layer (S3), you manage it at the logical layer (Tables/Rows), allowing different teams to use the same S3 data while seeing only the specific subsets they are authorized to view.

Formula / Concept Box

Hierarchical Outline

1. Setup and Registration
   • Data Lake Admin: Designate a non-root IAM user as the administrator.
   • S3 Registration: Register S3 prefixes to bring them under Lake Formation management.
   • Datalake Settings: Configure default permissions for new resources (opt-out of 'IAMAllowedPrincipal' for new databases).
2. Defining Permissions
   • Identity-based: Granting permissions directly to IAM principals.
   • Tag-based (TBAC): Using LF-Tags to grant access (e.g., Tag Sensitivity=Public grants access to all marked tables).
   • Cell-Level Security: Combining column-level exclusion and row-level filtering.
3. Integration and Consumption
   • Amazon Athena: Queries respect LF permissions automatically.
   • Amazon Redshift Spectrum: Uses Lake Formation to authorize access to external schemas.
   • Amazon EMR: Requires specific configuration (e.g., EMR Record Server) to enforce FGAC.

Visual Anchors

Data Access Workflow

Permission Layering

\begin{tikzpicture}[node distance=1.5cm, every node/.style={rectangle, draw, fill=orange!10, text width=4cm, align=center}] \node (s3) {Storage: Amazon S3$Encrypted, No Public Access)}; \node (lf) [above of=s3] {Governance: Lake Formation$Row/Column Filtering)}; \node (glue) [above of=lf] {Metadata: Glue Data Catalog$Schemas & Tables)}; \node (app) [above of=glue] {Compute: Athena, EMR, Redshift$End-User Access)}; \draw[thick, <->] (s3) -- (lf); \draw[thick, <->] (lf) -- (glue); \draw[thick, <->] (glue) -- (app); \end{tikzpicture}

Definition-Example Pairs

• Row-Level Security: Restricting access to records based on a predicate.
  • Example: A multi-tenant application where the client_id in the data must match the user's client_id attribute.
• Column-Level Security: Restricting access to specific vertical slices of a table.
  • Example: An HR database where everyone can see Employee_Name and Department, but only managers can see Salary.
• Cross-Account Sharing: Sharing Data Catalog resources with other AWS accounts without replicating data.
  • Example: A Producer account shares a Glue table with a Consumer account using AWS Resource Access Manager (RAM) via Lake Formation.

Worked Examples

Step-by-Step: Masking PII for a Healthcare Dataset

Scenario: You have a table patient_records in S3. Analysts need to query diagnosis data but must not see the phone_number or email columns. Additionally, they should only see data for patients in the 'Cardiology' department.

1. Register Location: In Lake Formation Console, go to Data Lake Locations and register the S3 path s3://my-health-data/records/.
2. Define Row Filter: Create a Data Filter named CardiologyOnly for table patient_records with the expression department = 'Cardiology'.
3. Grant Permissions:
   • Go to Permissions > Data lake permissions.
   • Select the Principal (AnalystRole).
   • Select the Database and Table.
   • Choose Data Filters and select CardiologyOnly.
   • In Column Permissions, select "Exclude columns" and check phone_number and email.
4. Verification: When the analyst runs SELECT * FROM patient_records in Athena, the excluded columns will not appear, and the results will be limited to Cardiology rows.

Checkpoint Questions

1. Why is it a best practice to avoid using the root user as the Data Lake Admin?
2. What is the benefit of using LF-Tags over standard IAM policies for a data lake with 500+ tables?
3. When registering an S3 location in Lake Formation, should you keep or remove existing S3 bucket policies for the same principals?
4. Which AWS service does Lake Formation use to facilitate cross-account data sharing?

▶Click to see answers

1. To follow the principle of least privilege and ensure accountability through dedicated IAM roles.
2. Scalability. You can assign tags to tables and permissions to tags, rather than managing 500 individual resource-based policies.
3. Avoid additional bucket policies to ensure a single, consistent source of truth for permissions within Lake Formation.
4. AWS Resource Access Manager (RAM).

Comparison Tables

Muddy Points & Cross-Refs

• Service-Linked Roles (SLR): Be careful! While easy to set up, SLRs are not recommended for production. Instead, create a custom IAM role for registering S3 locations to avoid policy size limits and improve security.
• IAMAllowedPrincipal: This is a default group that allows legacy IAM-based access. For Lake Formation to truly manage security, you must remove this group's permissions from your databases/tables.
• EMR Limitations: Amazon EMR on EC2 does not support SLR-registered locations for data access. Always use custom roles for EMR-related data locations.