Data Preparation for Bias Reduction: Splitting, Shuffling, and Augmentation

This guide covers critical techniques for ensuring data integrity and reducing prediction bias in machine learning workflows, specifically within the context of the AWS Certified Machine Learning Engineer - Associate curriculum.

Learning Objectives

After studying this guide, you should be able to:

• Explain how dataset shuffling prevents the model from memorizing sample order.
• Differentiate between Training, Validation, and Test datasets.
• Identify Data Augmentation techniques for image, text, and time-series data.
• Apply Amazon SageMaker Clarify metrics (CI, DPL) to identify pre-training bias.
• Prevent Data Leakage through proper splitting and normalization timing.

Key Terms & Glossary

• Facet: A specific feature or attribute in a dataset (e.g., gender, age group) analyzed for potential bias.
• Class Imbalance (CI): A metric indicating if one facet is significantly more frequent than others; a value near 0 denotes balance.
• Data Leakage: When information from the test/validation set "leaks" into the training process, causing artificially high performance metrics.
• Stratified Splitting: A technique ensuring each split (train/test) maintains the same proportion of labels as the original dataset.
• Synthetic Data: Artificially generated data used to balance classes when real data for a specific facet is scarce.

The "Big Idea"

[!IMPORTANT] The core goal of bias reduction in data preparation is Generalization. We want the model to learn underlying patterns that apply to new, unseen data rather than memorizing noise, order-based dependencies, or imbalances present in the training set.

Formula / Concept Box

Hierarchical Outline

1. Data Shuffling
   • Random Permutation: Breaks sequential dependencies.
   • Epoch-Based Shuffling: Re-randomizes between training passes.
   • Mini-Batch Shuffling: Ensures diversity within a single gradient update.
2. Dataset Splitting
   • Training Set: Used for model weight updates.
   • Validation Set: Used for hyperparameter tuning and early stopping.
   • Testing Set: Used for final evaluation; must remain "unseen."
3. Data Augmentation
   • Image: Rotation, flipping, scaling.
   • Text: Synonym replacement, sentence shuffling.
   • Time Series: Noise injection, interpolation.
4. AWS Tools for Bias Reduction
   • SageMaker Clarify: Detects pre-training and post-training bias.
   • SageMaker Data Wrangler: Provides no-code transformations for splitting and balancing.

Visual Anchors

The Data Preparation Pipeline

Visualizing Image Augmentation (Rotation)

Definition-Example Pairs

• Shuffling
  • Definition: Randomizing the order of data points to ensure the model doesn't learn patterns based on the sequence of input.
  • Example: In a dataset where all "Spam" emails appear first and "Not Spam" appear last, shuffling prevents the model from simply learning "the first 100 are always spam."
• Data Augmentation
  • Definition: Creating synthetic variations of existing data to increase diversity without collecting new samples.
  • Example: Rotating an MRI scan image by 10 degrees. The diagnosis remains the same, but the model learns to recognize the condition regardless of the patient's exact orientation in the scanner.

Worked Examples

Scenario: Handling a Rare Class

You are training a model to detect fraudulent credit card transactions. Only 0.1% of your data is fraudulent.

1. Identify Bias: Use SageMaker Clarify to compute the Class Imbalance (CI) metric. You find it is close to 1.
2. Mitigation Strategy:
   • Oversampling: Duplicate the minority fraud cases.
   • Augmentation: Use SMOTE (Synthetic Minority Over-sampling Technique) to generate synthetic fraud examples.
3. Splitting: Use Stratified Splitting in Data Wrangler to ensure that the 0.1% fraud cases are distributed proportionally across your train, validation, and test sets.

Checkpoint Questions

1. Why should you normalize data after splitting it into training and test sets?
2. What is the primary difference between a Random split and an Order split in SageMaker Data Wrangler?
3. How does epoch-based shuffling differ from mini-batch shuffling?

▶Click to see answers

1. To avoid Data Leakage. If you normalize using the mean/variance of the whole dataset, information from the test set influences the training set scales.
2. Random split is used when data order doesn't matter; Order split preserves sequence, which is vital for time-series data to prevent using future data to predict the past.
3. Epoch-based shuffles the entire dataset before each full pass; Mini-batch shuffles data only within the specific chunk being processed by the gradient update.

Muddy Points & Cross-Refs

• Data Leakage vs. Overfitting: Overfitting happens when a model learns the training noise too well. Data Leakage is a process error where the model "cheats" by seeing test data early. Both result in poor real-world performance.
• Stratification: Students often forget this. Without stratification, a random split of a rare class might result in a test set with zero examples of that class.

Comparison Tables

Data Splitting Techniques