Establishing and Monitoring Performance Baselines in Machine Learning

Performance baselines are the foundational "yardsticks" used to evaluate the effectiveness of machine learning models. Without a baseline, it is impossible to quantify whether a complex model is actually providing value or if it is simply consuming more resources for marginal gains.

Learning Objectives

By the end of this guide, you should be able to:

• Define the purpose of a performance baseline in the ML lifecycle.
• Identify appropriate "simple models" to serve as initial baselines.
• Explain how to use SageMaker Monitoring to create constraints from baselines.
• Differentiate between data drift and model drift monitoring.
• Select evaluation metrics that align with specific business objectives.

Key Terms & Glossary

• Baseline: A reference point (often a simple model or heuristic) used to compare the performance of more complex models.
• Data Drift: A change in the statistical properties of input data over time (e.g., a feature's mean shifts).
• Model Drift (Concept Drift): A decline in model predictive power due to changes in the relationship between input features and the target variable.
• Ground Truth: The actual, verified "correct" answer for a prediction, used to calculate model drift in production.
• Constraint: A specific threshold (e.g., Recall > 0.8) generated during a baseline job that triggers an alert if violated.
• Deequ: An open-source tool (used by AWS) built on Apache Spark for measuring data quality in large datasets.

The "Big Idea"

Think of a performance baseline as the "Minimum Viable Model." In the same way an architect uses a scale model before building a skyscraper, an ML engineer uses a baseline to prove that the problem is solvable and to set a floor for performance. If a simple linear regression achieves 80% accuracy, a complex deep learning model costing 10x more must significantly exceed that 80% to justify its existence.

Formula / Concept Box

Hierarchical Outline

1. Phase 1: Initial Baseline Creation
   • Simple Model Selection: Use Linear Regression (regression) or Logistic Regression (classification).
   • Metric Selection: Align metrics with Business Objectives (e.g., Precision vs. Recall).
   • Data Validation: Use simple models to identify data leakage or bias early.
2. Phase 2: SageMaker Baseline Jobs
   • Metric Computation: Automated jobs calculate statistics (mean, median, standard deviation).
   • Constraint Generation: SageMaker creates a constraints.json file representing the "normal" range for these metrics.
3. Phase 3: Production Monitoring
   • Data Drift: Comparing real-time input data to the baseline using Deequ.
   • Model Drift: Comparing model predictions to Ground Truth labels stored in S3.
   • Alerting: Integration with CloudWatch and SNS to trigger retraining or human intervention.

Visual Anchors

ML Baseline Workflow

Visualizing a Baseline (ROC Curve)

This diagram represents why we need a baseline. The diagonal line represents a random guess baseline ($AUC = 0.5$).

\begin{tikzpicture} % Axes \draw [->] (0,0) -- (5,0) node[right] {False Positive Rate}; \draw [->] (0,0) -- (0,5) node[above] {True Positive Rate}; % Random Guess Baseline \draw [dashed, gray] (0,0) -- (4,4) node[right] {Random Guess (Baseline)}; % Simple Model \draw [blue, thick] (0,0) .. controls (1,3) and (2,3.5) .. (4,4) node[above left] {Simple Baseline Model}; % Complex Model \draw [red, thick] (0,0) .. controls (0.5,3.5) and (1,4) .. (4,4) node[above left] {Optimized Model}; % Labels \node at (2,-0.5) {0}; \node at (4,-0.5) {1}; \node at (-0.5,4) {1}; \end{tikzpicture}

Definition-Example Pairs

• Simple Model Baseline: Starting with a model that has high interpretability and low compute cost.
  • Example: Using Logistic Regression to predict customer churn before trying a Gradient Boosted Tree (XGBoost).
• Performance Constraints: Hard limits set on evaluation metrics that indicate a model is no longer fit for purpose.
  • Example: Setting a constraint that Recall must not drop below 0.8. If a new batch of data results in a recall of 0.75, a CloudWatch alarm triggers.
• Data Drift Monitoring: Detecting when the distribution of the features entering the model changes.
  • Example: A housing price model trained on data from 2020 (low interest rates) receives 2024 data (high interest rates); the "Interest Rate" feature mean has drifted significantly.

Worked Examples

Step-by-Step: Comparing a Baseline to a Complex Model

Scenario: You are building a model to predict whether a credit card transaction is fraudulent.

1. Set the Baseline: You train a Logistic Regression model.
   • Result: Precision: 0.85, Recall: 0.70.
   • Observation: The model is fast and cheap, but misses 30% of fraud.
2. Train the Complex Model: You train an XGBoost model with hyperparameter tuning.
   • Result: Precision: 0.86, Recall: 0.92.
3. Analysis:
   • Precision Improvement: +1% (Marginal).
   • Recall Improvement: +22% (Significant).
   • Decision: The XGBoost model is justified because the significant increase in Recall (catching fraud) outweighs the extra compute cost compared to the baseline.

Checkpoint Questions

1. Why is it beneficial to start with a simple model like Linear Regression before moving to a Deep Learning architecture?
2. Which service is used to log results from SageMaker Drift Monitoring and trigger alerts?
3. True or False: The baseline job typically runs after the model is deployed to production.
4. What is the difference between Data Drift and Model Drift?

▶Click to see answers

1. Simple models provide a reference point for improvement, help identify data issues like bias/leakage, and reduce initial computational costs.
2. Amazon CloudWatch (integrated with SNS for notifications).
3. False. The baseline job runs before training or retraining to establish constraints.
4. Data Drift focuses on changes in input features; Model Drift focuses on changes in the accuracy/quality of predictions compared to the actual outcome (ground truth).

Muddy Points & Cross-Refs

• Data Drift vs. Model Drift: This is a common exam pitfall. Remember: Data Drift = Input features change (e.g., users get younger). Model Drift = Predictions are wrong (e.g., the model can no longer identify a fraudulent transaction even if input ranges look normal).
• Interpretation with Clarify: While baselines set the performance floor, SageMaker Clarify is used to explain why the model is making certain predictions and to detect bias. These are often used together in the "Analyze Model Performance" phase.

Comparison Tables

Simple Baseline vs. Advanced Model

Data Drift vs. Model Drift

[!IMPORTANT] For the AWS exam, remember that Deequ is the underlying engine for SageMaker Model Monitor's data quality checks. Also, recall that Model Drift requires you to periodically upload labeled data to S3 for comparison.