Curriculum Overview: Evaluating FM Applications (RAG, Agents, & Workflows)

Welcome to the curriculum overview for evaluating Foundation Model (FM) applications. This guide maps out the critical concepts needed to assess the performance of GenAI applications—specifically those utilizing Retrieval-Augmented Generation (RAG), Bedrock Agents, and complex workflows—in alignment with the AWS Certified AI Practitioner (AIF-C01) exam objectives.

---

Prerequisites

Before diving into evaluation frameworks, learners must have a solid grasp of foundational AI/ML and AWS concepts. Ensure you are comfortable with the following:

• Foundation Models (FMs): Understanding what large language models are, how they are pre-trained, and the concept of inference parameters (temperature, top-p).
• RAG Architecture: Familiarity with Retrieval-Augmented Generation, vector databases (e.g., Amazon OpenSearch, pgvector in Aurora), and embeddings.
• Prompt Engineering: Basic understanding of how context, instructions, and few-shot prompting influence model output.
• AWS AI Services: Basic knowledge of Amazon Bedrock, Bedrock Knowledge Bases, and Bedrock Agents.

[!IMPORTANT] If the concept of converting text into numerical vectors (embeddings) is fuzzy, review Unit 2: Fundamentals of Generative AI before proceeding.

---

Module Breakdown

This curriculum is divided into a structured progression, taking you from core mathematical metrics to complex workflow evaluations.

---

Learning Objectives per Module

Module 1: The Dual-Pronged Evaluation Strategy

To effectively evaluate an FM, organizations must balance automated quantitative metrics with qualitative human assessment.

• Understand Automated Evaluation: Learn how to set up automated pipelines using benchmark datasets to gauge accuracy, speed, and scalability.
• Implement Judge Models: Understand the architecture of using a secondary AI to score the outputs of your primary FM.
• Apply Human-in-the-Loop: Recognize when to use human panels for subjective criteria like emotional intelligence, bias detection, and helpfulness.

Module 2: Standard Evaluation Metrics

Master the core quantitative metrics used to grade model text generation against human-written baselines.

• Differentiate between text evaluation metrics: Know exactly when to use ROUGE vs. BLEU vs. BERTScore.
• Understand N-gram overlap: Grasp the mathematical basis of how traditional metrics check for word sequence matches.

▶Deep Dive: The "Big Three" Metrics

Metric	Primary Use Case	How it Works	Example Real-World Application
ROUGE	Summarization	Measures overlap of n-grams (consecutive words) between FM output and human reference.	Grading an FM that summarizes 10-page legal contracts into 1-paragraph briefs.
BLEU	Translation	Evaluates the precision of phrase generation against reference texts.	Checking the accuracy of an English-to-Spanish translation tool.
BERTScore	Semantic Similarity	Uses deep contextual embeddings to check if the meaning matches, regardless of exact wording.	Grading a chatbot where "The car halted" is evaluated as correct against the reference "The vehicle stopped suddenly."

[!NOTE] Many evaluation metrics rely on the concepts of Precision and Recall, combined into the F1 Score: $F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

Module 3: Evaluating RAG Architectures

RAG applications have two distinct failure points: failing to retrieve the right data, or failing to generate a good answer from that data. You must learn to evaluate both independently.

• Evaluate the Retrieval Phase: Assess whether the vector database returns the correct chunks.
  • Context Relevance: Does the retrieved data actually relate to the user's query?
  • Context Coverage: Did the retrieval pull all necessary information?
• Evaluate the Generation Phase: Assess what the LLM does with the retrieved data.
  • Faithfulness: Is the answer derived only from the retrieved context (no hallucinations)?
  • Completeness & Correctness: Does the answer fully and accurately resolve the prompt?

Module 4: Assessing Agents and Multi-Step Workflows

Agents (like Amazon Bedrock Agents) perform multi-step tasks, utilizing tools, APIs, and RAG iteratively. Evaluating them goes beyond single-prompt metrics.

• Evaluate Multi-Step Reasoning: Assess how well the agent uses Chain-of-Thought reasoning to plan its actions.
• Measure Task Completion Rates: Shift from evaluating text quality to evaluating business outcomes (e.g., "Did the agent successfully process the refund?").
• Monitor System Latency and Cost: Understand the operational tradeoffs of agentic workflows, assessing whether the token consumption justifies the business value.

---

Success Metrics

How will you know you have mastered this curriculum? You should be able to:

1. Select the Correct Metric: Given a scenario (e.g., "You are building a French-to-English translator"), instantly identify the correct evaluation metric (BLEU).
2. Diagnose RAG Failures: Given a poorly performing RAG system, determine if the issue lies in the retrieval phase (needs better chunking/embeddings) or the generation phase (needs better prompt constraints).
3. Design an Evaluation Pipeline: Architect a dual-pronged evaluation system in Amazon Bedrock that utilizes both a Judge Model for speed and a human panel for emotional intelligence.

---

Real-World Application

Why does mastering FM evaluation matter in the real world?

Moving a generative AI workload from a "Proof of Concept" (PoC) to a production-ready enterprise application is impossible without a robust evaluation framework.

If you deploy a customer-facing support chatbot using Amazon Bedrock, you are exposed to risks like hallucinations (making up return policies) and nondeterminism (answering the same question differently). By implementing automated RAG evaluation metrics (specifically Faithfulness and Correctness), you establish a quantified safety net. This allows developers to tweak hyper-parameters, adjust vector chunking strategies, and update prompts with the mathematical certainty that their changes are improving the system, ultimately ensuring the AI safely drives business value.