Domain 5 Project — make what you built survive contact with reality

This project is different from the other four, and it is worth saying why.

The others each build one system that grows. Domain 5's six task statements are cross-cutting concerns — two belong to the support agent, two to the research coordinator, one to the extraction pipeline, one to a long exploration. Forcing them into a single narrative would misrepresent the domain, so this project hardens the systems you already have instead of building a new one.

One insight runs through all six stages: context is lossy in specific, predictable ways. Summarization eats numbers and dates. Long inputs lose their middles. Verbose tool results crowd out what matters. Attribution disappears at the moment findings are merged. Every answer in this domain is a structure that survives one of those losses, and each stage builds one.

Domain 5 is 15% of the exam — the smallest block, and the one that punishes vagueness hardest.

The project at a glance

Domain: 5 · Context Management & Reliability (15%)
Stages: 6, one per task statement
Difficulty: Advanced
Total time: 190 minutes
You need: The systems from Domains 1 and 4, or time to stand up small versions
Based on: Exam guide Preparation Exercises 3 and 4

What you will be able to do

Keep transactional facts intact across a conversation long enough to be summarized, and say what summarization destroys first
Escalate on the three legitimate triggers and reject sentiment and self-reported confidence as proxies for complexity
Propagate an error with enough context for a coordinator to act, and handle locally what it need not hear about
Carry claim-source mappings and dates through synthesis, and distinguish a genuine conflict from a temporal difference
Calibrate a confidence threshold against a labelled set and defend automating a segment from its own accuracy rather than an aggregate

Before you start

This project assumes Domains 1 and 4. Stages 5.1 to 5.3 harden the support agent and coordinator from the Domain 1 project; stage 5.5 hardens the extraction pipeline from Domain 4. If you have those, you are ready.

If you do not, each stage can be worked from a small stand-in — a two-tool agent for 5.1 and 5.2, a coordinator with two subagents for 5.3 and 5.6, any scored extraction for 5.5. Budget an extra hour or so overall, and expect the stages to teach slightly less, because a system you built five minutes ago has not yet accumulated the mess these stages are about.

Two stages need something you cannot fake quickly:

5.1 needs a conversation long enough to trigger summarization. Script it rather than typing it — twenty or thirty turns of realistic back-and-forth, saved so you can replay the same one after each change.
5.4 needs a codebase big enough that exploring it degrades the session. Something unfamiliar and genuinely large. A tidy example project will not drift, and the drift is the lesson.

bash

mkdir ccarf-domain-5 && cd ccarf-domain-5
python3 -m venv .venv && source .venv/bin/activate
pip install anthropic
export ANTHROPIC_API_KEY=your-key-here

Build it in this order

Stages are grouped by the system they harden — support agent, then coordinator, then extraction pipeline, then a long exploration — rather than by guide order. 5.4 is last because it is the only one needing a session long enough to degrade, which is worth attempting once the shorter lessons are already in hand.

Six stages

5.1 · Facts that survive a long conversation · 30 min
BUILD — Run the support agent through a conversation long enough to trigger summarization, and watch an amount or a date go vague. Then lift the transactional facts — amounts, dates, order numbers, statuses — into a persistent case-facts block included in every prompt, OUTSIDE the summarized history. Trim the 40-field order lookup to the five fields that matter before it ever enters the context. WHY — Progressive summarization keeps the gist and loses the figures, and the figures are what the next decision turns on. A block outside the history is the structure that survives that loss. YOU SHOULD SEE — The same long conversation reaching turn thirty with the refund amount still exact, and a visibly shorter context once the tool output is trimmed at the door. TRAP — Trimming after the fact. Summarizing something you already paid to put in the context saves nothing — the trim belongs before the append. STUCK? — Print the full message list at turn five and turn twenty-five and diff them. The thing that went missing between the two is the thing that needed its own block.

Stage 5.1 in full

First, watch it happen

Run the support agent through a long conversation about a refund. Not a contrived one — a realistic thread where the customer mentions an amount early, wanders, and asks about it again much later.

Then ask, at turn thirty: what exactly was the refund amount?

If it comes back as "around 130" when the document said 129.99, or "earlier this month" when the order was placed on the 2nd, you have reproduced the thing this stage is about. Progressive summarization keeps the gist and drops the precision, and precision is what the next decision needs.

Put the facts somewhere summarization cannot reach

The structure is a case-facts block: extracted transactional facts, maintained separately, included in every prompt, outside the history that gets compressed.

python

class CaseFacts:
    """Facts that must survive summarization, kept outside the history.

    The conversation gets summarized; this does not. Anything a later decision
    turns on lives here rather than in the messages.
    """

    def __init__(self):
        self.facts = {}

    def record(self, **pairs):
        self.facts.update({k: v for k, v in pairs.items() if v is not None})

    def as_block(self):
        if not self.facts:
            return ""
        lines = [f"- {key}: {value}" for key, value in sorted(self.facts.items())]
        return "CASE FACTS (authoritative, do not restate from memory):\n" + "\n".join(lines)


def build_messages(history, case_facts, user_message):
    # The block goes in EVERY request, ahead of the (possibly summarized)
    # history. It is small, so it costs little; it is authoritative, so the
    # model prefers it to a half-remembered figure.
    system_parts = [BASE_SYSTEM_PROMPT]
    if block := case_facts.as_block():
        system_parts.append(block)
    return {
        "system": "\n\n".join(system_parts),
        "messages": history + [{"role": "user", "content": user_message}],
    }

Record into it whenever a tool returns something transactional:

python

order = run_tool("lookup_order", {"order_id": "A-4417"})
case_facts.record(
    order_id="A-4417",
    order_total=order["total"],
    placed_at=order["placed_at"],
    order_status=order["status"],
)

Replay the same thirty-turn conversation. The amount should now be exact at turn thirty, because it was never in the part that got compressed.

Trim the tool output at the door

The second half of this stage costs nothing and pays every turn. An order lookup returns forty fields; a refund decision needs five.

python

RETURN_RELEVANT = ("order_id", "total", "placed_at", "status", "refundable_until")

def trim(order):
    # BEFORE the append, not after. Summarizing something you already paid to
    # put in the context saves nothing.
    return {key: order[key] for key in RETURN_RELEVANT if key in order}

Multiple choice · Medium

A support agent handles long conversations. By turn thirty it gives vague answers about amounts and dates that were precise earlier in the thread. What structure fixes this?

Prove stage 5.1 before moving on

Replay the identical conversation before and after. If you changed the script as well, you measured nothing.
Print the full message list at turn five and turn twenty-five and diff them. What went missing is what needed its own block.
Count the context tokens before and after trimming. If the number did not move, you are trimming after the append rather than before it.

When you get stuck

Every stage above ends with a STUCK? line — the first thing worth checking, which is usually the thing actually wrong.

Past that, in the app: the Help button in this page's toolbar sends the tutor the section you are currently reading, together with the project's title, so you can ask "why does this not work" without pasting anything in. It knows which stage you are on.

Replay the same conversation. Every claim in this domain is about what survives, and a script you re-run is the only way to compare two runs honestly.
Print the context, not the answer. The answer looks fine long after the context stopped being. What the model was given is the observable here.
Ask what each structure loses. For any option in this domain, name the specific thing it drops — the amount, the date, the source, the fact that a search failed rather than found nothing. The one that keeps what the next decision needs is the answer.

Where the depth lives

This page is for building. The reading is elsewhere, and repeating it here would put the same claims in three places to drift apart:

The Domain 5 roadmap — what the domain is about, the order to learn it in, and where it catches people
6 flashcard decks, 53 cards — one card per published objective
44 Domain 5 questions — judgement under exam conditions, inside the six production scenarios

Sources

CCAR-F Exam Guide v1.0, section 6 — Domain 5's six task statements, which are this project's six stages.
CCAR-F Exam Guide v1.0, section 8 — Preparation Exercise 3 (human review routing and confidence scores) and Exercise 4 (error propagation and provenance through synthesis).
Anthropic Messages API reference — statelessness and the conversation history contract.
Claude Code documentation — /compact and session behaviour during extended exploration.

Domain 5 Project — make what you built survive contact with reality

This project is different from the other four, and it is worth saying why.

Domain 5 is 15% of the exam — the smallest block, and the one that punishes vagueness hardest.

The project at a glance

Domain: 5 · Context Management & Reliability (15%)
Stages: 6, one per task statement
Difficulty: Advanced
Total time: 190 minutes
You need: The systems from Domains 1 and 4, or time to stand up small versions
Based on: Exam guide Preparation Exercises 3 and 4

What you will be able to do

Keep transactional facts intact across a conversation long enough to be summarized, and say what summarization destroys first
Escalate on the three legitimate triggers and reject sentiment and self-reported confidence as proxies for complexity
Propagate an error with enough context for a coordinator to act, and handle locally what it need not hear about
Carry claim-source mappings and dates through synthesis, and distinguish a genuine conflict from a temporal difference
Calibrate a confidence threshold against a labelled set and defend automating a segment from its own accuracy rather than an aggregate

Before you start

Two stages need something you cannot fake quickly:

5.1 needs a conversation long enough to trigger summarization. Script it rather than typing it — twenty or thirty turns of realistic back-and-forth, saved so you can replay the same one after each change.
5.4 needs a codebase big enough that exploring it degrades the session. Something unfamiliar and genuinely large. A tidy example project will not drift, and the drift is the lesson.

bash

mkdir ccarf-domain-5 && cd ccarf-domain-5
python3 -m venv .venv && source .venv/bin/activate
pip install anthropic
export ANTHROPIC_API_KEY=your-key-here

Build it in this order

Six stages

5.1 · Facts that survive a long conversation · 30 min
BUILD — Run the support agent through a conversation long enough to trigger summarization, and watch an amount or a date go vague. Then lift the transactional facts — amounts, dates, order numbers, statuses — into a persistent case-facts block included in every prompt, OUTSIDE the summarized history. Trim the 40-field order lookup to the five fields that matter before it ever enters the context. WHY — Progressive summarization keeps the gist and loses the figures, and the figures are what the next decision turns on. A block outside the history is the structure that survives that loss. YOU SHOULD SEE — The same long conversation reaching turn thirty with the refund amount still exact, and a visibly shorter context once the tool output is trimmed at the door. TRAP — Trimming after the fact. Summarizing something you already paid to put in the context saves nothing — the trim belongs before the append. STUCK? — Print the full message list at turn five and turn twenty-five and diff them. The thing that went missing between the two is the thing that needed its own block.

Stage 5.1 in full

First, watch it happen

Run the support agent through a long conversation about a refund. Not a contrived one — a realistic thread where the customer mentions an amount early, wanders, and asks about it again much later.

Then ask, at turn thirty: what exactly was the refund amount?

Put the facts somewhere summarization cannot reach

The structure is a case-facts block: extracted transactional facts, maintained separately, included in every prompt, outside the history that gets compressed.

python

class CaseFacts:
    """Facts that must survive summarization, kept outside the history.

    The conversation gets summarized; this does not. Anything a later decision
    turns on lives here rather than in the messages.
    """

    def __init__(self):
        self.facts = {}

    def record(self, **pairs):
        self.facts.update({k: v for k, v in pairs.items() if v is not None})

    def as_block(self):
        if not self.facts:
            return ""
        lines = [f"- {key}: {value}" for key, value in sorted(self.facts.items())]
        return "CASE FACTS (authoritative, do not restate from memory):\n" + "\n".join(lines)


def build_messages(history, case_facts, user_message):
    # The block goes in EVERY request, ahead of the (possibly summarized)
    # history. It is small, so it costs little; it is authoritative, so the
    # model prefers it to a half-remembered figure.
    system_parts = [BASE_SYSTEM_PROMPT]
    if block := case_facts.as_block():
        system_parts.append(block)
    return {
        "system": "\n\n".join(system_parts),
        "messages": history + [{"role": "user", "content": user_message}],
    }

Record into it whenever a tool returns something transactional:

python

order = run_tool("lookup_order", {"order_id": "A-4417"})
case_facts.record(
    order_id="A-4417",
    order_total=order["total"],
    placed_at=order["placed_at"],
    order_status=order["status"],
)

Replay the same thirty-turn conversation. The amount should now be exact at turn thirty, because it was never in the part that got compressed.

Trim the tool output at the door

The second half of this stage costs nothing and pays every turn. An order lookup returns forty fields; a refund decision needs five.

python

RETURN_RELEVANT = ("order_id", "total", "placed_at", "status", "refundable_until")

def trim(order):
    # BEFORE the append, not after. Summarizing something you already paid to
    # put in the context saves nothing.
    return {key: order[key] for key in RETURN_RELEVANT if key in order}

Multiple choice · Medium

A support agent handles long conversations. By turn thirty it gives vague answers about amounts and dates that were precise earlier in the thread. What structure fixes this?

Prove stage 5.1 before moving on

Replay the identical conversation before and after. If you changed the script as well, you measured nothing.
Print the full message list at turn five and turn twenty-five and diff them. What went missing is what needed its own block.
Count the context tokens before and after trimming. If the number did not move, you are trimming after the append rather than before it.

When you get stuck

Every stage above ends with a STUCK? line — the first thing worth checking, which is usually the thing actually wrong.

Replay the same conversation. Every claim in this domain is about what survives, and a script you re-run is the only way to compare two runs honestly.
Print the context, not the answer. The answer looks fine long after the context stopped being. What the model was given is the observable here.
Ask what each structure loses. For any option in this domain, name the specific thing it drops — the amount, the date, the source, the fact that a search failed rather than found nothing. The one that keeps what the next decision needs is the answer.

Where the depth lives

This page is for building. The reading is elsewhere, and repeating it here would put the same claims in three places to drift apart:

The Domain 5 roadmap — what the domain is about, the order to learn it in, and where it catches people
6 flashcard decks, 53 cards — one card per published objective
44 Domain 5 questions — judgement under exam conditions, inside the six production scenarios

Sources

CCAR-F Exam Guide v1.0, section 6 — Domain 5's six task statements, which are this project's six stages.
CCAR-F Exam Guide v1.0, section 8 — Preparation Exercise 3 (human review routing and confidence scores) and Exercise 4 (error propagation and provenance through synthesis).
Anthropic Messages API reference — statelessness and the conversation history contract.
Claude Code documentation — /compact and session behaviour during extended exploration.

Domain 5 Project — make what you built survive contact with reality

Domain 5 Project — make what you built survive contact with reality

The project at a glance

What you will be able to do

Before you start

Build it in this order

Six stages

5.1 · Facts that survive a long conversation · 30 min

Stage 5.1 in full

First, watch it happen

Put the facts somewhere summarization cannot reach

Trim the tool output at the door

Prove stage 5.1 before moving on

When you get stuck

Where the depth lives

Sources

Domain 5 Project — make what you built survive contact with reality

Domain 5 Project — make what you built survive contact with reality

The project at a glance

What you will be able to do

Before you start

Build it in this order

Six stages

5.1 · Facts that survive a long conversation · 30 min

Stage 5.1 in full

First, watch it happen

Put the facts somewhere summarization cannot reach

Trim the tool output at the door

Prove stage 5.1 before moving on

When you get stuck

Where the depth lives

Sources