Domain 4 Project — build an extraction pipeline, then the reviewer that judges it
Claude Certified Architect - Foundations (CCAR-F) › Domain 4: Prompt Engineering & Structured Output
Domain 4 Project — build an extraction pipeline, then the reviewer that judges it
Six stages that turn halfway through. You build a document extraction pipeline — schema, validation, retry, examples — and then you build the thing that judges output: a code reviewer with criteria precise enough to apply the same way twice, and an architecture that does not let the author mark its own work.
Domain 4 is 20% of the exam, and its sharpest line runs through the middle of this project: tool use eliminates syntax errors and leaves semantic ones untouched. Stage 1 gives you a response that validates perfectly. Stage 2 is where you discover it is wrong.
The advanced rating is honest — this is the longest of the five projects, and stages 4.4 and 4.6 both ask you to build something that checks something else.
The project at a glance
- 4 · Prompt Engineering & Structured Output (20%)
- 6, one per task statement
- Advanced
- 215 minutes
- Python 3.10+, the anthropic SDK, an API key, and a dozen messy documents
- Exam guide Preparation Exercise 3
What you will be able to do
- Design a JSON schema that makes fabrication unnecessary, and say which fields must be nullable and why
- Distinguish a validation failure a retry can fix from one it cannot, before spending the attempts
- Write few-shot examples aimed at measured failures, and explain why the reasoning matters more than the pairs
- Replace a confidence-based review filter with categorical criteria, and anchor severity to concrete examples
- Choose between the synchronous and batch APIs from the workflow's latency, and calculate the submission frequency an SLA needs
Before you start
The SDK setup from Domain 1, plus something to extract from:
mkdir ccarf-domain-4 && cd ccarf-domain-4
python3 -m venv .venv && source .venv/bin/activate
pip install anthropic
export ANTHROPIC_API_KEY=your-key-here
mkdir documentsThe documents matter more than the code. Put a dozen real ones in that directory — invoices, papers, reports, whatever you have — and make sure they are genuinely inconsistent. You specifically want:
- at least two that are missing a field the others have, because stage 4.3's whole lesson is what a required field does to those
- at least two with a different structure for the same information — inline citations against a bibliography, a table where another used prose — because those are what stage 4.2's examples are aimed at
- at least one whose numbers do not add up, because stage 4.4's semantic validation has nothing to catch otherwise
A clean, uniform corpus will make every stage of this project appear to work and teach you none of it.
Stage 4.5 submits 100 documents to the Batches API. Duplicating your dozen is fine — the stage is about custom_id correlation and failure handling, not about having 100 distinct sources.
Build it in this order
The pipeline is built first, then the reviewer that judges output, then batching — because you batch a pipeline that works, and 100 documents through a broken one is 100 wrong answers at a discount.
Six stages
4.3 · A schema that cannot be answered wrongly · 40 min
BUILD — An extraction tool whose input_schema IS your output shape: required fields where the source always has them, nullable ones where it may not, an enum with an unclear value, and an other-plus-detail pair for categories you did not anticipate. Run it over documents that are missing fields on purpose. WHY — tool_use with a JSON schema is the reliable way to get structured output, and schema design is where fabrication is prevented. A required field the source lacks is a guaranteed invented value — the model has no other way to satisfy you. YOU SHOULD SEE — Null coming back for genuinely absent fields rather than something plausible, and unclear appearing where the document really is ambiguous. TRAP — Marking everything required because it feels rigorous. It is the single most reliable way to make an extractor lie, and the exam asks about it directly. STUCK? — For each required field, find one real document that lacks it. If you can, the field is not required — it is optional and you were hoping.
Stage 4.3 in full
The schema is the output shape
With tool use, you do not ask for JSON and hope. You define a tool whose INPUT parameters are the shape you want, and read the result out of the tool_use block. The model fills the parameters; the arguments are your answer.
EXTRACT = {
"name": "record_invoice",
"description": "Record the fields extracted from one invoice document.",
"input_schema": {
"type": "object",
"properties": {
"invoice_number": {"type": "string"},
"issued_on": {"type": ["string", "null"], "description": "ISO 8601 date, or null if the document does not state one"},
"supplier": {"type": ["string", "null"]},
"stated_total": {"type": ["number", "null"]},
"currency": {
"type": "string",
"enum": ["GBP", "USD", "EUR", "other", "unclear"],
},
"currency_detail": {
"type": ["string", "null"],
"description": "The currency as written, when currency is other or unclear",
},
"line_items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"description": {"type": "string"},
"amount": {"type": "number"},
},
"required": ["description", "amount"],
},
},
},
"required": ["invoice_number", "currency", "line_items"],
},
}Four decisions in there, and each is examined:
Why each field is shaped that way
Two separate mechanisms, and this field uses both. Omitting it from required means it may be absent; allowing null in type means it may be present and empty. The exam guide calls both optional (nullable) — know that they are different, because omitting from required is the one that is safe everywhere.
An ambiguous document has somewhere honest to go instead of being forced into the nearest listed value.
An unanticipated category is RECORDED rather than lost. The enum stays closed; the information does not.
An invoice with no number is not an invoice. Everything else is a hope, and a hope in the required list is a fabrication instruction.
Run it, deliberately, over the broken ones
def extract(document_text):
response = client.messages.create(
model="claude-opus-5",
max_tokens=4096,
tools=[EXTRACT],
tool_choice={"type": "tool", "name": "record_invoice"},
messages=[{"role": "user", "content": document_text}],
)
for block in response.content:
if block.type == "tool_use":
return block.input
return NoneThe forced tool_choice is deliberate: with auto the model may reply in prose, and a pipeline that sometimes gets prose has to handle two shapes. Here the document type is known, so forcing the specific tool is right. Stage 4.5 revisits this — when the type is unknown, any is the setting, because the model must pick the schema.
Now run it over the two documents you chose for missing fields. null is the correct answer and you should see it. If you see a plausible date instead, the field is required and this is the lesson.
An extraction pipeline uses tool_use with a strict JSON schema. Every response validates. A reviewer nonetheless finds invoices where the line items do not add up to the recorded total. What does this show?
Prove stage 4.3 before moving on
- Run the missing-field documents and confirm you get null, not a plausible value.
- Feed it something in an unlisted currency and confirm you get other with the detail captured, rather than the nearest listed match.
- Keep every extraction. Stage 4.4 validates these exact outputs, and the ones that are wrong today are the fixtures you need.
When you get stuck
Every stage above ends with a STUCK? line — the first thing worth checking, which is usually the thing actually wrong.
Past that, in the app: the Help button in this page's toolbar sends the tutor the section you are currently reading, together with the project's title, so you can ask "why does this not work" without pasting anything in. It knows which stage you are on.
- Test on the awkward documents, not the clean ones. Every stage of this project appears to work on a uniform corpus and teaches nothing there.
- Keep the failures. Each stage's bad outputs are the next stage's fixtures — a pipeline you cannot re-run against yesterday's mistakes cannot be shown to have improved.
- Separate the two error classes out loud. Ask of every failure whether the shape is wrong or the meaning is wrong. The whole domain hangs off that distinction, and so do most of its questions.
Where the depth lives
This page is for building. The reading is elsewhere, and repeating it here would put the same claims in three places to drift apart:
- The Domain 4 roadmap — what the domain is about, the order to learn it in, and where it catches people
- 6 flashcard decks, 47 cards — one card per published objective
- 47 Domain 4 questions — judgement under exam conditions, inside the six production scenarios
Sources
- CCAR-F Exam Guide v1.0, section 6 — Domain 4's six task statements, which are this project's six stages.
- CCAR-F Exam Guide v1.0, section 8 — Preparation Exercise 3 (Build a Structured Data Extraction Pipeline), whose five steps supply four of these stages.
- Anthropic Messages API reference — tool_use with JSON schemas, the documented JSON Schema limitations, the three tool_choice settings, and the Message Batches API with custom_id correlation.