Recommend a Solution to Manage Secrets, Certificates, and Keys — Lesson

Every Azure workload eventually needs three things it cannot hard-code: a secret (a database password, an API key), a key (a cryptographic key used for encryption or signing), and a certificate (a signed public/private key pair with a lifetime). Storing any of these in source control, App Service configuration blades, or a developer's laptop is a well-trodden path to a breach headline. Azure's answer is Azure Key Vault — a managed, audited, policy-governed service that treats each of those three objects as first-class citizens with their own identity, version history, and access controls.

This lesson walks through the design choices an AZ-305 architect must make when recommending a Key Vault solution: which tier (Standard, Premium, or Managed HSM), which access model (classic access policies or Azure RBAC), which rotation strategy, and how to integrate consumer workloads using managed identities rather than embedded credentials. By the end you will be able to read a scenario — "Contoso Bank stores 500 signing keys and must demonstrate FIPS 140-2 Level 3 attestation" — and pick the right SKU, access model, and rotation pattern on first read.

Why This Matters

Credential leakage is the most common breach initiator in public-cloud incidents, and the overwhelming majority of leaked credentials come from application code, CI logs, or developer machines — not from compromised vaults. Designing a secrets solution correctly is therefore less about vault hardening and more about making the vault the path of least resistance for every team that needs to produce or consume a secret. If pulling a secret at runtime is harder than pasting one into an environment variable, developers will paste.

On the AZ-305 exam, secrets management surfaces wherever another service needs credentials: App Service pulling a connection string, AKS pulling a registry password, a Function signing a payload, an Event Grid webhook verifying a cert. A confident architect picks the right tier, the right access model, and the right rotation strategy without making the downstream workload rewrite its consumption pattern. Get this right and the rest of the security posture — identity, policy, monitoring — composes cleanly on top.

Prerequisites

Before this lesson, make sure you can answer the self-checks next to each topic. If not, pause and revisit the referenced node.

• Microsoft Entra ID and managed identities (Unit 1, Topic 1). Self-check: can you explain the difference between a system-assigned and user-assigned managed identity, and name a scenario where each is preferred?
• Azure RBAC fundamentals (Unit 1, Topic 2, LO-3). Self-check: what is the scope precedence order when conflicting role assignments exist at management-group, subscription, resource-group, and resource scopes?
• Resource Manager & Bicep basics. Self-check: can you deploy a resource with resource x 'Microsoft.KeyVault/vaults@2023-07-01' = { … } syntax and wire an output into a dependent resource?
• Public-key cryptography basics. Self-check: can you describe what is stored in a PEM-encoded certificate versus a PFX, and what the Private Key Usage Period extension means?
• TLS handshake lifecycle. Self-check: at which step in the handshake is the server certificate presented and by whom is it validated?

Learning Objectives

After completing this lesson you will be able to:

1. Analyse a workload's credential and key-material requirements and classify each artefact as a secret, key, or certificate.
2. Evaluate Key Vault tiers (Standard, Premium, and Managed HSM) against compliance, cost, and cryptographic-assurance requirements.
3. Design the access model — classic access policies versus Azure RBAC — for a new vault, including least-privilege role selection.
4. Recommend a rotation strategy for each artefact type using built-in policy, Event Grid notifications, or Automation runbooks.
5. Integrate Key Vault with consumer workloads via managed identity, CSI driver, or App Service references — without embedding any credential.
6. Design soft-delete, purge-protection, and backup strategies that satisfy common regulatory and disaster-recovery requirements.

Building Blocks

Before the deep dive, here are the foundational terms. Each is presented as Analogy → Formal definition → Why it matters.

Key Vault — Analogy: a smart safe with a logbook. Anyone with the right authorisation can ask the safe to produce an item, but the safe remembers every request. Definition: a managed Azure service that stores secrets, keys, and certificates behind an Azure Resource Manager resource with an authentication plane (Entra ID) and a data plane (REST API). Why it matters: centralising these three artefacts lets you apply one audit story, one rotation story, and one access model across every workload.

Secret — Analogy: a password in a lockbox. Definition: an opaque string (up to 25 KB) identified by a name and retrieved by authorised callers. Why it matters: most application credentials — SQL passwords, SAS tokens, third-party API keys — are modelled as secrets.

Key — Analogy: a private signet ring kept behind glass; the vault lets you stamp things but never hands the ring over. Definition: a cryptographic key (RSA or EC) where the private material never leaves the vault; callers invoke encrypt, decrypt, sign, verify, or wrapKey. Why it matters: this model powers customer-managed keys (CMK) for TDE, Storage, Cosmos DB, and bring-your-own-key (BYOK) for SaaS.

Certificate — Analogy: a driver's licence that expires. Definition: a chained X.509 object combining a key pair, a CA-signed public cert, and automatic lifecycle rules. Why it matters: the vault can auto-renew via a partnered CA (DigiCert, GlobalSign) and push the new cert into consumers, preventing outages from expired TLS.

HSM (Hardware Security Module) — Analogy: a tamper-proof vault inside the safe, certified by an external auditor. Definition: dedicated cryptographic hardware — FIPS 140-2 Level 2 (Premium) or Level 3 (Managed HSM) — where key material is generated and sealed. Why it matters: regulated workloads (PCI-DSS, FedRAMP High, EU financial services) often mandate HSM-backed keys.

Managed Identity — Analogy: a passport the workload carries that cannot be photocopied. Definition: an Entra ID identity automatically provisioned for an Azure resource; used to authenticate to Key Vault without storing a credential. Why it matters: the pattern that eliminates the "bootstrapping credential" problem — the app never sees a secret it could leak.

Access Policy — Definition: the classic Key Vault authorisation model using per-identity permission lists (get, list, set, etc.) on each vault. Why it matters: still the default on many older templates; you must know when to keep it and when to migrate to RBAC.

Azure RBAC (data plane) — Definition: modern authorisation using built-in roles such as Key Vault Secrets User, Key Vault Crypto Officer, and Key Vault Administrator, assigned at vault, resource-group, or subscription scope. Why it matters: unifies auth with the rest of Azure, supports Privileged Identity Management (PIM) and management-group scope.

Soft-delete — Definition: a vault-level and object-level retention feature where deleted objects enter a recoverable state for a configurable retention window (7–90 days; default 90). Why it matters: protects against accidental and malicious deletion. Enabled by default and cannot be disabled on new vaults.

Purge Protection — Definition: a one-way flag that prevents permanent deletion even by an administrator until the soft-delete retention window elapses. Why it matters: required by several compliance baselines and by customer-managed-key scenarios for Storage and SQL.

Deep Dive

Secrets, keys, and certificates — three models, one vault

Key Vault's three object types look similar from the portal but are fundamentally different artefacts with different consumer patterns, audit footprints, and rotation mechanics. Getting the distinction right is the first architectural decision.

A secret is an opaque, retrievable string. When an App Service asks for a secret it receives the plaintext value — the vault's job is protecting the path to that value, not the value itself. Rotation means writing a new version of the secret and plumbing consumers to pick up that version (via Event Grid, restart hook, or @Microsoft.KeyVault reference).

A key never leaves the vault. The consumer sends data to the vault and asks the vault to sign, verify, encrypt, or wrap using a named key. This is the model underpinning customer-managed keys for Azure Storage, Azure SQL TDE, Cosmos DB, and Azure Disk Encryption. Because the private material never materialises in application memory, a compromised VM cannot exfiltrate the key — it can only use it for the duration of the breach.

A certificate is a composite object: a private key (treated internally as a Key Vault key), a public certificate, and a policy describing lifetime, renewal, and the issuing authority. The vault can be configured with a trusted CA provider (DigiCert or GlobalSign natively) to auto-renew certificates before expiry, or it can manage self-signed certs where the business need is limited to internal trust anchors.

[!TIP] When a scenario says "store the SSL certificate", it is almost always a Key Vault certificate object (not a secret). Using the certificate object type gives you auto-rotation, chain assembly, and PFX export for services like App Service, Application Gateway, and API Management.

Tier selection: Standard, Premium, and Managed HSM

There are effectively three tiers to choose from, and the exam regularly tests the boundary between them. The decision rests on three axes: cryptographic assurance (FIPS level), single-tenancy, and price.

The decision tree below captures how to navigate the choice in practice.

Figure 1 — Tier-selection decision tree. The first branch separates secrets/certificate workloads from key-only workloads; regulated key-only workloads drive the Managed HSM decision.

[!WARNING] Managed HSM does not store secrets or certificates — only keys. If a regulated workload needs BOTH FIPS Level 3 keys AND secrets/certs, you deploy a pair: a Managed HSM for keys plus a Premium vault for secrets/certs.

Access control: access policies vs RBAC

Key Vault offers two authorisation models for the data plane. Microsoft recommends RBAC for all new deployments, but brownfield estates frequently still run on access policies, and the exam tests whether you can pick the right model and migrate cleanly.

Built-in RBAC roles the architect should know cold:

[!IMPORTANT] When migrating from access policies to RBAC, flip the vault's enableRbacAuthorization flag only after the equivalent role assignments are in place — otherwise the data plane is briefly unauthorised for every consumer and applications start returning 403 Forbidden.

Below is a minimal Bicep template that provisions a Premium vault in RBAC mode with purge protection enabled.

Rotation and lifecycle

Rotation is the hardest part of secrets design, and the piece most often weak in real customer environments. Key Vault offers three distinct mechanisms depending on artefact type.

For secrets, Key Vault does not rotate for you — the originating system (the database, the SaaS provider) generates the new credential and writes it back to the vault. Patterns:

1. Event-driven pull: Event Grid fires Microsoft.KeyVault.SecretNearExpiry 30 days before expiry. A Function App rotates the upstream credential and calls az keyvault secret set with the new value.
2. Scheduled push: an Automation runbook or CI pipeline runs on a cadence (monthly) and writes a fresh version.
3. Direct reference: consumers use @Microsoft.KeyVault(SecretUri=https://…) references in App Service / Functions; Azure refreshes the cached value roughly every 24 hours.

For keys, the vault itself can rotate on policy. This is the preferred pattern for CMK because the vault creates the new key version and the consuming service (Storage, SQL) discovers it on its next key fetch.

For certificates, the vault handles renewal end-to-end when you configure an integrated CA issuer. The policy specifies the percentage of lifetime at which renewal occurs (typically 80%) and the vault orchestrates the re-issuance, downloads the new cert, and stores it as a new version — consumers such as App Service pick up the new cert at the next poll without any deploy.

[!NOTE] Auto-rotation for keys is GA on Standard and Premium, but some downstream services (Azure SQL TDE with customer key at database level) require explicit notification of the new key version. Check the consumer service's CMK guidance before enabling policy-driven rotation.

Integrating applications via Managed Identity

The architectural ideal: the workload holds no secret that lets it access the vault. It authenticates using its managed identity, which Entra ID vouches for, and Key Vault authorises based on an RBAC role assignment. There are three consumption patterns.

Figure 2 — Managed-identity-based secret retrieval. Note the complete absence of any persistent credential on the application side.

Worked Examples

Example 1 — Easy: storing a database connection string for a web app

Problem: Contoso has an App Service running a .NET API that connects to an Azure SQL Database. Today the connection string lives in App Service configuration. Compliance flagged this as a finding; the architect is asked to recommend a Key Vault-based design with zero code change.

Solution:

1. Create a Standard tier vault (Premium is unnecessary — this is an ordinary secret, not a cryptographic key).
2. Enable RBAC authorisation; leave soft-delete and purge protection on (defaults).
3. Enable the App Service's system-assigned managed identity.
4. Grant the identity the Key Vault Secrets User role on the vault.
5. Write the connection string as a secret named sql-conn.
6. Replace the App Service setting SqlConnection value with @Microsoft.KeyVault(SecretUri=https://contoso-kv.vault.azure.net/secrets/sql-conn/).

[!NOTE] Key insight: the developer's code still reads Configuration["SqlConnection"] — the @Microsoft.KeyVault reference is opaque to the application. This is the pattern compliance teams want because it has the lowest adoption friction.

Example 2 — Medium: customer-managed keys for storage in a regulated bank

Problem: Contoso Bank must store all PII under a customer-managed key. Regulation requires keys to be protected by a FIPS 140-2 Level 2 HSM at minimum, with quarterly rotation auditable via the Azure Activity Log. The solution must span 3 regions for DR.

Solution:

1. Deploy a Premium tier vault per region with purge protection mandatory (Azure Storage CMK requires it).
2. Generate an RSA 3072-bit HSM-backed key in each vault; enable a rotation policy with timeAfterCreate: P90D (quarterly) and a 7-day notification trigger.
3. Configure the Azure Storage account with CMK referencing the vault; use key-version autodiscovery so rotation propagates automatically.
4. Enable vault diagnostic settings to Log Analytics; build a KQL alert on KeyRotate events.
5. For DR: use a Geo-Redundant Storage (GRS) account with its own CMK in the secondary region; replicate the primary key's wrapped version using BYOK export/import into the secondary vault.

[!NOTE] Key insight: when CMK spans regions, each region gets its own vault — keys are not geo-replicated. The replication discipline is therefore either independent keys (simpler, different key per region) or BYOK-synchronised keys (higher overhead, single key material). AZ-305 prefers independent keys unless the workload specifically requires cross-region key identity.

Example 3 — Hard: rotating TLS certificates across 50 Application Gateways

Problem: Contoso Global operates a hub-spoke topology with 50 Application Gateway instances, each terminating TLS on *.contoso.com. Today the wildcard cert is manually uploaded on each gateway quarterly by an operations team. The architect is asked to centralise.

Solution:

1. Move the wildcard cert to a Standard vault in the hub subscription; configure an integrated DigiCert issuer with auto-renewal at 80% lifetime.
2. Enable managed identity on each Application Gateway and assign Key Vault Certificate User (read) role on the vault.
3. Configure each gateway's HTTPS listener with a Key Vault reference (not an uploaded PFX). The gateway polls the vault every ~4 hours and rolls to the new version automatically.
4. Wire an Event Grid subscription on Microsoft.KeyVault.CertificateNewVersionCreated to a Logic App that posts a Teams message — a human-in-the-loop confirmation that rotation occurred.
5. For blast-radius control, use separate vaults per environment (prod, non-prod) and separate CA issuers to avoid a shared failure mode.

[!NOTE] Key insight: referencing a Key Vault certificate on Application Gateway moves the cert's rotation from the gateway's control plane to the vault's. The gateway's configuration no longer changes when the cert rotates — which means no ARM deployment, no change-management ticket, and no outage window.

Visual Explanations

Visual 1 — Decision tree for tier and access model

Caption: The tier is driven by compliance; the access model is always RBAC for greenfield. Legacy vaults can keep access policies during migration windows but should migrate off before major changes.

Visual 2 — Hub-spoke secrets architecture (TikZ)

Caption: A regulated hub-spoke pattern. The Managed HSM serves key-only workloads (CMK for Storage/SQL) while a Premium vault holds TLS certs and app secrets. Spokes reach both via private endpoint in the hub VNet.

Visual 3 — Feature-parity comparison

Caption: The table highlights that Managed HSM is narrower than it first looks — keys only, no secrets or certs. Architectures requiring both FIPS Level 3 keys and secrets deploy a pair of resources.

Common Mistakes

❌ Myth: "Enabling RBAC on a vault breaks existing applications." ✅ Reality: Applications see no difference — they still call the same data-plane REST endpoints. Only the authorisation layer changes. What breaks is unauthenticated configuration: any code path that still relies on the vault's access-policy list without a matching RBAC role assignment receives 403 Forbidden. Why it's tricky: The failure mode is silent until an app tries its first secret fetch after the migration. Always roll RBAC out with assignments staged in advance and flip enableRbacAuthorization as the last step.

❌ Myth: "Purge protection is optional; I can turn it off if I delete the vault by mistake." ✅ Reality: Purge protection is a one-way gate. Once enabled, neither you nor the subscription owner can bypass it — the soft-delete retention must elapse. Several services (CMK for Azure Storage, Azure SQL) refuse to bind to a vault without purge protection, so toggling it off is frequently not even an option. Why it's tricky: Developers experimenting in a dev vault enable purge protection to match prod, then cannot redeploy for 90 days. Keep a separate, non-protected sandbox vault for experiments.

❌ Myth: "Secrets stored in Key Vault are automatically rotated." ✅ Reality: Only keys and certificates support built-in rotation. Secrets require the originating system to generate a new value and write it back — there is no way for Key Vault to rotate a SQL password because the vault does not talk to SQL. Rotation automation is a consumer-side pattern (Function, Automation, CI job). Why it's tricky: The portal's rotation-policy UI only shows up on keys, but customers generalise from misleading third-party blog posts. Always ask "who knows how to generate the new value?" — that is who runs the rotation.

❌ Myth: "Managed HSM is just Premium with a bigger price tag." ✅ Reality: Managed HSM is a different service with its own URI, its own SKU family, and a local RBAC model (managed inside the HSM's security domain, not in Azure RBAC). It also excludes secrets and certificates entirely and is charged per pool rather than per operation. Why it's tricky: Some architects answer "Premium" for FIPS Level 3 requirements and lose a point; others answer "Managed HSM" for a secrets workload and lose a point. Match the tier to the artefact type first, compliance second.

Practice Exercises

Exercise 1 🟢 Easy

Contoso has an App Service that needs to retrieve an API key for a third-party SMS provider. The key is provided quarterly by the provider in an email to the operations inbox. Which Key Vault design minimises human steps between email and app?

▶💡 Hint

Consider where the rotation trigger comes from. Is it the vault, the app, or the operations team?

▶✅ Solution

Use a Standard vault in RBAC mode. Expose the API key as a secret and reference it from App Service using @Microsoft.KeyVault(SecretUri=…). When operations receives the quarterly email they run a single CLI command (az keyvault secret set) to write a new version; App Service picks up the value within ~24 hours with no restart. No auto-rotation is possible here — the originating system is an email, not an API — so the design reduces human work to a single step.

Exercise 2 🟡 Medium

A workload requires customer-managed keys for Azure SQL TDE. Regulation requires the keys to be HSM-backed and rotated every 180 days. Pick the tier, the rotation policy, and the RBAC role you would grant to the SQL server's managed identity.

▶💡 Hint

TDE with CMK uses key wrap and unwrap operations, not sign. Pick the smallest-scope role that grants both.

▶✅ Solution

Tier: Premium (HSM Level 2 is sufficient unless the regulation explicitly demands Level 3). Rotation policy: timeAfterCreate: P180D with a 7-day notify trigger. RBAC: Key Vault Crypto Service Encryption User — the purpose-built role that grants wrapKey/unwrapKey/get without broader key-management rights. Assign this role to the SQL server's system-assigned managed identity at vault scope. Purge protection must be enabled — SQL TDE with CMK mandates it.

Exercise 3 🟡 Medium

You are migrating a 5-year-old vault from access policies to RBAC. The vault has 60 service-principal assignments and is consumed by 8 production workloads. What is the safest sequence?

▶💡 Hint

What happens to data-plane calls the instant you flip the authorisation flag?

▶✅ Solution

1. Audit: export the current policy assignments and map each to a built-in RBAC role (e.g., get/list on secrets → Key Vault Secrets User).
2. Create all equivalent RBAC role assignments while access policies are still active (access policies continue to authorise during the transition window).
3. Validate on a staging vault by setting enableRbacAuthorization: true and confirming every principal can still read what it needs.
4. Flip enableRbacAuthorization: true in a change window on the production vault; monitor 403 rates via diagnostic logs.
5. After a soak period, remove the legacy access policies.

Exercise 4 🔴 Hard

Design a secrets solution for a multi-tenant SaaS that hosts $1{,}000$ tenants, where each tenant's data is encrypted with a tenant-specific customer-managed key. The SaaS's own application code should not see any tenant's key material. Which tier(s), vault count, and access pattern?

▶💡 Hint

Consider blast radius, quotas, and whether one vault per tenant or one HSM pool hosting all keys is safer.

▶✅ Solution

Use Managed HSM (keys only, FIPS Level 3, pool-priced so the marginal cost of each additional key is near zero) with one key per tenant inside a small number of HSM pools (2–3 for regional spread). The SaaS's encryption service authenticates with its managed identity and receives the Key Vault Crypto User role on the pool — but the app only ever invokes wrapKey/unwrapKey with a named key, never retrieves private material. Tenant offboarding = delete the key; the encrypted data becomes permanently unreadable (crypto-shredding). This beats "one vault per tenant" because Standard/Premium vaults have a $1{,}024$-identity ceiling and operation-per-second limits per vault; $1{,}000$ tenants would strain those limits and multiply management overhead.

Exercise 5 🔴 Hard

An architect proposes storing PFX certificate bytes as Key Vault secrets because the API is simpler than certificate objects. Give three reasons to push back, and one situation where the architect might be right.

▶💡 Hint

Think about rotation, consumer integration, and audit. But also think about short-lived workloads with no renewal needs.

▶✅ Solution

Push back because: (1) No auto-renewal — certificate objects integrate with DigiCert/GlobalSign for automatic renewal at 80% lifetime; a secret-stored PFX must be rotated manually. (2) No consumer integration — App Service, Application Gateway, and API Management natively bind to Key Vault certificates, not arbitrary secrets. (3) Audit weakness — certificate-specific operations (GetCertificate, ListCertificates) give auditors clean filters; opaque secret retrieval loses that semantic layer. The architect might be right for short-lived self-signed certs used internally for service-to-service mTLS where renewal is handled by CI re-issuance and no Azure service needs to bind natively — here, secret semantics are enough and the simpler API reduces automation code.

Exercise 6 🟡 Medium

Your team needs just-in-time elevation to the Key Vault Administrator role for break-glass secret recovery. What Azure feature supports this, and on which access model?

▶💡 Hint

Think about the identity layer, not the vault layer.

▶✅ Solution

Microsoft Entra Privileged Identity Management (PIM) — available only on the RBAC access model. Configure an eligible assignment (not active) for the break-glass group on the Key Vault Administrator role at vault scope, with approval workflow and a time-bound activation (e.g., 1 hour). Legacy access policies cannot participate in PIM and are therefore an explicit blocker for many compliance baselines.

Exercise 7 🟢 Easy

Name three Azure services that consume Key Vault keys (not secrets) under the customer-managed-key pattern.

▶💡 Hint

Encryption at rest is the common denominator.

▶✅ Solution

Azure Storage (blob/file/queue/table CMK), Azure SQL Database / Managed Instance (TDE with CMK), Azure Cosmos DB (account-level CMK), Azure Disk Encryption (OS/data disks), Azure Backup (CMK for Recovery Services Vault) — any three.

Summary & Concept Map

Key takeaways:

• Three artefact types: secrets (retrievable strings), keys (operated-on in place), certificates (composite with renewal policy) — each has a different rotation mechanic.
• Three tiers: Standard (software), Premium (HSM L2, adds BYOK), Managed HSM (HSM L3, keys only). Pick based on cryptographic assurance, then match artefact support.
• Two access models: access policies (legacy, per-vault) and Azure RBAC (modern, supports PIM, management-group scope). All greenfield designs use RBAC; migrations stage role assignments before flipping the flag.
• Rotation: the vault rotates keys and certificates natively; secrets rotate via the originating system writing back.
• Managed identity everywhere: consumer workloads never hold a vault credential — the platform's identity layer does.
• Purge protection + soft-delete: soft-delete is mandatory; purge protection is strongly recommended and required by most CMK scenarios.
• Blast-radius pattern: for multi-tenant or cross-region workloads, prefer Managed HSM pools with per-tenant keys over vault-per-tenant sprawl.

Figure — Concept map linking the vault's artefacts, access model, tiers, and consumer integration. The sibling Managed HSM service is intentionally shown parallel to Key Vault because it is a distinct resource, not a vault tier.