Lesson204 words
Deployment resiliency
Design and implement a resiliency strategy for deployment
Resiliency here means the deployment survives its own failure — it detects trouble and reverses without a human reading logs at 3am.
Automatic reversal
yaml
strategy:
runOnce:
deploy:
steps: [ { script: ./deploy.sh } ]
postRouteTraffic:
steps: [ { script: ./smoke-and-metrics.sh } ]
on:
failure:
steps: [ { script: ./rollback.sh } ]
success:
steps: [ { script: ./cleanup.sh } ]The important pairing: postRouteTraffic is what makes on: failure meaningful. Health is measured while real traffic flows, so a failure detected there triggers rollback based on production behaviour rather than on whether a script exited zero.
Layers
| Layer | Mechanism |
|---|---|
| Detect | postRouteTraffic checks; Query Azure Monitor alerts |
| Reverse | on: failure steps; slot swap back; feature flag off |
| Contain | Canary and rolling — limit exposure before you detect |
| Serialise | Exclusive lock — stop a second deployment landing mid-incident |
Idempotency
Retries and partial failures mean a deployment step may run twice. Scripts must be safe to re-run — the resiliency plan is only as good as the least idempotent step in it.
Primary sources