Design and implement agent and runner infrastructure

Every job needs a machine. Choosing which machine is a cost, security and maintainability decision, and the exam tests the trade-off rather than the click-path.

The options

Microsoft-hosted agents exist only in Azure DevOps Services — they are not available in Azure DevOps Server. Any on-premises scenario is self-hosted by definition.

Why teams move to self-hosted

Three reasons recur, and they map directly to exam scenarios:

1. Software you must install that is not on the hosted image.
2. Network reachability — the build must reach a private endpoint, an on-premises database, or a licence server.
3. Speed through persistence. Machine-level caches and configuration survive between runs on a self-hosted agent. A Microsoft-hosted agent is a clean machine every time, which is excellent for reproducibility and terrible for incremental build caches.

That third point is the trade-off in one line: hosted buys you cleanliness, self-hosted buys you warm caches. You cannot have both on the same agent.

Billing models differ in kind, not just amount

Microsoft-hosted capacity is bought as parallel jobs — a concurrency slot, not minutes. Two jobs at once needs two parallels regardless of how long they run. GitHub-hosted agents for Azure Pipelines bill per minute of usage instead, which inverts the optimisation: with concurrency billing you care about how many jobs overlap; with per-minute billing you care about how long they take.

Maintainability

Self-hosted is not free after setup. You own agent version upgrades, OS patching, tool installs, disk hygiene between runs, and the security posture of a machine that holds pipeline credentials. VM Scale Set agents and Managed DevOps Pools exist to keep the elasticity while handing back most of that burden — which is why Microsoft now recommends Managed DevOps Pools over building your own autoscaling scale-set pool.

Primary sources

• https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents
• https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted

Agents and runners — quick notes

Traps

1. "Use Microsoft-hosted for the on-prem server" — impossible; hosted is Services-only.
2. "Hosted agents cache between runs" — false, every run is a clean machine.
3. Concurrency billing ≠ per-minute billing; the optimisation differs.

Configure alerts for events in GitHub Actions and Azure Pipelines

Routes

Either can also call out from within the run — a step that posts on failure — which is the flexible option when the condition is more specific than "the run failed".

What deserves an alert

The same discipline as everywhere else in this unit: alert on things a human must act on.

The condition that gets forgotten

A step that reports failure must itself run on failure. Every step carries an implicit succeeded(), so a notification step placed after a failing step is skipped — the same defect as publishing test results without condition: succeededOrFailed().

A failure notification that only fires on success is worse than none, because the silence is read as good news.

Route to an owner

An alert to a shared inbox nobody owns is not an alert. Route to a team with the ability to act, and make ownership explicit.

Primary sources

• https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-overview
• https://learn.microsoft.com/en-us/azure/azure-monitor/overview

Pipeline alerting — quick notes

The classic bug: a notification step without condition: failed() (or succeededOrFailed()) is skipped when the thing it reports on fails. Silence then reads as good news.

Analyze metrics by using collected telemetry

Usage tells you what to work on

Usage data is what makes a hypothesis testable — it is the loop that closes feature flags and A/B testing back into a decision. Without it, "we shipped it" is the end of the story rather than the middle.

Performance analysis

Work from the user inward:

1. Which operations are slow? Request duration by name, at p95 and p99.
2. What are they waiting on? Dependency telemetry — database, HTTP, queue.
3. Where inside the code? Traces and profiling.

Skipping to step 3 is the common mistake: it produces a detailed answer about something that was never the problem.

Aggregates hide people

An average conceals the tail, and a healthy overall error rate conceals a single customer failing every request. Segment — by operation, region, client version, tenant — because one broken tenant inside a 0.1% global error rate is invisible in the aggregate and total for that customer.

Correlate with deployments

Release annotations put deployments on the chart. Most performance regressions have a deployment immediately before them, and seeing the two together is the fastest available diagnosis.

Primary sources

• https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview

Analyzing telemetry — quick notes

Performance, in order: which operations are slow (p95/p99) → what are they waiting on (dependencies) → where in the code (traces).

• Skipping to code profiling answers the wrong question in detail.
• Aggregates hide people — segment by operation, region, version, tenant.
• A healthy global error rate can hide one customer failing every request.
• Correlate with release annotations — regressions usually follow a deployment.

Recommend appropriate access levels

Access level and permissions are different axes. Access level decides which features a user may use at all — it is a licensing question. Permissions decide what they may do with the things they can see.

Azure DevOps access levels

Stakeholder is the answer whenever a scenario describes someone who needs visibility and sign-off but not code: a product owner tracking a backlog, a manager approving a release. It costs nothing, so assigning Basic to those people is pure waste — and grants code access they never needed.

GitHub

Outside collaborator is the parallel concept: repository-scoped access without organisation membership, correct for contractors and partners.

Reviewing regularly

Access levels drift upward. People are granted Basic during a project and keep it for years. Periodic review is part of the design, not an afterthought — and it is cheapest when access came from group membership rather than individual grants.

Primary sources

• https://learn.microsoft.com/en-us/azure/devops/organizations/security/access-levels

Access levels — quick notes

• Access level = which features (licensing). Permissions = what you may do.
• Product owner / approver who never touches code → Stakeholder.
• GitHub parallel for contractors → outside collaborator.

Automate container scanning

A container image bundles your application and an operating system userland. Both carry vulnerabilities, and they age differently: your code changes when you change it, the base image accumulates CVEs on someone else's schedule.

Two things to scan

Where scanning goes in the pipeline

Scan before push so a vulnerable image never enters the registry, and scan in the registry continuously so an image that was clean at build time raises an alert when a new CVE lands. Both are needed for the same reason a quiet repository still needs Dependabot alerts: the code stopped changing, the threat landscape did not.

CodeQL in a container

Running CodeQL analysis inside a container is a documented objective, and it needs advanced setup — the generated workflow file is where you specify the container and the build. Default setup cannot express it.

The typical reason is a compiled language whose build environment lives in the container: CodeQL must observe the real build to analyse it, so the analysis has to run where that build runs.

Rebuilding is the fix

Most base-image findings are resolved by rebuilding on a patched base rather than by changing your code — which makes an automated periodic rebuild a security control, not merely hygiene.

Primary sources

• https://docs.github.com/en/code-security/code-scanning
• https://learn.microsoft.com/en-us/azure/container-registry/scan-images-defender

Container scanning — quick notes

• Scan before push → keep vulnerable images out of the registry.
• Scan in the registry continuously → catch CVEs disclosed after build.
• CodeQL in a container requires advanced setup — default setup cannot express it.
• Most base-image findings are fixed by rebuilding on a patched base.

Trap: scanning only at build time. The image ages even when your code does not.