Recommend a Compute Solution for Batch Processing — Lesson

A pharmaceutical research team needs to simulate $200{,}000$ molecular configurations against a target protein. Each simulation runs $20{-}90$ minutes on a 4-vCPU box; the entire campaign needs to complete in a weekend. The lead engineer's first design is a VMSS of 50 on-demand VMs running scripts in parallel — accurate, expensive, and operationally heavy. The architect replaces it with Azure Batch using a low-priority pool that scales from zero to 400 nodes, runs the $200{,}000$ tasks, and scales back to zero on Sunday night. The campaign finishes Saturday afternoon. The bill is one-third of what the VMSS approach would have cost. This lesson is about reaching for batch services — Azure Batch, Synapse Spark pools, Azure Databricks jobs, CycleCloud for HPC — instead of building parallel processing yourself on VMs.

We will work through Azure's batch-processing story the way the AZ-305 exam expects you to: distinguishing Azure Batch (HPC-style pool/job/task model) from Synapse Spark and Databricks (data-analytics-shaped batch) and from Container Apps Jobs (container-shaped batch). Reference: the AZ-305 exam study guide, particularly Chapter 4 Skill 4.1 on batch processing and HPC.

Why This Matters

Batch processing is where compute economics get dramatic. A typical batch workload — Monte Carlo, video transcoding, ML training, genomics, financial risk — runs $10{-}1000{\times}$ the parallelism of a steady-state web service for $1{-}100{\times}$ less duration. Picking the right batch service moves the bill by an order of magnitude and the wall-clock time by another. Done well, batch is the most cost-effective place to spend compute. Done poorly (parallel scripts on a VMSS, an oversized AKS cluster, hand-rolled queueing on Functions), it is a perpetual source of operational toil.

The AZ-305 exam tests this LO because batch processing is also the area with the highest variance in candidate skill. Many architects know Functions and App Service cold but reach for VMs when given a "process $200{,}000$ files" problem. If you can match a workload — embarrassingly parallel HPC, Spark-shaped data analytics, ML training, periodic ETL, video transcoding — to the right batch service and configure pool / job / task structure (or Spark pool / cluster) correctly, you will pass this slice of the exam and design batch compute like a senior architect. Every "we have $X$ files to process" workshop and every "this overnight job is too slow" complaint touches this LO.

Prerequisites

Before working through this lesson, make sure you can answer each prompt below in one or two sentences.

• Embarrassingly parallel workloads. Can you describe what makes a workload "embarrassingly parallel"? — Self-check: do simulations of independent inputs qualify?
• HPC concepts. Are you familiar with MPI (Message Passing Interface) and what it implies for inter-node latency? — Self-check: is MPI part of every HPC workload?
• Apache Spark basics. Do you know what a DataFrame, a job, a stage, and a task are in Spark? — Self-check: which one is the unit Spark schedules to executors?
• GPU compute fundamentals. Can you describe when a workload genuinely benefits from a GPU? — Self-check: is a typical web API GPU-bound?
• Eviction-tolerance. Are you fluent with Spot VMs and low-priority Batch nodes — what happens on eviction? — Self-check: should production singletons run on Spot?

If any of these feels shaky, pause and review the batch and HPC modules in Unit 4 of the AZ-305 guide before continuing.

Learning Objectives

By the end of this lesson, you will be able to:

1. Analyse a batch workload's shape (embarrassingly parallel vs Spark-style vs MPI-coupled) and translate it to a batch service.
2. Evaluate trade-offs between Azure Batch, Synapse Spark pools, Azure Databricks jobs, Container Apps Jobs, and AKS Jobs for a given workload.
3. Design an Azure Batch pool structure (auto-scale formula, low-priority mix, application packages, container support) for a campaign.
4. Recommend a Spark host (Synapse Spark pool, Databricks, or HDInsight) based on team skillset, governance, and existing investments.
5. Recognise common anti-patterns — VM-based parallelism for HPC, AKS for a one-off batch run, low-priority pools for production singletons — and rewrite them.
6. Configure identity, networking, and storage integration so that batch workloads can read input data and write outputs at scale.

Building Blocks

Read this section as a glossary. Each term follows the same shape: an everyday analogy, a formal definition, then the reason it matters for the exam.

Batch processing — Running a finite, known workload to completion as a one-shot or scheduled job. Like a print queue: jobs go in, they run, you collect the output, the queue empties. Formally, a workload model with a defined start and end (not a long-running service). It matters because the design constraints — start fast, scale out heavily, scale back to zero — are very different from steady-state services.

Azure Batch — A managed service that runs HPC-style pool / job / task workloads. Like a queue-of-work plus a fleet-of-workers, both managed. Formally, three core resource types: pool (the worker fleet), job (a logical grouping of work), task (one unit of work). Bills per-VM-second of pool nodes plus a tiny per-task fee. It matters because Batch is the canonical answer for embarrassingly parallel work.

Pool — A managed VM fleet for Azure Batch. Formally, a logical resource that auto-scales (per a formula), runs application packages, mounts shared file systems, and executes tasks. Pools can mix Regular and Low-Priority nodes. It matters because pool design is most of the optimisation lever in Batch — node SKU, auto-scale formula, mix of priorities.

Task — One unit of work in Azure Batch. Formally, a command line (and optional container image, input data, environment) that runs on one pool node. Each task is independent; success/failure is per-task. It matters because the task model is what makes Batch suitable for embarrassingly parallel work — $200{,}000$ tasks scale identically to 200.

Low-priority node — A pool node sold at $60{-}90\%$ discount with eviction risk. Formally, a Batch pool node that Azure can evict on 30 seconds' notice when capacity is needed; Batch reschedules evicted tasks automatically. It matters because most batch workloads tolerate eviction (the task simply restarts on another node), so low-priority is usually correct.

Spot VM (Batch context) — The modern flavour of "discounted with eviction risk" capacity. Formally, Batch supports both lowpriority and spot node priorities; Spot has a configurable max price and rebill semantics. It matters because Spot is the recommended modern default for cost-sensitive batch — lowpriority is the legacy term, still supported.

Apache Spark pool — A managed Spark cluster you bring up for analytics workloads. Formally, in Azure: Azure Synapse Analytics Spark pools (Microsoft-managed, governance-integrated) and Azure Databricks workspaces (Databricks-managed, MLflow integrated). It matters because Spark workloads — joins, aggregations, ML feature engineering — are the canonical fit for these services rather than Azure Batch.

MPI (Message Passing Interface) — A protocol for inter-process communication used by tightly coupled HPC workloads. Formally, a standard that lets processes on multiple nodes exchange messages with near-network-line-rate latency. It matters because MPI workloads need low-latency networking (InfiniBand-class), which requires HPC-specific VM families (HB, HC, HX) and Proximity Placement Groups.

Azure CycleCloud — A management layer for HPC clusters on Azure VMs that brings open-source schedulers (Slurm, OpenPBS, LSF) to Azure compute. Formally, a service that provisions and auto-scales HPC clusters according to scheduler queues. It matters because HPC teams who run Slurm on-prem can lift their workflow to Azure with CycleCloud without rewriting to Azure Batch.

Container Apps Jobs — A serverless container batch primitive in Container Apps. Formally, a Microsoft.App/jobs resource that runs containers as one-shot or scheduled jobs, scaling from zero. It matters because it bridges container-packaged batch with serverless economics — and is often a simpler answer than Azure Batch when each task is a container.

Deep Dive

1. The batch-service decision — pick by workload shape, not by familiarity

The first cut for a batch workload is shape, not size.

[!TIP] Match the conceptual shape first. A team running Slurm jobs on-prem usually wants CycleCloud even though Azure Batch could technically handle the same compute — the workflow matters as much as the work.

2. Azure Batch deep dive — pool, job, task

Azure Batch is the canonical answer for embarrassingly parallel work. The three-tier resource model (pool / job / task) is what the exam tests most often.

A pool is sized and scaled by an auto-scale formula — a small expression that returns a target node count and is re-evaluated on a schedule (default every 15 min, configurable).

[!IMPORTANT] Always set $NodeDeallocationOption = taskcompletion on auto-scale formulas. Without it, scale-down can kill nodes mid-task, wasting compute and forcing reschedules. With it, nodes drain completed tasks before deallocation.

[!WARNING] Auto-scale formulas have a syntax all their own and are not Python or JavaScript. Test formulas with the Batch CLI's pool autoscale evaluate before applying — typos return cryptic errors at runtime.

3. Cost optimisation in Azure Batch — Spot, low-priority, and the priority mix

Spot / low-priority nodes give $60{-}90\%$ off pay-as-you-go capacity. The eviction model in Batch is forgiving: when a node is evicted, its in-progress task is rescheduled to another node automatically. This makes Spot the right default for almost every batch workload.

Mixing Dedicated and Spot in a single pool is supported and a common pattern: a small dedicated baseline guarantees forward progress; a large Spot tail absorbs the parallelism cheaply.

[!NOTE] taskSlotsPerNode lets one node run multiple small tasks concurrently — set to roughly the vCPU count for CPU-bound work. nodeFillType: 'Pack' packs tasks onto existing nodes before scaling out, improving Spot economics.

4. Spark workloads — Synapse vs Databricks vs Fabric

Spark-shaped workloads (large-table joins, aggregations, ML feature eng, streaming ETL) belong on Spark, not Azure Batch. Azure has three Spark hosts the exam considers:

Choosing between them is often driven by team familiarity, governance posture, and existing investments. The exam frequently tests this with a "team already uses MLflow" $\to$ Databricks or "data engineering team is building a Synapse pipeline" $\to$ Synapse Spark pools pattern.

5. HPC with CycleCloud — the Slurm bridge

For teams running tightly coupled MPI workloads — fluid dynamics, finite element, weather, genomics — Azure Batch is one option, but CycleCloud is often a better fit. It exposes a familiar scheduler (Slurm, OpenPBS, LSF) and auto-scales VMs into queues as jobs are submitted.

[!TIP] HPC-bound VMs (HB, HC, HX series) include InfiniBand networking that delivers $<2$ microsecond inter-node latency. Place them in a Proximity Placement Group to keep the latency budget honest.

6. Storage and identity for batch

Batch workloads read large inputs and write large outputs. Pair every pool with:

Worked Examples

Easy — embarrassingly parallel image processing

Problem. A team needs to process $20{,}000$ JPEG images: resize, watermark, write back to storage. Each image takes $\sim 5$ seconds. The campaign must complete in 1 hour. Recommend a batch service.

Solution. Azure Batch with a Spot pool of $\sim 30$ Standard_D4s_v5 nodes (4 vCPU each, 4 task slots per node $= 120$ concurrent tasks). $20{,}000$ tasks at 5 s each $= 100{,}000$ task-seconds $\sim 833$ minutes of compute, spread across 120 concurrent slots $= \sim 7$ minutes wall-clock plus pool start-up. Total cost: tens of dollars. Storing inputs in Blob with read access via managed identity.

[!NOTE] For a one-off campaign, Container Apps Jobs is also a viable answer if each task is packaged as a container — slightly less scale-out headroom but no Batch account to manage.

Medium — Spark-shaped data engineering

Problem. A data team needs to join 5 TB of customer events with 200 GB of dimension tables and write the result to a Delta Lake. The team uses MLflow for downstream model training. Recommend a host.

Solution. Azure Databricks with an autoscaling cluster. Databricks gives first-class MLflow and Delta Lake; the cluster autoscales from 0 to $\sim 30$ Spot-priced executors for the join, then scales back. Photon engine accelerates Spark joins by $2{-}3\times$ on this kind of workload. If the team had been on Synapse instead, Synapse Spark pools would have been the answer — both are valid Spark hosts; team familiarity drives the choice.

Hard — HPC simulation with MPI

Problem. A research team runs a fluid-dynamics simulation that spreads across 40 nodes communicating via MPI. Each simulation takes $\sim 2$ hours. The team currently submits jobs via Slurm on-prem and wants to migrate to Azure. Recommend a topology.

Solution. Use Azure CycleCloud with a Slurm cluster, deploying Standard_HB120rs_v3 (120 vCPU, AMD EPYC, 200 Gb InfiniBand) nodes in a Proximity Placement Group for low MPI latency. Pool size $0{-}40$ nodes, autoscaled by Slurm queue depth. Use Azure NetApp Files for the shared filesystem at $/shared. The team continues to use sbatch` to submit jobs — operational continuity is preserved while migrating to cloud capacity.

[!IMPORTANT] HPC VM sizes (HB, HC, HX) are designed for MPI workloads. Standard D-series VMs lack InfiniBand and will not meet MPI latency budgets — a workload that performs fine on-prem will run $2{-}5\times$ slower on D-series VMs.

Visual Explanations

Figure 1 — Batch-service decision flow

The shape determines the service. Team familiarity is the tie-breaker between Synapse and Databricks. The exam tests this decision regularly with workload descriptions.

Figure 2 — Azure Batch pool / job / task topology

The pool is the worker fleet; the job is the logical container of work; each task is one unit. Tasks are scheduled across pool nodes, with auto-scale formulas growing or shrinking the pool with queue depth.

Figure 3 — Batch vs Spark vs ACA Jobs quick chooser

Common Mistakes

❌ Myth: "VMSS plus parallel scripts is fine for batch." ✅ Reality: Rolling your own batch on a VMSS reproduces what Azure Batch does — task scheduling, eviction handling, retries, output capture — at much higher operational cost. Reach for Azure Batch whenever the workload is "$N$ independent tasks". Why it's tricky: Teams familiar with VMs default to VM-based parallelism. The hidden cost is operational toil, not pricing per hour.

❌ Myth: "Spark workloads belong on Azure Batch because they are batch." ✅ Reality: Spark workloads have a fundamentally different execution model — DAG of stages, shuffles, broadcast joins — that needs a Spark host. Azure Batch does not understand Spark; Synapse Spark and Databricks do. Why it's tricky: The word "batch" appears in both contexts. The architecture is the difference.

❌ Myth: "Spot/low-priority nodes are too risky for batch — production must be dedicated." ✅ Reality: Azure Batch re-schedules evicted tasks automatically. For workloads with task-level retry tolerance (i.e., almost all batch), Spot is the right default. Reserve dedicated nodes for time-critical campaigns. Why it's tricky: "Production" conflates with "guaranteed" — but batch production is task-level granular, not service-level.

❌ Myth: "An MPI workload on Azure D-series will run fine." ✅ Reality: MPI workloads need low-latency RDMA-class interconnect. D-series VMs lack InfiniBand and run MPI workloads $2{-}5\times$ slower than HB/HC/HX class. Pair HPC VMs with a Proximity Placement Group. Why it's tricky: "It works" $\ne$ "it works well". The wall-clock degradation is silent until you measure.

Practice Exercises

🟢 Exercise 1. A team has $50{,}000$ PDF documents to OCR. Each PDF takes $\sim 30$ seconds. They want it done overnight. Cost matters. Recommend a service.

▶💡 Hint

Embarrassingly parallel, eviction-tolerant.

▶✅ Solution

Azure Batch with a Spot pool. $50{,}000$ tasks at $30$ s each = $\sim 25$ hours of compute. With $100$ Spot nodes of $4$ vCPU and $4$ task slots ($400$ concurrent tasks), the wall-clock time is $\sim 2$ hours — easy overnight. Spot pricing saves $\sim 70\%$ off PAYG. Inputs in Blob, OCR output back to Blob, identity via system-assigned managed identity on the Batch account.

🟡 Exercise 2. A data team needs to compute daily aggregates over a $10$ TB Delta Lake. Notebooks are in Python and use MLflow. Recommend a host.

▶💡 Hint

MLflow and Delta Lake are first-class on which platform?

▶✅ Solution

Azure Databricks. MLflow and Delta Lake are Databricks-native. An autoscaling cluster ($2{-}30$ workers, Photon engine on) handles the aggregations efficiently. Schedule the job via Databricks Jobs (or Azure Data Factory if the orchestration is broader). Cost is paid per DBU-hour; Spot-priced workers reduce DBU cost by $40{-}50\%$ for batch jobs.

🟡 Exercise 3. A research team wants to migrate $200$ Slurm-based MPI jobs to Azure. They submit jobs via sbatch and don't want to rewrite. Recommend an approach.

▶💡 Hint

Slurm is a scheduler. CycleCloud knows it.

▶✅ Solution

Use Azure CycleCloud to deploy a Slurm cluster with HPC VMs (HB120rs_v3 or similar). Researchers continue to submit jobs via sbatch; CycleCloud autoscales VM nodes into the cluster as jobs arrive and scales them out when queues drain. Place compute nodes in a Proximity Placement Group for MPI latency. Shared filesystem on Azure NetApp Files.

🔴 Exercise 4. An Azure Batch pool runs $50{,}000$-task campaigns nightly. Auto-scale formula is set to a fixed $\text{nodes} = 200$. Cost is high. Identify two changes.

▶💡 Hint

Static node counts waste compute outside the campaign window.

▶✅ Solution

(1) Replace the fixed formula with one that scales based on $PendingTasks so the pool scales to zero when the campaign isn't running. (2) Switch most or all nodes from Dedicated to Spot — task-level retry covers eviction risk. Combined, these typically cut nightly batch cost by $80\%$ or more.

🔴 Exercise 5. A team's Container Apps Job runs $4$ hours nightly. They are considering moving to Azure Batch. When does the move pay off?

▶💡 Hint

Single long job vs. many tasks.

▶✅ Solution

For a single long-running container, Container Apps Job is the simpler and usually cheaper answer — no Batch account, simple deployment model. The move to Azure Batch pays off when the workload decomposes into many tasks: instead of one $4$-hour container, $1{,}000$ $14$-second tasks run in parallel across $50$ Spot nodes and finish in $5$ minutes at fractional cost. The change is architectural (parallelism), not operational (service).

🟢 Exercise 6. True or false: Azure Batch requires customers to manage pool VMs (patching, OS updates).

▶💡 Hint

Look up the Batch service's responsibility line.

▶✅ Solution

False. Batch manages the pool VM lifecycle: provisioning, OS image management, patching (via the auto-OS-upgrade option). Customers select the image and VM SKU; Batch operates the fleet. Container support means tasks can even run in customer containers without touching the host OS.

🟡 Exercise 7. Design a Bicep snippet for a Batch pool of Standard_D8s_v5 nodes with Spot priority, scaling $0{-}100$ on pending task count.

▶💡 Hint

Look at Microsoft.Batch/batchAccounts/pools with autoScale + targetNodeCommunicationMode.

▶✅ Solution

bicepCopy

resource pool 'Microsoft.Batch/batchAccounts/pools@2024-02-01' = { parent: ba name: 'pool-cpu-spot' properties: { vmSize: 'Standard_D8s_v5' deploymentConfiguration: { virtualMachineConfiguration: { imageReference: { publisher: 'microsoft-azure-batch', offer: 'ubuntu-server-container', sku: '20-04-lts', version: 'latest' } nodeAgentSkuId: 'batch.node.ubuntu 20.04' } } scaleSettings: { autoScale: { evaluationInterval: 'PT15M' formula: '$TargetLowPriorityNodes = min(100, max(0, avg($PendingTasks.GetSample(180 * TimeInterval_Second))));\n$NodeDeallocationOption = taskcompletion;' } } taskSlotsPerNode: 8 taskSchedulingPolicy: { nodeFillType: 'Pack' } } }

Summary & Concept Map

The headline takeaways from this lesson:

• Pick by shape, not by familiarity. Embarrassingly parallel $\to$ Azure Batch (or ACA Jobs). Spark $\to$ Databricks / Synapse. MPI $\to$ Azure Batch HPC or CycleCloud. Containerised one-shot $\to$ Container Apps Jobs.
• Azure Batch pool / job / task model is the canonical batch abstraction. Pool sizing and auto-scale formula are most of the optimisation lever.
• Spot / low-priority should be the default for batch. Task-level retry covers eviction risk.
• Spark workloads belong on Spark hosts. Databricks for MLflow-centric teams, Synapse Spark for Synapse-centric teams, Fabric for the newest unified analytics.
• HPC needs HPC VM SKUs. Standard D-series will not meet MPI latency budgets — use HB / HC / HX in a PPG.
• Storage and identity matter as much as compute. Pair Batch pools with Blob (input/output), Files / NetApp (shared FS), and managed identity (no keys).

Walk the map from workload to shape to service. The supporting layers — storage, identity, networking, observability — are largely service-agnostic and matter equally everywhere.