Recommend a Solution for Migrating Unstructured Data — Lesson

A large global media company plans to move 400 TB of video assets from on-prem storage to Azure for archive and distribution. The engineering manager runs the math: at the company's 1 Gbps internet link, the transfer would take $\sim 36$ days of continuous saturation — assuming no other traffic, no failures, and no contention. The architect intervenes: order an Azure Data Box Heavy (1 PB capacity), copy the data over the local network in a week, ship the appliance to Microsoft, and have the data online in Azure within 14 days. Cost for the appliance and shipping: a few thousand dollars. Saved: a month of broken internet, operational misery, and the kind of executive panic that follows multi-week schedule slips. This lesson is about choosing the right tool for moving unstructured data — AzCopy, Azure Storage Mover, Azure File Sync, the Azure Data Box family — so the transfer matches the data volume, the deadline, and the available bandwidth.

We will work through Azure's unstructured-data migration story the way the AZ-305 exam expects you to: choosing among online tools (AzCopy, Azure Storage Mover, Robocopy + AzCopy, Azure File Sync), offline tools (Azure Data Box Disk / 80 TB / Heavy / Gateway), and combining them. Reference: the AZ-305 exam study guide, particularly Chapter 4 Skill 4.3 on unstructured-data migration.

Why This Matters

Unstructured data — blobs, files, images, logs, videos, scientific datasets, machine-learning training corpora — is usually the largest single thing to move in any migration project. A petabyte over a 1 Gbps pipe takes months; a petabyte on physical disks takes days. The choice between online and offline is mostly about bandwidth and time-to-Azure. The AZ-305 exam tests this LO because the wrong choice creates schedule slippage that propagates through the rest of the migration.

The career payoff is concrete: every "we have a lot of files to move" workshop, every storage-migration scoping meeting, every media or scientific-data or backup-archival workload touches this LO at some point during the migration project. If you can match a volume + bandwidth + deadline to the right tool, configure throughput and integrity options well, and combine offline seeding with online delta sync when appropriate, you will pass this slice of the exam and design unstructured-data migrations like a senior architect.

Prerequisites

• Bandwidth math. Can you estimate transfer time at a given bandwidth? — Self-check: how long does 1 TB take at 1 Gbps?
• Storage account services. Are you familiar with Blob, Files, Queues, Tables? — Self-check: which is the typical target for unstructured data?
• SMB / NFS. Do you know which one is Windows-native? — Self-check: which Azure service exposes both?
• AzCopy. Have you used the CLI? — Self-check: which authentication options does it support?
• Network throughput vs latency. Can you describe why bulk transfer benefits from parallelism? — Self-check: how many concurrent threads does AzCopy default to?

If any of these feels shaky, pause and review LO-19 (unstructured storage), LO-28 (HA for unstructured data), and LO-41 (assessment) of the AZ-305 guide before continuing this lesson on actually moving the data.

Learning Objectives

1. Analyse an unstructured data workload (volume, file count, growth, bandwidth) and translate it to the right migration tool.
2. Evaluate trade-offs between AzCopy, Storage Mover, Robocopy + AzCopy, Azure File Sync, Azure Data Box family.
3. Recommend online vs offline transfer based on volume / bandwidth.
4. Design a hybrid plan combining Data Box for the bulk seed and online tools for the delta sync.
5. Configure AzCopy and Storage Mover with managed identity, parallelism, and integrity verification.
6. Recognise anti-patterns — saturating production internet for offline data, using AzCopy alone for petabyte-scale, ignoring delta after a Data Box seed.

Building Blocks

AzCopy — Microsoft's command-line tool for transferring data into / out of Azure Storage. Formally, a cross-platform binary that uses HTTPS and parallel block uploads to maximise throughput. Best for ad-hoc transfers up to multi-TB.

Azure Storage Mover — Azure's managed service for at-scale migrations from on-prem NFS shares (and other sources) to Azure Blob. Formally, Microsoft.StorageMover/storageMovers with agents and job-definition / job-run resources that orchestrate replication. Best for repeatable / continuous file-share migration scenarios.

Azure File Sync — Hybrid file-share replication. Formally, syncs Windows Server file shares with Azure Files cloud shares, with tiered cloud storage and multi-server endpoints. Best when retaining on-prem access is required during/after migration.

Azure Data Box family — Physical appliances Microsoft ships to customers for offline data transfer. SKUs: Data Box Disk (up to 35 TB usable across 5 SSDs), Data Box (80 TB rugged appliance), Data Box Heavy (770 TB usable in a wheeled chassis), Data Box Gateway (a continuously-streaming virtual appliance, not for one-shot bulk). Best when bandwidth-constrained.

Storage Migration Service (SMS) — Windows Server-side migration tool for legacy file servers. Formally, a feature of Windows Admin Center that orchestrates SMB share migration including ACLs and share metadata. Best when migrating Windows file servers in-place to Azure VMs / Azure Files.

Online migration — Transfer over the internet or ExpressRoute. Use when bandwidth × time fits the data volume.

Offline migration — Transfer via physical appliance. Use when bandwidth is the bottleneck.

Delta sync — A second-pass transfer that picks up only changes since the previous run. Required after Data Box seeds (the initial seed captures a snapshot; the delta brings the target up to source-current).

ACL / metadata preservation — Whether the migration tool preserves file permissions, ownership, timestamps. AzCopy preserves most; Storage Mover preserves; Storage Migration Service preserves ACLs natively; Data Box copy via AzCopy preserves; bare file-copy may not.

Deep Dive

1. Volume × bandwidth — online vs offline

The breakpoint is roughly: if online would take longer than the deadline (or longer than you can spare bandwidth), go offline.

[!TIP] Above $\sim 40$ TB at 1 Gbps, the Data Box family usually wins. Below that, online tools usually win on operational simplicity.

2. Online tools — when to pick each

[!NOTE] AzCopy's --block-size-mb matters for large files: larger block sizes improve throughput up to a point. 100 MB is a sensible default for $> 1$ GB files; smaller blocks for many small files.

3. Offline — the Data Box family

[!IMPORTANT] Data Box transfers a snapshot of the source. After return-and-ingest, source data has continued to evolve. Plan an explicit delta sync step (with AzCopy or Storage Mover) to capture and replay the changes between the snapshot time and the planned cutover time, including any files modified during the appliance's transit.

4. Azure File Sync — keep on-prem reads alive

Azure File Sync is not a one-shot migration tool — it's a continuous-sync platform. Configure a cloud endpoint (Azure Files share) and one or more server endpoints (Windows Servers with hot caches). Cold files tier to the cloud; hot files stay on-prem. Useful when you want a long-running hybrid arrangement rather than a clean cutover.

5. Storage Mover — managed migration as a service

Storage Mover orchestrates NFS-to-Blob migrations and (newer) SMB-to-Files migrations at scale. Define endpoints (source NFS share, target Azure container), then define jobs that map source-to-target with copy rules, retries, and monitoring. Better than scripting AzCopy for large estates because of built-in state tracking, retries, scheduling, and per-job audit.

6. The hybrid plan — seed with Data Box, follow with online delta

For very large estates, the canonical pattern combines tools:

Day $1{-}7$: copy bulk to Data Box. Day $8{-}21$: ship and Microsoft ingests. Day $14{-}25$: run online delta sync continuously. Day 26: final cutover (brief source quiesce, last delta, repoint applications).

7. ACL, metadata, and integrity considerations

Some workloads care about file ACLs (e.g., file shares with permission inheritance), timestamps (legal hold), or content integrity (MD5). Match tool to need:

[!WARNING] Naive cp or xcopy strips ACLs and metadata. Always use a migration-aware tool for shares with permission semantics.

8. Performance tuning for AzCopy

Throughput is rarely the network ceiling — it's tool tuning. Key levers:

Pair --log-level INFO with --output-type json for parseable diagnostics. On Linux, set TCP window sizing (net.core.rmem_max) for high-bandwidth-delay paths.

[!TIP] For a 1 Gbps WAN, expect $\sim 100{-}120$ MB/s effective. Lower means there's a bottleneck — CPU, disk, or network. Run a baseline with iperf first to know the network ceiling.

9. Cost and shipping logistics for Data Box

Data Box pricing has three components: appliance rental (flat fee), shipping, and Azure ingress (free for inbound). Approximate ranges (illustrative; check current pricing):

Plus shipping (white-glove for Data Box Heavy). For one-time petabyte transfers, the cost is negligible relative to the bandwidth-replacement alternative. Multiple Data Boxes can be requested in parallel for very large estates.

[!IMPORTANT] Data Box is not available in every Azure region. Verify region availability before planning. Some sovereign clouds (China, Government) have separate Data Box programs with different SLAs.

10. Validating an unstructured-data migration

A migrated estate needs validation that goes beyond "the bytes transferred". Checks:

For petabyte-scale migrations, sampling is the practical approach: pick 1% of files randomly, hash both ends, confirm match. Full hash of every file at PB scale is rarely feasible in the cutover window.

Worked Examples

Easy — pick the tool

Problem. 2 TB of blobs from on-prem object store to Azure Blob. 1 Gbps internet. Recommend.

Solution. AzCopy online with managed-identity authentication. At 1 Gbps, 2 TB takes $\sim 5$ hours saturated, a few days at fractional bandwidth. Use --put-md5 --check-md5 FailIfDifferent for integrity verification. Data Box is overkill at this size — operational overhead would outweigh the bandwidth savings.

Medium — large volume, narrow pipe

Problem. 200 TB of media files, 100 Mbps internet, deadline 45 days. Recommend.

Solution. Data Box Heavy (or two regular Data Boxes in parallel). 200 TB at 100 Mbps online would take $\sim 200$ days continuous saturation — infeasible. Order the Data Box appliance(s), copy in roughly a week over the LAN, ship to Microsoft, who ingests into the target storage account within about two weeks. Run a delta sync with AzCopy or Storage Mover for any files modified between the Data Box snapshot date and final cutover. Total elapsed time fits comfortably inside the 45-day deadline with margin.

Hard — hybrid file share with continuous sync

Problem. A 50 TB corporate file share with permission inheritance must move to Azure Files. Users need uninterrupted on-prem access during and after migration.

Solution. Azure File Sync is the canonical answer. Deploy a Windows Server endpoint pointing to the existing share data; create the cloud endpoint (an Azure Files Premium share with ZRS redundancy) and add it to a sync group with the server endpoint. Enable cloud tiering so cold files migrate to Azure transparently while hot files stay locally cached. Users continue accessing the existing on-prem server during the entire migration period. After migration completion, the on-prem server remains as a hot cache for users who prefer LAN access (or is decommissioned and replaced by direct Azure Files mount over SMB / Private Link if all users have migrated to remote-friendly tooling).

11. Target storage choices for migrated data

Where the migrated data lands in Azure shapes long-term cost and operations. The exam tests recognising the right target.

Plan target tier at migration time. A Data Box ingest into Hot tier is straightforward; later tier transitions add operations. Use Lifecycle Management policies on Blob to age data automatically from Hot $\to$ Cool $\to$ Cold $\to$ Archive based on last-modified time.

12. Resumability and failure recovery

A multi-day transfer must survive transient failures. Look for tools that resume gracefully:

A killed AzCopy process can be resumed by pointing to the same destination with --overwrite ifSourceNewer. Storage Mover persists job state in Azure, so an agent failure does not lose progress.

[!WARNING] Long-running AzCopy without journal-aware retries is a frequent cause of "we restarted from scratch" frustration. Use the azcopy jobs resume <jobid> workflow rather than re-invoking azcopy copy.

Visual Explanations

Figure 1 — Tool decision flow

Figure 2 — Data Box flow

Figure 3 — Quick chooser

Figure 4 — Decision matrix by volume and bandwidth

The two-axis decision matrix that experienced architects keep in their head:

The "right answer" is partly negotiable: if you can wait, online tools work for larger volumes; if you need it yesterday, offline wins. Always check the deadline before defaulting.

Figure 5 — Steady-state hybrid storage

For workloads that don't fully cut over (continuous hybrid), the Azure File Sync or Data Box Gateway patterns establish ongoing replication rather than one-shot migration. These are operational architectures rather than migration projects per se — but the AZ-305 exam treats them under this LO when the question is "how do we keep on-prem and Azure in sync long-term?".

Figure 7 — Hybrid combined pattern

A practical multi-tool plan for very large migrations:

The total elapsed time depends on Data Box logistics (carrier shipping time, customs delays for international shipments, ingest queue depth at Microsoft) and on how active the source is during the delta-sync window. Plan with comfortable margin; never promise a hard customer-facing deadline that depends entirely on a Microsoft-controlled ingest queue you cannot accelerate yourself.

Common Mistakes

❌ Myth: "AzCopy can do anything — use it for the whole migration." ✅ Reality: AzCopy is great for one-shot transfers but lacks the orchestration, retries, and scheduling that Storage Mover gives at scale.

❌ Myth: "Order one Data Box for any large transfer." ✅ Reality: Above 80 TB usable, Data Box requires multiple appliances or upgrade to Data Box Heavy. Plan capacity accordingly.

❌ Myth: "After Data Box ingests, the migration is done." ✅ Reality: Data Box transfers a snapshot. Source data has changed since. Plan delta sync to capture changes between snapshot and cutover.

❌ Myth: "Robocopy is fine for moving Windows shares to Azure Files." ✅ Reality: Robocopy with /COPY:DATSOU /MIR is acceptable but lacks orchestration; AzCopy or Azure File Sync handle large or repeatable scenarios more reliably.

Practice Exercises

🟢 Exercise 1. 5 TB blob migration, 1 Gbps, weekend deadline.

▶✅ Solution

AzCopy online; saturated transfer fits well within the weekend.

🟡 Exercise 2. 300 TB media at 200 Mbps. Deadline 30 days.

▶✅ Solution

Data Box Heavy. Online would take $\sim 150$ days; Data Box completes the bulk in $\sim 14$ days plus delta sync.

🟡 Exercise 3. 50 TB corporate file share, users still active on-prem, 1 Gbps.

▶✅ Solution

Azure File Sync with cloud tiering. Online sync over weeks; users continue using on-prem.

🔴 Exercise 4. 10 TB NFS share, 500 Mbps, repeatable monthly snapshots needed.

▶✅ Solution

Storage Mover with scheduled jobs. Initial copy over a few days; monthly delta jobs orchestrated by Storage Mover schedules.

🔴 Exercise 5. 1 PB scientific dataset, 1 Gbps shared with the rest of the campus.

▶✅ Solution

Multiple Data Box Heavy appliances in parallel. Online transfer would either saturate the campus link or take a year.

🟢 Exercise 6. True or false: AzCopy preserves NTFS ACLs.

▶✅ Solution

False (mostly). AzCopy preserves blob metadata but not NTFS ACLs. For ACL preservation use Azure File Sync or Storage Migration Service.

🟡 Exercise 7. Sketch an AzCopy command with managed identity and integrity verification.

▶✅ Solution

bashCopy

azcopy login --identity azcopy copy 'D:/data' \ 'https://stproddata.blob.core.windows.net/archive' \ --recursive --put-md5 --check-md5 FailIfDifferent \ --block-size-mb 100

Figure 6 — Selecting the right Azure Files tier

For migrations that target Azure Files, the SKU choice depends on the workload's IOPS and capacity needs.

Most migration projects start with Standard and upgrade to Premium for the workloads that need it after observing real usage. Right-sizing the tier post-migration is much cheaper than over-buying upfront.

Summary & Concept Map

• Online vs offline is decided by volume × bandwidth × deadline. Above $\sim 40$ TB at 1 Gbps, Data Box usually wins.
• AzCopy for ad-hoc; Storage Mover for managed orchestration; Azure File Sync for hybrid file shares.
• Data Box family scales from 35 TB to 770 TB usable. Pick by total transfer size.
• Always plan a delta sync after a Data Box seed.
• Preserve ACLs and metadata via the right tool. AzCopy alone is not enough for permission-bearing shares.
• Storage Migration Service modernises Windows file servers in-place.