Recommend a Solution for Data Integration

Data integration moves, transforms, coordinates, and publishes data across systems. The architect's challenge is rarely “Which Azure service can copy a file?” Many services can. The useful question is: Which integration model satisfies the workload's latency, transformation, connectivity, governance, and operating constraints without turning every data movement into custom code?

This lesson covers modern Microsoft choices: Data Factory in Microsoft Fabric for new unified analytics programs, Azure Data Factory (ADF) for established Azure integration and hybrid estates, Azure Synapse pipelines where ingestion is anchored in an existing Synapse workspace, Azure Databricks for code-heavy engineering, and event/workflow services when the requirement is not a batch data pipeline at all. It also covers the integration runtime, ETL versus ELT, network isolation, orchestration, and observability.

Learning objectives

After this lesson, you should be able to:

1. Separate analytical data integration from application workflow and event-streaming requirements.
2. Choose between Fabric Data Factory, Azure Data Factory, Synapse pipelines, Databricks, Logic Apps, and streaming services.
3. Select Azure, self-hosted, or Azure-SSIS integration runtime for ADF/Synapse connectivity.
4. Decide between ETL and ELT based on the destination and transformation model.
5. Design secure hybrid connectivity without exposing source databases publicly.
6. Recommend orchestration, metadata, retry, monitoring, and deployment practices.
7. Explain why a plausible neighboring service does not meet a scenario's dominant constraint.

First classify the integration workload

Before naming a service, ask six questions:

1. What moves? Rows, files, events, API messages, or business documents?
2. How quickly? Nightly batch, every few minutes, seconds, or continuous stream?
3. Where are the endpoints? Public cloud, private Azure network, on-premises, another cloud, or SaaS?
4. What processing occurs? Copy only, schema mapping, visual transformations, SQL, Spark, or application logic?
5. Where should transformation compute run? In the pipeline, in the destination warehouse/lakehouse, or in a dedicated engine?
6. Who operates it? Data engineers, BI teams, integration specialists, application developers, or a mixed platform team?

Text equivalent: classify the payload first. Bulk analytical data proceeds to a batch/stream decision and then to the team's platform; business workflow uses Logic Apps, discrete events use Event Grid, and durable telemetry streams use Event Hubs with a processor.

Compare the main choices

Microsoft currently describes Data Factory in Fabric as the next generation of Azure Data Factory and recommends that new data-integration programs evaluate Fabric first. That does not make every existing ADF architecture obsolete. ADF remains a managed Azure service with broad connectors, established hybrid integration runtimes, mature deployment patterns, and many production estates. For an exam scenario, respect explicit platform constraints: an existing Synapse or ADF investment is evidence, not noise.

Data Factory in Microsoft Fabric

Choose Fabric Data Factory when the organization is building a unified analytics platform around Fabric workspaces, OneLake, lakehouses, warehouses, Power BI, real-time intelligence, and data science. Pipelines coordinate movement and activities; Dataflow Gen2 provides a low-code Power Query transformation path. The benefit is the shared SaaS operating model and integrated governance rather than merely a different copy engine.

Check capacity, region, connector, private-network, deployment, and lifecycle requirements against current Fabric documentation. A requirement for a classic ADF feature should be verified rather than assumed equivalent.

Azure Data Factory

ADF is a managed data integration and orchestration service. Its core objects are:

• Linked service: connection and authentication information for a data store or compute service.
• Dataset: a typed reference to data, such as a table, folder, file format, or object path.
• Pipeline: a logical workflow of activities.
• Activity: copy, transformation, external compute, or control-flow work.
• Trigger: schedule, tumbling-window, or event-based pipeline start.
• Integration runtime: the execution/connectivity infrastructure used by activities.

ADF does not become the system of record simply because it orchestrates the flow. Data remains in sources, staging areas, lakes, warehouses, or sinks. Store configuration as code, parameterize environment differences, and keep credentials out of pipeline JSON.

Azure Synapse pipelines

Synapse pipelines share much of ADF's pipeline model and connector experience. Choose them when the organization already operates Synapse and wants orchestration beside Synapse SQL pools, Spark pools, notebooks, and workspace security. Do not deploy both ADF and Synapse pipelines without a clear ownership boundary; duplicated orchestration creates two monitoring planes and competing schedules.

Azure Databricks

Choose Databricks when transformation is the center of gravity: large-scale Spark processing, Delta Lake/lakehouse design, code review, reusable libraries, streaming tables, advanced data quality, or data-science collaboration. ADF or Fabric can still orchestrate Databricks jobs. The choice is not always either/or: a factory can provide connectors, schedules, and control flow while Databricks performs complex transformations.

Logic Apps, Event Grid, Event Hubs, and Functions

Use Logic Apps for business-process integration: SaaS connectors, approvals, B2B messages, API coordination, and long-running workflows. Use Event Grid for discrete event notification and reactive routing. Use Event Hubs for high-throughput durable event ingestion, often followed by Stream Analytics, Fabric event streams, Functions, or Spark structured streaming. Use Functions for bounded event-driven code, not as an improvised replacement for a governed, multi-stage data platform.

Integration runtime: where movement and dispatch happen

For ADF and Synapse pipelines, the integration runtime (IR) is the compute and connectivity bridge.

Azure integration runtime

Azure IR is Microsoft-managed and supports cloud data movement, activity dispatch, and Mapping Data Flow execution in supported scenarios. Choose it for publicly reachable cloud endpoints or for private connectivity through supported managed virtual network/private endpoint patterns. Select an appropriate region when data residency, performance, or egress matters.

Self-hosted integration runtime

Self-hosted IR runs on Windows infrastructure that the organization manages. It reaches on-premises or private-network data stores and initiates outbound connections, so databases need not be opened to inbound internet traffic. Use multiple nodes for availability and throughput, size them for concurrent movement, patch them, monitor them, and place them near data sources to reduce latency.

Self-hosted IR is a connectivity and execution bridge—not a reason to embed passwords in a pipeline. Use managed identity where supported, Key Vault references for unavoidable secrets, and least-privilege source/sink accounts.

Azure-SSIS integration runtime

Azure-SSIS IR executes existing SQL Server Integration Services packages in a managed Azure environment. Choose it for a deliberate SSIS lift-and-shift or staged modernization. Do not select it for a new cloud-native pipeline merely because the organization once used SSIS.

ETL versus ELT

ETL extracts data, transforms it in the integration/processing tier, then loads a curated result. Choose ETL when sensitive fields must be removed before landing, the sink has limited compute, or only conforming records may enter the target.

ELT extracts and loads raw or lightly standardized data into a scalable lakehouse or warehouse, then transforms it there using SQL, Spark, or platform-native tools. Choose ELT when preserving raw history matters, destination compute is powerful, multiple consumers need different transformations, or schema evolution is expected.

Many modern platforms use both: validate and redact at ingress, land immutable data, then perform dimensional or medallion transformations in the destination. The architectural decision includes data contracts, retention, cataloging, lineage, and replay—not just where one SQL statement runs.

Text equivalent: sources land in an immutable bronze layer, quality rules create validated silver data, business transformations create gold serving models, and preserved raw data supports replay.

Reliability and restartability

A pipeline is not reliable merely because each activity can retry. Design for end-to-end repeatability:

• Use a watermark, change tracking, change data capture, or source-provided sequence to define incremental work.
• Land files under deterministic batch/run paths and record run metadata.
• Make sink operations idempotent through merge/upsert keys, partition replacement, or transactional staging.
• Quarantine invalid records with reason codes rather than silently discard them.
• Bound retries and route persistent failures to an operational queue or incident.
• Separate transient connectivity failure from deterministic schema/data-quality failure.
• Track freshness, row counts, rejected counts, duration, throughput, and cost.
• Test replay after partial failure and late-arriving data.

Tumbling-window triggers are useful for stateful, contiguous time windows with dependencies and reruns. A simple schedule trigger is sufficient when windows do not require that behavior. Event triggers are useful when object arrival is the business signal, but the pipeline must still handle duplicates, ordering, and files that arrive before they are complete.

Security and networking

Use managed identities for factory/workspace access to Azure services where supported. Reference Key Vault for credentials that cannot be eliminated. Give the pipeline only the source read and sink write/data-plane roles it requires. Treat a linked service as configuration—not permission.

For private sources, use self-hosted IR or supported managed private connectivity. Plan name resolution, firewall rules, outbound paths, proxy behavior, and certificate trust. Do not solve private connectivity by assigning a broader Azure role; RBAC and network reachability are independent gates.

Data residency can constrain the IR location, staging location, logs, and failure payloads. Sensitive data in diagnostic logs or rejected-record samples is still sensitive. Apply masking, retention, and access controls to the operational plane.

Worked scenario: retail analytics across hybrid systems

Fabrikam has an on-premises SQL Server ERP, a SaaS commerce platform, clickstream events, and a new Fabric analytics environment. ERP changes must appear in curated analytics within 30 minutes. Clickstream dashboards require seconds-level freshness. The company cannot accept inbound internet connections to its datacenter.

Recommended design:

1. Use a clustered/self-hosted IR near the ERP to make outbound connections and extract incrementally. Land raw changes in the governed lake/OneLake path and transform through Fabric data pipelines/Dataflow Gen2 or notebooks according to team skills.
2. Use supported Fabric/ADF SaaS connectors for commerce data; apply watermarks and replayable landing paths.
3. Send clickstream telemetry to Event Hubs or the chosen Fabric real-time ingestion path. Process it with a streaming engine rather than polling through a nightly pipeline.
4. Use managed identity and private endpoints where supported, with Key Vault only for credentials that the source cannot replace.
5. Maintain one operational ownership model: pipeline SLAs for batch feeds and consumer-lag/freshness SLAs for the event stream.

Infrastructure and deployment example

The following sketch shows the intent, not a complete production template: a factory identity receives secret access, while a self-hosted IR is registered separately inside the private network.

Promote factories/workspaces through development, test, and production using source-controlled definitions and environment parameters. Avoid manual production edits that cannot be reproduced. Validate connector and API versions, because integration products evolve rapidly.

Common exam traps

• ADF stores the integrated data. False. It orchestrates movement and processing; stores/sinks retain data.
• Self-hosted IR requires inbound public access. False. It is designed to initiate outbound connectivity to the service.
• Every near-real-time requirement means ADF. False. Durable telemetry streams usually require Event Hubs or a Fabric real-time path plus stream processing.
• Logic Apps is a large-scale analytical ETL engine. False. It excels at application and business workflow integration.
• Synapse pipelines and ADF must both be deployed. False. Choose a clear orchestration boundary.
• Mapping Data Flows run on customer-managed Databricks. False. They use managed Spark execution; explicitly invoke Databricks when that is the selected engine.
• ELT means no ingress validation. False. Security, schema, and data-contract checks may still be required before or during landing.
• A linked service grants access. False. Authentication material and source/sink authorization still control access.

Retrieval checkpoint

Summary

Choose a data-integration solution by classifying payload, latency, endpoints, transformation complexity, destination compute, and operator skills. Fabric Data Factory is the natural first evaluation for new Fabric-centered analytics; ADF remains strong for Azure and hybrid orchestration; Synapse pipelines fit an established Synapse boundary; Databricks fits code-heavy engineering. Use Logic Apps for workflows and event services for streaming/event-driven integration. Make pipelines restartable, idempotent, observable, privately connected, and reproducibly deployed.

Sources and freshness

Reviewed 2026-08-02. Scope follows the April 17, 2026 AZ-305 study guide. Current behavior is grounded in Microsoft's Data Factory in Fabric overview, ADF/Synapse integration runtime documentation, Azure Architecture Center data lake guidance, and current service documentation for ADF, Synapse, Event Hubs, Event Grid, Logic Apps, and Databricks. The attached Exam Ref supplies historical teaching context; live Microsoft documentation controls current product recommendations.