Skip to main content

Create Ingestion Pipeline

You'll understand how Lakeflow Connect ingestion pipelines work and choose a deployment method in ~5 min.

Prereqs: Create connection, familiarity with Lakeflow Connect

Why this matters​

A connection lets Databricks reach an external system. An ingestion pipeline uses that connection to continuously replicate data — including change data capture (CDC) — into Unity Catalog tables. Without a pipeline, the connection sits idle and data stays in the source system.

How it works​

Lakeflow Connect ingestion pipelines consist of two components:

  • Ingestion gateway — reads data from the source system and stages it in cloud storage. Runs on classic compute inside your VPC (required for network access to the source).
  • Ingestion pipeline — reads staged data and writes it into UC tables. Can run serverless because it accesses UC default storage directly.
warning

As of March 2026, the Databricks UI creates fully serverless ingestion pipelines. If classic compute is required for the ingestion gateway (e.g., for network connectivity reasons), deploy the pipeline through Databricks Asset Bundles (DABs).

See database connector components for details on which component runs where.

Video tutorials​

Lakeflow Connect overview​

SQL Server — Change Data Capture (CDC)​

Salesforce​

SharePoint​

ServiceNow​

Deploy with DABs​

For pipelines that require classic compute on the gateway, or for version-controlled, repeatable deployments:

  • DABs definition — Define and deploy an ingestion pipeline using Databricks Asset Bundles.

Next​