Skip to main content

Change data capture (CDC)

You'll ingest database tables into Unity Catalog with a CDC gateway and ingestion pipeline in ~15 min.

Prereqs: Databases and SaaS ingestion, managed database connector requirements

The big picture​

This walkthrough creates the standard CDC architecture. A gateway runs continuously on classic compute and writes source changes to a Unity Catalog staging volume. A scheduled serverless ingestion pipeline applies those changes to destination streaming tables.

The screenshots show a SQL Server setup. Other supported CDC database connectors follow the same overall workflow, but source preparation, authentication methods, and available settings vary by connector. Follow the source setup documentation for your database.

Before you start​

You need:

  • A supported source database configured for CDC according to its connector documentation.
  • A network path from classic compute in the workspace to the source database.
  • Permission to create a Unity Catalog connection, or USE CONNECTION on an existing one.
  • Permission to create the gateway compute, destination tables, and staging volume.

Choose an authentication method supported by your database. The screenshots use a username and password. Use an existing connection instead if your organization manages source credentials centrally.

Create the CDC pipeline​

Walkthrough · 9 steps · ~15 min

Create the connection, gateway, and ingestion pipeline, then choose the source and destination.

1

In the Databricks workspace, open Data Ingestion, then select your database source under Databricks connectors. This example selects SQL Server.

Data Ingestion page with SQL Server selected under Databricks connectors
2

On the Connection page, click Create connection.

SQL Server ingestion wizard with the Create connection button
3

Choose an authentication method and enter the required connection details. This example uses Username and password for SQL Server. Click Create connection.

Create a SQL Server connection dialog with username, password, host, and port fields
4

Select Change data capture. Name the ingestion pipeline and choose an event log catalog and schema. Name the ingestion gateway and choose its staging catalog and schema, then click Create pipelines and continue.

SQL Server ingestion setup with change data capture selected and pipeline and gateway settings configured
5

Expand the source database and schema. Select the tables that you want to ingest, then click Next.

SQL Server source selection with customers and orders selected
6

Select the destination catalog and schema, then click Save and continue.

CDC destination selection with the dev catalog and sql-server-cdc schema selected
7

On Database setup, click Validate. This checks the source support objects, CDC configuration, and connector permissions.

SQL Server database setup page with the Validate button
8

Confirm that every check passes and the database reports Validation successful, then click Next.

SQL Server database setup showing successful validation for support objects, CDC, and permissions
9

Create a schedule for the ingestion pipeline based on your data freshness requirements. Add notifications if needed, then click Save and run pipeline.

warning

The schedule controls the ingestion pipeline. Do not stop the gateway. It must run continuously so it captures changes before the source removes them.

CDC schedules and notifications page with an hourly schedule and Save and run pipeline button

Reference: Managed database connectors (Databricks docs)

Verify​

Verify · 1 steps · ~2 min

Confirm the first ingestion update completed and the gateway is still running.

1

Open the ingestion pipeline in Jobs & Pipelines. Confirm the run is Completed, the selected streaming tables have green status indicators, and Gateway status is Running.

Completed CDC ingestion pipeline with customers and orders streaming tables and a running gateway

Where people trip​

Database validation fails

Expand the failed validation group and use Complete configuration to see what the source is missing. Common causes include missing support objects, CDC or change tracking that is not enabled, and insufficient permissions for the connection user. The exact checks depend on the connector.

The gateway cannot reach the source database

The gateway runs on classic compute in the workspace network. Check routing, private connectivity, firewall rules, and the database port before changing pipeline settings.

The gateway was stopped

Restarting the gateway does not recover change data that the source already removed. If the source retention window passed, run a full refresh for the affected tables after the gateway is healthy.

Next​