Scaling and Decomposing Pipelines

There is no single rule for how to divide pipelines. Choices depend on organizational structure, CI/CD practices, data complexity (sources, transforms, volumes), latency and SLAs, and related constraints.

Warning

Be aware of current Pipeline and concurrency limits for Spark Declarative Pipelines. Limits change over time; check:

• https://docs.databricks.com/en/resources/limits.html

• https://docs.databricks.com/en/delta-live-tables/limitations.html

Once you have chosen a logical grouping (see Bundle Scope and Structure), you can decompose a large pipeline where natural boundaries exist.

Example: start with one pipeline that has two Flow Groups flowing into a target via staging tables:

The same design decomposed into three pipelines:

• Each Flow Group is its own pipeline, targeting a final staging table.

• A final pipeline merges upstream staging tables into the target table.

For decomposed multi-source samples, see samples/pattern-samples (Multi Source Streaming Decomposed Staging and Final pipelines).

See also Data Flow and Pipeline Patterns for the pattern catalog and Mix and Match Patterns.