Data Flow and Pipeline Patterns

Reference patterns for designing data flows and pipelines.

Start with Base Patterns (bronze → silver → gold), then adopt Advanced composition patterns when sources, keys, or join semantics require them.

• Operating models and modeling paradigms: Architecture

• Multi-Source Streaming (Flows)

• Bundle Scope and Structure

• Scaling and Decomposing Pipelines

• Mix and Match Patterns

• SDP concepts (datasets, flows): Lakeflow Spark Declarative Pipelines — key concepts

When selecting a pattern, start with ownership model, modeling approach, source characteristics (streaming, static, CDC), and latency needs.

Note

These patterns are not an exhaustive catalog of everything the Lakeflow Framework supports. They are a curated starting point — common building blocks and composition approaches that align with the bundled pattern-samples and what most teams implement first. Combine, extend, or depart from them as your sources, modeling, and operational constraints require.

Tip

Runnable samples: End-to-end specs live in samples/pattern-samples (bronze → silver → gold, multi-source streaming, stream-static joins, CDC from snapshot, gold materialized views, and more). See The Samples for deploy steps. Each pattern page links to the matching specs in that bundle; advanced pattern pages also include multi-day example data flows for SCD and late-arriving behavior.

Base Patterns

These patterns match the default pattern-samples bronze, silver, and gold pipelines. Most new projects begin here. See also Base Patterns.

Pattern	Description
Pattern - Basic 1:1	Basic 1:1 — the default building block for most pipelines: bronze source ingest and source-aligned silver or lightly transformed conformed entities (append, SCD1, or SCD2) from one streaming source to one target.
Pattern - Gold Materialized Views	Gold — materialized views — gold serving tables from silver using full or incremental MV refresh.
Construct CDC Stream from Snapshot Source	CDC from snapshot — build a CDC stream from snapshot sources (building block).

Advanced composition

Use when multiple sources, snapshot CDC, or stream-static join semantics apply. See also Advanced Composition.

Pattern	Description
Multi-Source Streaming	Multi-source streaming — shared keys, staging append/merge, basic transforms.
Stream-Static Basic	Stream-static — basic — one driving stream joined to static tables.
Stream-Static - Streaming Data Warehouse	Stream-static — streaming DWH — CDF-driven joins when any table may change.

Scaling and decomposing pipelines

Decompose monolithic pipelines across flow groups and staging tables. See Scaling and Decomposing Pipelines.

Mix and Match Patterns

Combine multiple patterns within one pipeline or data flow. See Mix and Match Patterns.