Pattern - Basic 1:1¶

Description¶

The basic 1:1 pattern is the default building block for most lakehouse pipelines. Use it whenever one streaming source maps to one target table and you only need row-level transforms—no joins, no multi-source merges, no stream-static semantics.

This pattern covers the situations teams hit most often:

Bronze / source ingest — stream or batch load from staging or raw sources into bronze tables using append or auto CDC flows.
Silver / conformed basic entities — promote one upstream streaming table (typically bronze) into a source-aligned silver entity, or apply light transformation and conformance (schema enforcement, cleansing, typing) before serving downstream gold layers.

The same flow shape repeats at both layers: an input view over a single CDF-enabled stream, an append or auto CDC flow, and one streaming target table defined in the data flow spec. Most new projects implement this pattern many times across bronze and silver before adopting Advanced composition patterns.

Use when:

You are ingesting data or performing one-to-one loads.
You are promoting one upstream table into a conformed entity table.
You need append-only, SCD1, or SCD2 behavior on the target.
You only need basic single-row transforms (data type conversion, formatting, cleansing, data quality).

Layers: Bronze and Silver

Models (silver): 3NF / ODS, Data Vault hubs and satellites, conformed dimensions

Data Flow Components:

No.	Component	Description	M / O
1	Input View	Input view over the streaming source table (raw/staging at bronze, upstream bronze or silver at silver). Optionally reads CDF when the source has change data feed enabled.	M
2	Flow	Append or auto CDC flow to the target streaming table (SCD1/2 via `cdcSettings` when required).	M
3	Target Table	Streaming table defined in the data flow spec.	M

Feature Support¶

Supported	Not Supported
Append Only & SCD 1/2 Basic transforms such as: Data type conversion Concatenation Single row calculations Formatting Schema enforcement, cleansing, and data quality expectations Quarantine for failed quality checks	Complex transforms Joins across multiple streaming sources (see Multi-Source Streaming) Stream-static joins (see Stream-Static Basic) Window By

Sample¶

Bronze

Bundle: samples/pattern-samples
Pipeline: Lakeflow Framework - Pattern - Base Bronze Samples Pipeline
Samples:
- samples/pattern-samples/src/dataflows/base_samples/bronze/dataflowspec/customer_main.json
- samples/pattern-samples/src/dataflows/base_samples/bronze/dataflowspec/customer_address_main.json

Silver

Pipeline: Lakeflow Framework - Pattern - Base Silver Samples Pipeline
Samples:
- samples/pattern-samples/src/dataflows/base_samples/silver/dataflowspec/customer_main.json (SCD2)
- samples/pattern-samples/src/dataflows/base_samples/silver/dataflowspec/customer_address_main.json (SCD2)

Example Data Flow¶

Day 1 Load¶

Source Table (Append-Only)¶

CUSTOMER

customer_id	first_name	last_name	email	load_timestamp
1	John	Doe	john.doe@example.com	2023-01-01 10:00
2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00

Target Table¶

Append-Only Scenario¶

customer_id	first_name	last_name	email	load_timestamp
1	John	Doe	john.doe@example.com	2023-01-01 10:00
2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00

SCD1 Scenario¶

customer_id	first_name	last_name	email	load_timestamp
1	John	Doe	john.doe@example.com	2023-01-01 10:00
2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00

SCD2 Scenario¶

customer_id	first_name	last_name	email	_START_AT	_END_AT
1	John	Doe	john.doe@example.com	2023-01-01 10:00	NULL
2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00	NULL

Day 2 Load¶

Day 2 Source Table (Append-Only)¶

CUSTOMER

customer_id	first_name	last_name	email	load_timestamp
1	John	Doe	john.doe@example.com	2023-01-01 10:00
2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00
1	John	Doe	jdoe@example.com	2023-01-02 10:00
3	Alice	Green	alice.green@example.com	2023-01-02 10:00
4	Joe	Bloggs	joe.bloggs@example.com	2023-01-02 10:00

Day 2 Target Table¶

Day 2 Append-Only Scenario¶

customer_id	first_name	last_name	email	load_timestamp
1	John	Doe	john.doe@example.com	2023-01-01 10:00
2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00
1	John	Doe	jdoe@example.com	2023-01-02 10:00
3	Alice	Green	alice.green@example.com	2023-01-02 10:00
4	Joe	Bloggs	joe.bloggs@example.com	2023-01-02 10:00

Day 2 SCD1 Scenario¶

customer_id	first_name	last_name	email	load_timestamp
1	John	Doe	jdoe@example.com	2023-01-01 10:00
2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00
3	Alice	Green	alice.green@example.com	2023-01-02 10:00
4	Joe	Bloggs	joe.bloggs@example.com	2023-01-02 10:00

Day 2 SCD2 Scenario¶

customer_id	first_name	last_name	email	_START_AT	_END_AT
1	John	Doe	jdoe@example.com	2023-01-02 10:00	NULL
1	John	Doe	john.doe@example.com	2023-01-01 10:00	2023-01-02 10:00
2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00	2023-01-02 10:00
3	Alice	Green	alice.green@example.com	2023-01-01 10:00	NULL
4	Joe	Bloggs	joe.bloggs@example.com	2023-01-01 10:00	NULL