Pattern - Basic 1:1

Description

Suitable for ingestion and basic 1:1 loads. Use when:

• You are ingesting data or performing one-to-one loads.

• You only need to perform basic single row transforms.

Layers: Generally Bronze

Data Flow Components:

No.	Component	Description	M / O
1	Input View	Input view created over the streaming source table. This view can optionally read from CDF if the source table is CDF enabled.	M
2	Flow	Append or Change flow to streaing target table.	M
3	Target Table	A streaming table, the schema of which is specified in the dataflowspec.	M

Feature Support

Supported	Not Supported
Append Only & SCD 1/2 Basic transforms such as: Data type conversion Concatenation Single row calculations Formatting Cleansing & Data Quality Rules	Complex transforms Joins Multiple streaming sources Window By

Sample

• Bundle: dlt_framework/src/samples/bronze_sample

Example Data Flow

Day 1 Load

• Source Table (Append-Only)

  CUSTOMER

  customer_id	first_name	last_name	email	load_timestamp
  1	John	Doe	john.doe@example.com	2023-01-01 10:00
  2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00

• Target Table

  Append-Only Scenario

  customer_id	first_name	last_name	email	load_timestamp
  1	John	Doe	john.doe@example.com	2023-01-01 10:00
  2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00

  SCD1 Scenario

  customer_id	first_name	last_name	email	load_timestamp
  1	John	Doe	john.doe@example.com	2023-01-01 10:00
  2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00

  SCD2 Scenario

  customer_id	first_name	last_name	email	_START_AT	_END_AT
  1	John	Doe	john.doe@example.com	2023-01-01 10:00	NULL
  2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00	NULL

Day 2 Load

• Source Table (Append-Only)

  CUSTOMER

  customer_id	first_name	last_name	email	load_timestamp
  1	John	Doe	john.doe@example.com	2023-01-01 10:00
  2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00
  1	John	Doe	jdoe@example.com	2023-01-02 10:00
  3	Alice	Green	alice.green@example.com	2023-01-02 10:00
  4	Joe	Bloggs	joe.bloggs@example.com	2023-01-02 10:00

• Target Table

  Append-Only Scenario

  customer_id	first_name	last_name	email	load_timestamp
  1	John	Doe	john.doe@example.com	2023-01-01 10:00
  2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00
  1	John	Doe	jdoe@example.com	2023-01-02 10:00
  3	Alice	Green	alice.green@example.com	2023-01-02 10:00
  4	Joe	Bloggs	joe.bloggs@example.com	2023-01-02 10:00

  SCD1 Scenario

  customer_id	first_name	last_name	email	load_timestamp
  1	John	Doe	jdoe@example.com	2023-01-01 10:00
  2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00
  3	Alice	Green	alice.green@example.com	2023-01-02 10:00
  4	Joe	Bloggs	joe.bloggs@example.com	2023-01-02 10:00

  SCD2 Scenario

  >

  customer_id	first_name	last_name	email	_START_AT	_END_AT
  1	John	Doe	jdoe@example.com	2023-01-02 10:00	NULL
  1	John	Doe	john.doe@example.com	2023-01-01 10:00	2023-01-02 10:00
  2	Jane	Smith	jane.smith@example.com	2023-01-01 10:00	2023-01-02 10:00
  3	Alice	Green	alice.green@example.com	2023-01-01 10:00	NULL
  4	Joe	Bloggs	joe.bloggs@example.com	2023-01-01 10:00	NULL