Data Engineers
write pipeline specs
Platform Team
owns the framework
Pipeline Bundles
resource.yml
dataflow.json
schema.json
dq.json
Framework Bundle
entry notebook
central shared config
validator
interpreter
DABs Deploy
DABs Deploy
(once)
Databricks Workspace
Workspace Files
resource.yml
defines the pipeline
dataflow specs
bundle path
framework code
framework path
Spark Declarative Pipeline
dataflow
schema
DQ rules
validator
interpreter
shared config
defines
filter, interpret, standardise
creates
reads
runs
references
framework