Lakeflow Framework documentation
The Lakeflow Framework is a metadata-driven data engineering framework built for Databricks Lakeflow Spark Declarative Pipelines (SDP). It accelerates and simplifies deployment through a configuration-driven, pattern-based approach that supports both centralized and domain-oriented operating models across the medallion architecture.
Start with What is the Lakeflow Framework? for the full overview, then use the sections below to get hands-on.
Contents:
- What is the Lakeflow Framework?
- Getting Started
- Concepts
- Features
- Auto Complete / IntelliSense
- Builder Parallelization
- Change Data Capture (CDC)
- Change Data Feed (CDF)
- Data Quality - Expectations
- Data Quality - Quarantine
- Direct Publishing Mode
- Framework configuration
- Liquid Clustering
- Logging
- Logical Environments
- Materialized Views
- Mandatory Table Properties
- Multi-Source Streaming
- Operational Metadata
- Python Dependency Management
- Python Code, Libraries & Init Scripts
- Python Function Transforms
- Python Source
- Schema Definitions
- Data Flow Specification Format
- Secrets Management
- Soft Deletes
- Source Types
- Spark Configuration
- Substitutions
- Table Migration
- Target Types
- Templates
- Validation
- Versioning - DataFlow Specs
- Versioning - Framework
- UI Integration
- Deploy the Framework
- The Samples
- Build and Deploy Pipelines
- Bundle Scope and Structure
- Building a Pipeline Bundle
- Deploying a Pipeline Bundle
- Pipeline Execution
- Patterns: Data Flows and Pipelines
- Data Flow Spec Reference
- Creating a Standard Data Flow Spec Reference
- Creating a Flows Data Flow Spec Reference
- Creating a Materialized View Data Flow Spec Reference
- Orchestration
- Framework Development & Contributors