Lakeflow Framework documentation

Add your content using reStructuredText syntax. See the reStructuredText documentation for details.

Contents:

• Introduction
  • Core Concepts
• Getting Started
  • Pre-Requisites
  • Setup
  • Understanding the Framework
  • Developing your first Pipeline Bundle
• Concepts
  • Databricks Asset Bundles (DABs)
  • Bundle Types
    • Framework Bundle
      • Framework Configuration
    • Pipeline Bundles
      • Data Flow Specs
      • Pipeline Resource YAML Files
      • Pipeline Bundle Configuration
  • Pipelines
  • Data Flows & Data Flow Specs
    • Standard Data Flow
    • Flows Data Flow
      • Flow Groups Explained
      • Staging Tables Explained
      • Flows Explained
    • Materialized Views
    • Patterns
• Features
  • Auto Complete / Intellisense
    • Overview
      • Autocompletion Example
      • Validation Example
    • Configuration for VS Code
  • Builder Parallelization
    • Overview
    • Parameters
    • Configuration
      • Global Configuration
    • Troubleshooting
  • Change Data Capture (CDC)
    • Use of AUTO CDC FROM SNAPSHOT
    • Configuration
  • Change Data Feed (CDF)
    • Overview
    • Configuration
      • Enabling CDF on a Table
      • Reading From CDF in a View
    • Important Considerations:
  • Data Quality - Expectations
    • Configuration
      • Defining Expectations
      • Enabling and Referencing Expectations
    • Expectations Schema
    • Examples
  • Data Quality - Quarantine
    • Configuration
  • Direct Publishing Mode
  • Liquid Clustering
    • Use Cases
    • Selecting Clustering Keys
    • Configuration
  • Logging
    • Log Levels
    • Configuration
      • Setting the Log Level in the Pipeline Yaml
      • Setting the Log Level in the Databricks UI
      • Permissions to View Logs
    • Viewing the Logs
    • Example Log Messages
  • Logical Environments
    • Configuration
      • 1. databricks.yml Configuration
      • 2. pipeline resource YAML files Configuration
    • Passing the logical environment
      • 1. Environment Variable
      • 2. Command Line Argument
  • Materialized Views
    • Overview
    • Sample Bundle
    • Data Flow Spec Configuration
      • Data Flow Spec Configuration Schema
      • Source Type Details
      • Additional Configuration Options
    • Example Configuration
  • Mandatory Table Properties
    • Configuration
    • Configuration Schema
    • Common Properties
    • Example Configuration
  • Multi-Source Streaming
    • Configuration
    • Key Features
    • Important Considerations
    • See Also
  • Operational Metadata
    • Configuration
    • Configuration Schema
    • Best Practice Configuration
    • Examples
  • Python Dependency Management
    • Overview
    • Framework Dependencies
      • Location
      • Framework requirements.txt
    • Pipeline Bundle Dependencies
      • Configuring Pipeline Environment
      • Using a Requirements File
      • Inline Dependencies
      • Installing from Unity Catalog Volumes
    • Best Practices
      • Version Pinning
      • Documentation
      • Testing Dependencies Locally
    • Limitations
    • Troubleshooting
      • Dependencies Not Found
      • Version Conflicts
  • Python Extensions
    • Overview
    • How It Works
      • Directory Structure
    • Dependency Management
    • Extension Examples
      • Source Extensions
      • Transform Extensions
      • Sink Extensions
    • Additional Resources
  • Python Function Transforms
    • Overview
    • Sample Bundle
    • Configuration
      • Using Extensions
      • Using File Path
    • pythonTransform Schema
  • Schemas
    • Schema Types
    • Schema File Location
    • Data Flow Spec Configuration
    • StructType JSON Format
      • Generating the Schema Definition
      • For Example:
    • Text DDL Format
      • Examples
  • Data Flow Specification Format
    • Overview
    • Configuration
      • Framework-Level Configuration
      • Pipeline-Level Configuration
    • Supported File Types and Naming Conventions
      • JSON Format
      • YAML Format
    • Example Specification
    • Best Practices
    • Configuration Examples
      • Example 1: Framework Enforces JSON Format
      • Example 2: Framework Allows Format Flexibility
      • Example 3: Framework Defaults to YAML
    • Troubleshooting
      • Format Mismatch Errors
      • Override Not Permitted
      • Invalid Format Value
      • Validation Errors After Conversion
      • Mixed Format Detection
    • See Also
  • Secrets Management
    • Overview
    • Configuration
    • Configuration Schema
    • Referencing Secrets in Data Flow Specs
    • Best Practices
    • Troubleshooting
      • Secret Access Denied
      • Configuration Errors
  • Soft Deletes
    • Configuration
  • Source Types
    • Source Types
    • General Data Flow Spec Configuration
    • Detailed Source Type Configuration Details
      • Python Source
      • SQL Source
  • Spark Configuration
    • Configuration
    • Configuration Schema
    • Common Configuration Properties
    • Example Configuration
    • Advanced Usage
      • Dynamic Configuration
      • Performance Tuning
  • Substitutions
    • Configuration
    • Configuration Schema
    • Examples
  • Table Migration
    • General Concepts
    • Configuration
    • Migration with Auto Starting Versions
      • How it Works
    • Migration without Auto Starting Versions
  • Target Types
    • Target Types
    • General Data Flow Spec Configuration
    • Detailed Target Type Configuration Details
      • Materialized Views
  • Templates
    • Overview
    • How It Works
      • Anatomy of a Template Definition
      • Anatomy of a Template Dataflow Specification
      • Template Processing
    • Example Usage
      • Example: Basic File Source Ingestion Template
    • Parameter Types
    • Key Features
      • Python Function Path Search Priority
      • Error Handling
      • Validation
      • Unique Identifiers
    • Best Practices
      • Naming Conventions
      • Development and Testing
      • Maintainability
    • Limitations
  • Validation
    • Overview
    • How Validation Works
      • Ignoring Validation Errors
  • Versioning - DataFlow Specs
    • Overview
    • Mapping File Structure
    • Mapping Operations
      • Rename Operations
      • Move Operations
      • Delete Operations
      • Complete Example
    • Configure Global DataFlow Version
    • Configure Individual DataFlow Specification Version
    • Transformation Order
    • Best Practices
    • Version Management
    • Troubleshooting
  • Versioning - Framework
    • Overview
    • Deploy the framework with a specific version
    • Set framework version for pipelines
    • Best Practices
    • Version Management
  • UI Integration
    • Overview
    • Configuration to enable Pipeline Editor UI Integration
• Deploy the Framework
  • Deploying the Framework
    • Deploying From Your Local Machine
    • Deploying via CI/CD
  • Setting up CI/CD
    • Prerequisites and Assumptions
    • Main Steps in a CI/CD Pipeline
    • Example CI/CD bash script
  • Versioning - Framework
    • Overview
    • Deploy the framework with a specific version
    • Set framework version for pipelines
    • Best Practices
    • Version Management
• The Samples
  • Deploying the Samples
    • Interactive Deployment
    • Single Command line deployment:
  • Using the Samples
  • Destroying the Samples
  • TPC-H Sample
• Build and Deploy Pipelines
  • Bundle Scope and Structure
    • Bundle Scope
    • Bundle Structure
  • Building a Pipeline Bundle
    • Prerequisites
    • Steps to build a Pipeline Bundle
      • 1. Create a new Pipeline Bundle
      • 2. Update the databricks.yml File
      • 3. Select your Bundle Structure
      • 4. Select your Data Flow Specification Language / Format
      • 5. Setup your Substitutions Configuration
      • 6. Build your Data Flows
      • 7. Create your Pipeline Definitions
  • Deploying a Pipeline Bundle
    • Deploying From Your Local Machine
    • Deploying via CI/CD
  • Pipeline Execution
    • Useful Pipeline Settings
  • Patterns: Data Flows and Pipelines
    • Patterns Overview
    • Patterns Documentation
      • Pattern - Basic 1:1
      • Pattern - Multi-Source Streaming
      • Pattern - Stream-Static - Basic
      • Pattern - Stream-Static - Streaming Data Warehouse
      • Pattern - CDC Stream from Snapshots
    • Multi-Source Streaming and Flow Groups
    • Mix and Match
    • Scaling and Pipeline Scope
      • Pipeline Scope
      • Decomposing Pipelines
• Data Flow Spec Reference
  • Creating a Standard Data Flow Spec Reference
    • Example:
    • Dataflow Metadata Configuration
    • Source Configuration
    • Target Configuration
    • Change Data Capture (CDC) Configuration
      • cdcSettings
      • cdcSnapshotSettings
      • CDC Historical Snapshot Source Configuration
    • Data Quality and Quarantine Configuration
      • quarantineTargetDetails
    • Table Migration Configuration
      • tableMigrationDetails
  • Creating a Flows Data Flow Spec Reference
    • Example:
    • Dataflow Metadata Configuration
    • Target Configuration
    • Flow Group Configuration
      • Staging Table Configuration
      • Flow Configuration
      • Flow Details
      • View Configuration
    • Change Data Capture (CDC) Configuration
      • cdcSettings
      • cdcSnapshotSettings
      • CDC Historical Snapshot Source Configuration
    • Data Quality and Quarantine Configuration
      • quarantineTargetDetails
    • Table Migration Configuration
      • tableMigrationDetails
  • Creating a Materialized View Data Flow Spec Reference
    • Schema
    • Example:
    • Dataflow Metadata Configuration
    • Source Configuration
    • Table Configuration
    • Data Quality and Quarantine Configuration
      • quarantineTargetDetails
• Orchestration
• Framework Development & Contributors
  • Development Environment Setup
    • Setting up for development as a contributor to the Lakeflow Framework
  • GIT
    • Branching Strategy
      • Feature Branch Guidelines
      • Fix Branch Guidelines
    • Release Strategy
  • Development Steps
    • Issue Creation
    • Branch Management
    • Development Process
    • Pull Request Process
    • Post-Merge Steps
  • Updating the Documentation
    • Source Files
    • Writing Documentation
    • Styling
    • Generating the Documentation
      • Supported Formats
      • Dependencies
      • Building the Documentation
      • Clean the Documentation