logo
Lakeflow Framework
Metadata-Driven Development
Initializing search
    lakeflow_framework
    • Home
    • Get Started
    • Architecture
    • Samples
    • Build
    • Deploy
    • Features
    • Contributors
    lakeflow_framework
    • Home
    • Get Started
      • What is the Lakeflow Framework?
      • Quick Start
    • Architecture
    • Samples
    • Build
      • Bundle Scope and Structure
      • Building a Pipeline Bundle
      • Data Flow Spec Reference
        • Creating a Standard Data Flow Spec Reference
        • Creating a Flows Data Flow Spec Reference
        • Creating a Materialized View Data Flow Spec Reference
      • Data Flows & Pipeline Patterns
        • Base Patterns
          • Pattern - Basic 1:1
          • Pattern - Gold Materialized Views
        • Advanced composition
          • Pattern - Multi-Source Streaming
          • Pattern - Stream-Static - Basic
          • Pattern - Stream-Static - Streaming Data Warehouse
          • Pattern - CDC Stream from Snapshots
        • Scaling and decomposing pipelines
        • Mix and Match Patterns
    • Deploy
      • Before you deploy
      • Deploy the Framework
        • Deployment options
        • Deploy from local machine
        • Installing as a wheel
      • Deploy Pipeline Bundles
      • Setting up CI/CD
    • Features
      • Metadata-Driven Development
        • Data Flow Specification Format
        • Schema Management
        • Substitutions
        • Templates
        • Validation
      • Authoring and Tooling
        • Auto Complete / IntelliSense
        • UI Integration
      • Configuration Management
        • Framework configuration
        • Mandatory Table Properties
        • Spark Configuration
        • Operational Metadata
        • Secrets Management
        • Logging
        • Builder Parallelization
      • Supported Sources and Targets
        • Supported Source Types
        • Supported Target Types
        • SQL Source
      • Key Databricks Features
        • Change Data Capture (CDC)
        • Change Data Feed (CDF)
        • Multi-Source Streaming (Flows)
        • Liquid Clustering
        • Materialized Views
        • Schema-related Databricks Features
        • Soft Deletes
        • Target Catalog and Schema
      • Python Development and Extensibility
        • Python Code, Libraries & Init Scripts
        • Python Function Transforms
        • Python Dependency Management
        • Python Source
      • Data Quality
        • Data Quality - Expectations
        • Data Quality - Quarantine
      • Environments and Versioning
        • Logical Environments
        • Versioning - DataFlow Specs
        • Versioning - Framework
      • Migrations
        • Table Migration
      • Features A–Z
    • Contributors
      • Set up your environment
      • Branching, versioning & releases
      • Contribution workflow
      • Import conventions
      • Write & build docs
      • Contributing to contrib

    Metadata-Driven Development¶

    Spec-as-code capabilities native to Lakeflow Framework — how you express pipelines, manage schemas, reuse patterns, parameterize values, and validate specs before deploy.

    Substitutions are the parameterization mechanism in the data flow spec (similar to bundle variable substitution). Their primary use is environment-specific values; see also Logical Environments.

    • Data Flow Specification Format
    • Schema Management
    • Substitutions
    • Templates
    • Validation
    © Copyright 2026, Databricks.
    Created using Sphinx 9.0.4. and Sphinx-Immaterial