Getting Started

The following section is a quick start guide on how to get started with the Lakeflow Framework as a data engineer.

If you are evaluating fit first, start with What is the Lakeflow Framework?.

Prerequisites

  1. Databricks CLI installed and configured, if you are using DABs to locally deploy the Lakeflow Framework and Pipeline Bundles.

  2. Access to a Databricks workspace.

  3. VSCode installed.

Setup

Follow the below steps to get yourself setup to learn and use the Lakeflow Framework:

  1. Deploy the Framework

  2. The Samples

  3. Auto Complete / IntelliSense

Understanding the Framework

The framework supports both centralized and domain-oriented delivery models; use Framework Concepts for operating model guidance and Data Flow and Pipeline Patterns to select the right implementation pattern for your modelling approach.

  1. Framework Concepts

  2. Step through the feature-samples bundle — run the feature_samples_run_job and inspect the resulting tables in the {namespace}_feature schema. This is the simplest entry point as all features share a single schema.

  3. Framework Features

Developing your first Pipeline Bundle

  1. Select from one of the recommended pipeline patterns that best fits your use case, as documented in Data Flow and Pipeline Patterns

  2. Build and deploy a data pipeline bundle. Refer to Build and Deploy Pipelines.