Getting Started

The following section is a quick start guide on how to get started with the Lakeflow Framework as a data engineer.

Pre-Requisites

  1. Databricks CLI installed and configured, if you are using DABs to locally deploy the Lakeflow Framework and Pipeline Bundles.

  2. Access to a Databricks workspace.

  3. VSCode installed.

Setup

Follow the below steps to get yourself setup to learn and use the Lakeflow Framework:

  1. Deploy the Framework

  2. The Samples

  3. Auto Complete / Intellisense

Understanding the Framework

  1. Framework Concepts

  2. Step through and execute one of the basic samples and inspect the create_dataflow_spec

  3. Framework Features

Developing your first Pipeline Bundle

  1. Select from one of the recommended pipeline patterns that best fits your use case, as documented in Data Flow and Pipeline Patterns

  2. Build and deploy a data pipeline bundle. Refer to Build and Deploy Pipelines.