Set up your environment

Prepare a local development environment for contributing to the Lakeflow Framework — clone the repo, install dependencies, and configure your editor.

For the full contribution path after setup, see Contribution workflow. For import rules, see Import conventions. For documentation lockfiles and make html, see Write & build docs.

Prerequisites

Before you begin, verify:

  • Git installed

  • Python 3.10+ installed (see requires-python in pyproject.toml)

  • pip available in your shell

  • IDE (optional) — VS Code or Cursor recommended for Data Flow Spec IntelliSense and yapf formatting

Step 1 — Clone the repository

Open a terminal and clone the framework repository:

git clone https://github.com/databricks-solutions/lakeflow_framework.git
cd lakeflow_framework

Step 3 — Install dev dependencies

The dev dependencies are pinned and hash-verified in requirements-dev.lock (generated from requirements-dev.txt). Installing from the lockfile guarantees a reproducible environment that matches CI.

From the repository root:

pip install --require-hashes --no-deps -r requirements-dev.lock

requirements-dev.lock includes everything in requirements-docs.lock, so you do not need a separate install step for building documentation.

Note

--require-hashes makes pip verify each package against the lockfile. --no-deps is safe here because the lockfile already contains the full resolved dependency set.

Step 4 — Optional: editable install for IDE and pytest

For a lighter setup that resolves imports without locking all transitive hashes (useful for IDE auto-complete and local pytest runs):

pip install -e ".[contrib]"

This installs the framework in editable mode from pyproject.toml, including the [contrib] extra. Use pip install -e ".[all]" to pull in future contrib sub-extras as they land.

Build a distribution wheel at any time with:

python -m build

Step 5 — Set up VS Code extensions

Open the Lakeflow Framework workspace in VS Code (or Cursor). On first open, the editor prompts you to install recommended workspace extensions.

If you missed the prompt:

  • Run Extensions: Show Recommended Extensions, or

  • Open the Extensions view and select Workspace Recommendations

Install the recommended extensions — including yapf for Python formatting used in Contribution workflow.

Step 6 — Verify the setup

From the repository root, confirm unit tests run:

pytest tests/ -m "not integration and not spark"

See tests/README.md for layout, markers, and conventions.

When you change dependencies

If you add, remove, or bump a dependency in any requirements*.txt file, regenerate all lockfiles from the repo root:

./scripts/generate_lockfiles.sh

Commit the regenerated .lock files with your dependency change. See Write & build docs for documentation lockfile details.

Note

To deploy the framework to a Databricks workspace for integration testing, follow Deploy framework from local machine or Quick Start.

See also