Usage Guide

This guide walks through an end-to-end conversion of an Azure Data Factory pipeline to Lakeflow Jobs. You will install the flowx skills in your agentic tool, hand it a directory of pipeline templates for conversion, and create a Declarative Automation Bundle you can use to deploy a Databricks Lakeflow Job.

Export pipeline templates as JSON

In the Azure Data Factory portal, open Manage → ARM template → Export ARM template. This should create a directory with the following structure:

text

adf-export/
├── pipeline/        # one JSON file per pipeline
├── dataset/         # one JSON file per dataset
├── linkedService/   # one JSON file per linked service
└── trigger/         # one JSON file per trigger (optional)

Exporting pipelines using Powershell

You can also use the Get-AzDataFactoryV2Pipeline PowerShell cmdlet to programmatically dump each pipeline definition as JSON.

Upload the exported pipelines

Upload the exported pipelines to a Unity Catalog volume, workspace folder, or local path:

bash

databricks fs cp -r ./adf-export dbfs:/Volumes/main/default/adf_export

Run the end-to-end migration

Prompt your agent with the path to your JSON templates and a target output directory:

text

flowx migrate pipelines in `/Volumes/main/default/adf_export`. Save output in `./flowx_output/`.

Your agent will use flowx's migrate skill to run an end-to-end migration consisting of several phases:

discover parses JSON files, builds an inventory, assigns a translation strategy for each resource, and creates a complexity report.
convert converts each activity to an intermediate representation. The agent will ask for confirmation before running any LLM-based translation.
package creates a Declarative Automation Bundle with job configuration, code, and setup scripts (e.g. to create Databricks Secret Scopes or Unity Catalog connections).

Running migration step-by-step

To run a single step, invoke its skill directly (e.g. @flowx-discover) or ask your agent to run a single step.

Review the output

flowx writes all artifacts into a shared output folder (./flowx_output by default) with the following structure:

text

flowx_output/
├── databricks.yml          # Bundle configuration (from package)
├── resources/              # Job configuration (from package)
├── src/                    # Code required to run the pipeline (from package)
├── SETUP.md                # Setup instructions (from package)
├── metadata/
│   ├── inventory.json      # discover: activity inventory
│   ├── profile_report.csv  # profile: per-pipeline complexity report
│   ├── <pipeline>.arm.json # discover: verbatim original ADF/ARM pipeline source
│   └── configuration.json  # modify: the collected configuration answers
└── .work/                  # transient intermediates (translation report, IR, gaps.json); pruned by prepare

The bundle contains:

databricks.yml file with deployment targets, global parameters, and other variables
resources/ folder with job and pipeline configuration
src/ folder with code required to run the pipeline
SETUP.md file that details any deployment pre-requisites

Secret handling

Connection strings, credentials, and other protected configuration parameters are emitted as SecretInstruction steps that require Databricks Secrets. Run the setup scripts to add any required secret values before deploying and running pipelines in your workspace.

When running with workspace auth (e.g. Genie Code), package can optionally persist this run's coverage to a Unity Catalog table — one row per pipeline stamped with a UUID run_id, run_date, and run_by (record-results) — and install a published AI/BI coverage dashboard over that table (install-dashboard). See Configuration options for details.

For Airflow, activities is the independent source-audit count rather than the number of tasks the translator happened to emit. Reporting distinguishes deterministic, agentic, failed, and excluded candidates and carries reconciliation status, translation-path coverage, deterministic coverage, unresolved agentic outcomes, and mechanically validated code-attached coverage. Code attachment is not a certification that provider-authored code is semantically correct.

Deploy the bundle

To deploy translated pipelines to your Databricks workspace, run the following command from the root of the bundle directory:

bash

databricks bundle validate
databricks bundle deploy --target <your deployment target>

Deploying across environments

Bundles created by flowx are standard Declarative Automation Bundles. You can target different environments, integrate with CI/CD, or further customize the YAML before deploying. See the Declarative Automation Bundles documentation for more information.

Export pipeline templates as JSON

Upload the exported pipelines

Run the end-to-end migration

Review the output

Deploy the bundle

On this page