Usage Guide

Export pipeline templates as JSON

In the Azure Data Factory portal, open Manage → ARM template → Export ARM template. This should create a directory with the following structure:

adf-export/
├── pipeline/        # one JSON file per pipeline
├── dataset/         # one JSON file per dataset
├── linkedService/   # one JSON file per linked service
└── trigger/         # one JSON file per trigger (optional)

Upload the exported pipelines

Upload the exported pipelines to a Unity Catalog volume, workspace folder, or local path:

databricks fs cp -r ./adf-export dbfs:/Volumes/main/default/adf_export

Run the end-to-end migration

Prompt your agent with the path to your JSON templates and a target output directory:

flowx migrate pipelines in `/Volumes/main/default/adf_export`. Save output in `./flowx_output/`.

Your agent will use flowx's migrate skill to run an end-to-end migration consisting of several phases:

1. discover parses JSON files, builds an inventory, assigns a translation strategy for each resource, and creates a complexity report.
2. convert converts each activity to an intermediate representation. The agent will ask for confirmation before running any LLM-based translation.
3. package creates a Declarative Automation Bundle with job configuration, code, and setup scripts (e.g. to create Databricks Secret Scopes or Unity Catalog connections).

Review the output

flowx writes all artifacts into a shared output folder (./flowx_output by default) with the following structure:

flowx_output/
├── databricks.yml          # Bundle configuration (from package)
├── resources/              # Job configuration (from package)
├── src/                    # Code required to run the pipeline (from package)
├── SETUP.md                # Setup instructions (from package)
├── metadata/
│   ├── inventory.json      # discover: activity inventory
│   ├── profile_report.csv  # profile: per-pipeline complexity report
│   ├── <pipeline>.arm.json # discover: verbatim original ADF/ARM pipeline source
│   └── configuration.json  # modify: the collected configuration answers
└── .work/                  # transient intermediates (translation report, IR, gaps.json); pruned by prepare

The bundle contains:

1. databricks.yml file with deployment targets, global parameters, and other variables
2. resources/ folder with job and pipeline configuration
3. src/ folder with code required to run the pipeline
4. SETUP.md file that details any deployment pre-requisites

When running with workspace auth (e.g. Genie Code), package can optionally persist this run's coverage to a Unity Catalog table — one row per pipeline stamped with a UUID run_id, run_date, and run_by (record-results) — and install a published AI/BI coverage dashboard over that table (install-dashboard). See Configuration options for details.

For Airflow, activities is the independent source-audit count rather than the number of tasks the translator happened to emit. Reporting distinguishes deterministic, agentic, failed, and excluded candidates and carries reconciliation status, translation-path coverage, deterministic coverage, unresolved agentic outcomes, and mechanically validated code-attached coverage. Code attachment is not a certification that provider-authored code is semantically correct.

Deploy the bundle

To deploy translated pipelines to your Databricks workspace, run the following command from the root of the bundle directory:

databricks bundle validate
databricks bundle deploy --target <your deployment target>

Packaging modes

A multi-pipeline migration can be laid out into bundles in three ways, selected with --packaging-mode on package:

When a pipeline calls another (ExecutePipeline) that lands in a different bundle, the caller's run_job_task job id is rewritten to a bundle variable (${var.<callee>_job_id}) and wired at deploy time. A top-level DEPLOY.md lists every bundle, its cross-bundle dependencies, and the suggested callees-first deploy order.

Deploying multiple bundles in order

For a multi-bundle migration, the ordered deployer resolves the cross-bundle job ids for you rather than deploying each bundle by hand:

python -m flowx.adapter deploy --output-dir ./flowx_output --target <your deployment target>

It discovers the bundles, deploys callees before callers, reads each deployed job's numeric id from databricks bundle summary, and injects it into callers via --var <callee>_job_id=<id>. Add --dry-run to print the deploy order without deploying.