Architecture

flowx translates data pipelines into Databricks Lakeflow Jobs packaged as Declarative Automation Bundles (DABs). A set of Python functions is exposed through two surfaces: agent skills invoked through a CLI and MCP tools called by an agent session.

Three-phase pipeline

ADF JSON ──▶ discover ──▶ convert ──▶ package ──▶ databricks.yml (DAB)
            (parse +     (activities    (IR → jobs,
             classify)    → IR)          notebooks, setup)

All three phases share one output_dir (default ./flowx_output):

Phase	Module	Reads	Writes
profile	parser/adf_loader.py	ADF JSON exports	metadata/inventory.json, metadata/profile_report.csv, metadata/<pipeline>.arm.json
translate	translator/engine.py	inventory + ADF source	.work/translation_report.json (transient IR)
prepare	bundler/dab_writer.py	translation report	databricks.yml, resources/, src/, setup/ (and prunes .work/)

Each activity is classified with a TranslationStrategy:

• DETERMINISTIC (in-process translators)
• AGENTIC (LLM-assisted gaps)
• UNSUPPORTED

The reporting layer (reporting/) can write per-run coverage to a Unity Catalog table and publish an AI/BI dashboard.

Two surfaces over one core

                    ┌─────────────────────────────┐
   Agent skills ───▶│  python -m flowx.adapter │───▶ phase modules (discover/
   (discover/...,   │  (unified CLI entry point)   │     convert/package) +
    setup, migrate) └─────────────────────────────┘     adapter operations
                                  ▲
                                  │ subprocess (same contract)
                    ┌─────────────────────────────┐
   MCP tool  ──────▶│  flowx.mcp (FastMCP)     │
   (Claude / Genie) │  1 tool, 12 commands →       │
                    │  adapter bridge              │
                    └─────────────────────────────┘

The unified flowx.adapter CLI is the single contract. Both surfaces go through it:

• Agent skills (skills/) shell out to python -m flowx.adapter <phase|op> … directly.
• MCP tool (src/flowx/mcp/) is a thin bridge: the single flowx(command, parameters) tool builds the same adapter arguments for the chosen command, runs them as a subprocess via flowx.mcp.runner, then reads back the JSON/CSV artifacts and returns a structured result. No phase logic is duplicated, so the tool surface can't drift from the tested CLI. A single tool also keeps flowx to one of the host's tool slots (e.g. Genie Code's 20-tool cap).

MCP tool layer

Module	Purpose
mcp/server.py	FastMCP server; registers tools and builds the stdio / streamable-HTTP apps
mcp/runner.py	Subprocess bridge to flowx.adapter with artifact summarizers (for running translation without the mcp dependency)
mcp/__main__.py	python -m flowx.mcp entry point (stdio default, --http for hosting)

The flowx tool's command selects the adapter operation: inputs, discover, convert, merge_agentic, inspect, apply_answers, materialize_lookup, workspace_paths, package, migrate, record_results, and install_dashboard (with parameters carrying that command's arguments).

Deployment topology

The MCP server runs in whichever transport fits the calling tool. This is chosen when the setup skill is invoked:

• Local / Claude Code: setup installs mcp + uvicorn + starlette into the venv; the server runs over stdio and is registered with the MCP client.
• Databricks Genie Code: setup runs app/deploy.sh, which stages a self-contained bundle (app entrypoint + a vendored copy of the pure-Python flowx source), syncs it to the workspace, and creates/deploys the mcp-flowx Databricks App. The app serves the MCP streamable-HTTP transport at <app-url>/mcp (health at /) via uvicorn → Starlette → FastMCP. To meet Genie Code's requirements the server runs stateless (stateless_http=True) and enables CORS (origins via FLOWX_ALLOWED_ORIGINS). You add it in Genie Code under Settings → MCP Servers → Add Server → Custom MCP server; Genie connects over OAuth, access is governed by the app's Databricks Apps permissions, and the app authenticates to the workspace as its own service principal.

  Local (Claude Code / other agents)        Databricks Genie Code
  ───────────────────────────────────       ─────────────────────────────────
  agent ◀── stdio ──▶ python -m              Genie ◀── HTTPS /mcp ──▶ Databricks App
                      flowx.mcp                                   (uvicorn → Starlette
                      (in the venv)                                    → FastMCP streamable-HTTP)
                                                                       app authenticates as its
                                                                       own service principal

See Installation for the exact commands and the app README for deployment details.