Framework configuration

Applies To:

Framework Bundle

Configuration Scope:

Global

Databricks Docs:

NA

Framework-level settings (global JSON/YAML, substitutions, secrets, spec mappings, operational metadata) live under src/config/default/. Individual values can be overridden per-deployment using sparse files in src/local/config/ — only the keys you want to change are needed.

Configuration

Scope: Global (framework bundle)
Default: src/config/default/ — authoritative, always read.
Override: src/local/config/ — sparse override; deep-merged on top of defaults.

Under src/config/default/ you normally have:

Mandatory

  • Global file: exactly one of global.json, global.yaml, global.yml. More than one is an error.

  • Mappings: the dataflow_spec_mapping/ directory must exist.

Optional

Inside the global file, all top-level keys are optional. Common ones:

Key

See

pipeline_bundle_spec_format

Data Flow Specification Format

mandatory_table_properties

Mandatory Table Properties

spark_config

Spark Configuration

table_migration_state_volume_path

Table Migration

dataflow_spec_version

Versioning - DataFlow Specs

override_max_workers / pipeline_builder_disable_threading

Builder Parallelization

Local override (src/local/config/)

Place sparse JSON/YAML files in src/local/config/ to override individual keys without copying the entire default file. The framework deep-merges the overlay on top of the defaults at runtime:

  • Dict values are merged recursively — only the keys present in the overlay are changed.

  • Non-dict values and lists are replaced wholesale.

  • Keys not present in the overlay retain their default values.

Example — change one global setting without touching the rest of global.json:

{
  "dataflow_spec_version": "0.0.3"
}

Save this as src/local/config/global.json. All other keys from src/config/default/global.json are kept unchanged.

For directory-based config (e.g. dataflow_spec_mapping/), place the entire override directory in src/local/config/ — the local directory takes full precedence over the default.

See src/local/config/README.md in the framework bundle for the full list of supported files and migration instructions.

Deprecated since version v0.14.0:

The config/override/ mechanism (whole-tree replacement) is deprecated as of v0.14.0 and will be removed in v1.0.0. Migrate to src/local/config/ sparse files instead.

Migration steps:

  1. Identify which keys in your config/override/ files differ from config/default/.

  2. Create sparse files in src/local/config/ containing only those keys.

  3. Remove all files from config/override/ (leave the .gitkeep).

A DeprecationWarning is emitted at pipeline startup when config/override/ contains non-hidden files.