Target Details Reference

The target details object specifies how and where data should be written in your dataflow. This section documents the configuration options available for different target formats.

Delta Target Details

When using Delta format as your target, the following properties are available:

Field	Type	Description
table	`string`	The name of the target table.
database (optional)	`string`	The database name for the target table. If not specified, the default database will be used.
tableProperties (optional)	`object`	A map of Delta table properties to set on the target table. Common properties include: `delta.autoOptimize.optimizeWrite` `delta.autoOptimize.autoCompact` `delta.enableChangeDataFeed`
partitionColumns (optional)	`array[string]`	List of columns to partition the table by.
clusterBy (optional)	`array[string]`	List of columns to cluster the table by.
clusterByAuto (optional)	`boolean`	When true, the clustering keys will be automatically selected based on the data in the table.
schemaPath (optional)	`string`	Path to a schema file that defines the expected structure of the target table.
mergeSchema (optional)	`boolean`	When true, allows the schema to be updated when new columns are present in the source.
overwriteSchema (optional)	`boolean`	When true, allows complete replacement of the existing schema with a new one.
comment	`string`	A description for the materialized view.
spark_conf (optional)	`object`	A list of Spark configurations for the execution of this query.

Delta Sink Target Details

When using a Delta Sink as a target, the following properties are available:

Field	Type	Description
name	`string`	The name of the Delta Sink.
sinkOptions	`object`	The options for the Delta Sink.

Delta sinkOptions

Please refer to the Databricks documentation for the most up to date information on the Delta Sink: https://docs.databricks.com/en/dlt/dlt-sinks

You must specify one of the below properties:

Field	Type	Description
tableName	`string`	The fully qualified name of the Delta table to write to. Three level namespace for UC e.g. catalog_name.schema_name.table_name
path	`string`	The path to the Delta table to write to e.g. /Volumes/catalog_name/schema_name/volume_name/path/to/data

Kafka Sink Target Details

When using a Kafka Sink as a target, the following properties are available:

Field	Type	Description
name	`string`	The name of the Kafka topic to write records to.
sinkOptions	`object`	Kafka configuration properties as key-value pairs.

Kafka sinkOptions

The kafkaOptions object contains the following properties:

Option	Type	Description
kafka.bootstrap.servers	`string`	The Kafka bootstrap servers connection string
kafka.group.id	`string`	The consumer group ID
kafka.security.protocol	`string`	Security protocol to use (defaults to SASL_SSL)
kafka.ssl.truststore.location	`string`	Location of the SSL truststore file
kafka.ssl.truststore.password.accessKeyName	`string`	Access key name for the truststore password
kafka.ssl.truststore.password.secretScopeName	`string`	Secret scope name containing the truststore password
kafka.ssl.keystore.location	`string`	Location of the SSL keystore file
kafka.ssl.keystore.password.accessKeyName	`string`	Access key name for the keystore password
kafka.ssl.keystore.password.secretScopeName	`string`	Secret scope name containing the keystore password

Foreach Batch Sink Target Details

When using a Foreach Batch Sink as a target, the following properties are available:

Field	Type	Description
name	`string`	The name of the Foreach Batch Sink.
type	`string`	The type of the Foreach Batch Sink. Supported values: [“basic_sql”, “python_function”]
config	`object`	The configuration for the Foreach Batch Sink type.

Foreach Batch basic_sql config

The basic_sql object contains the following properties:

Field	Type	Description
database	`string`	The database name for the target table. If not specified, the default database will be used.
table	`string`	The name of the target table.
sqlPath	`string`	The path to the SQL file to execute.
sqlStatement	`string`	The SQL statement to execute. This is an alternative to sqlPath and is mutually exclusive with it.
partitionBy (optional)	`array[string]`	List of columns to partition the table by.
clusterBy (optional)	`array[string]`	List of columns to cluster the table by.
tableProperties (optional)	`object`	A map of Delta table properties to set on the target table.

Note

The SELECT statement specified via the sqlPath or sqlStatement property must:

reference micro_batch_view as the source table in the FROM clause of the query that retrieves data from the sourve view.
be a batch query i.e. do not wrap the micro_batch_view in a STREAM() function.

Basic example:

SELECT
  *
FROM micro_batch_view

Subquery Example:

SELECT
  *
FROM (

  SELECT
    *
  FROM micro_batch_view
)

CTE Example:

WITH source_cte AS (
  SELECT
    *
  FROM micro_batch_view
)

SELECT
  *
FROM source_cte

Foreach Batch python_function config

The python_function config object contains the following properties:

Field	Type	Description
functionPath (optional)	`string`	The path to the Python file, which should live in the python_functions subdirectory.
module (optional)	`string`	The module to import the Python function from.
tokens (optional)	`object`	A map of tokens to pass to the Python function.

Important

You must select one of functionPath or module.