Azure ADLS
You'll create an access connector, grant it access to the storage account, wrap it in a Unity Catalog storage credential, and create an external location that points Databricks at an existing ADLS Gen2 container in ~20 min. Need a storage account and container first? See the bonus at the end.
Prereqs: Cloud Object Storage overview, Azure and Unity Catalog privileges listed below
Existing or new storage account
Unity Catalog reaches cloud storage through an external location — a named object that points at a path in an ADLS Gen2 container through a storage credential. Which storage account you point it at depends on your situation:
- A new storage account. Usually you create a fresh ADLS Gen2 storage account to hold your catalog's data — a clean place for Unity Catalog to write managed tables and volumes. Create the storage account and container first (see the bonus), then follow this walkthrough to wire it up.
- An existing storage account. When a storage account already holds data, it's usually because you need to ingest that data into Databricks. Point an external location at it so Unity Catalog can read (and optionally write) the data in place — no new storage needed.
Either way, the setup is the same: create the access connector, grant it access to the storage account, wrap it in a storage credential, then create the external location on top and verify it. This page assumes the storage account and container already exist; if they don't, start from the bonus.
The big picture
You want an ADLS Gen2 storage account Databricks can reach, wired up through an access connector, a Unity Catalog storage credential, and an external location. Azure has the most moving parts of the three clouds, so there are a few more steps here. If your workspace uses serverless private networking, the last part attaches a network connectivity configuration so serverless compute can reach the storage account over a private endpoint.
Permissions you need
Check these before you start. A missing Azure role is why Create is greyed out in the portal. A missing Unity Catalog privilege is why the storage credential or external location never saves.
The storage account must have hierarchical namespace enabled (ADLS Gen2). That option is only available at create time.
Azure
Assign these Azure RBAC roles on the resource group (or subscription) that will hold the access connector and storage account:
| Role | Why you need it |
|---|---|
| Storage Account Contributor or Contributor | Create the storage account. Change its Networking settings: public network access, firewall, VNet rules, and private endpoints. |
| User Access Administrator or Owner | Assign Storage Blob Data Contributor to the access connector. Contributor cannot grant Azure roles. |
| Network Contributor | Create private endpoints and attach them to a virtual network. Skip this if you leave public access on for this walkthrough. |
Contributor on the resource group covers creating the storage account and editing its network settings. It does not let you assign IAM roles. If you cannot open Access control (IAM) > Add role assignment, you are missing User Access Administrator or Owner.
Azure Databricks
You need these Unity Catalog privileges. Metastore admins and account admins already have them. Workspace admin is not enough on its own.
| Privilege | On | Why you need it |
|---|---|---|
CREATE STORAGE CREDENTIAL | Metastore | Create the credential that wraps the access connector. |
CREATE EXTERNAL LOCATION | Metastore | Create the path object. |
CREATE | The storage credential | Bind that credential to an external location. |
If you are not a metastore admin, ask one to grant your group:
GRANT CREATE STORAGE CREDENTIAL, CREATE EXTERNAL LOCATION ON METASTORE TO `your-group`;
Creating a network connectivity configuration in the account console also needs account admin.
Create an access connector
The access connector is the managed identity Databricks uses to authenticate to your storage account. Create it first, because everything else points back to it.
Create the access connector in the Azure portal and copy its Resource ID.
In the Azure portal, search for Access Connector for Azure Databricks and select it.
Click Create.
Fill in the Basics tab:
- Subscription: the subscription where the connector will be deployed.
- Resource group: choose or create a resource group (for example
starter-journey). - Name: a descriptive name (for example
starter-journey-ac). - Region: the same region as your Databricks workspace.
Click Review + create, then Create.
When the deployment finishes, click Go to resource and copy the Resource ID from the connector's overview page. You paste it into the storage credential later.
Reference: Connect to ADLS Gen2 using Unity Catalog (Databricks docs)
Add permissions to the storage account
The connector's managed identity has no access to the storage account yet. Grant it the Storage Blob Data Contributor role, or the connection test at the end fails. If you plan to use file events on the external location, grant the queue and event roles too.
Assign the Storage Blob Data Contributor role to the access connector on the storage account.
On the storage account, click Access control (IAM) in the left menu.
Click + Add > Add role assignment.
Search for Storage Blob Data Contributor, select it, and click Next.
In the Members section, set the assignment type to Managed identity, click + Select members, choose Access Connector for Azure Databricks, and select your connector.
Click Review + assign.
Repeat the steps above to add each of these roles to the same access connector:
- Storage Blob Data Contributor
- Storage Queue Data Contributor
- EventGrid EventSubscription Contributor
Ask Genie: storage credential and external locations
Open Databricks Assistant (Genie) in the workspace if you want the Unity Catalog SQL for the credential and the three environment locations, instead of (or before) the click-through below.
Create a Unity Catalog storage credential that uses Azure Managed Identity and my Access Connector for Azure Databricks. Show the CREATE STORAGE CREDENTIAL SQL and tell me which fields I need from the access connector Resource ID.
Create a storage credential in Databricks
The storage credential links Unity Catalog to the access connector you made earlier.
Create a storage credential that references the access connector's Resource ID.
In the Databricks workspace, go to Catalog > External Data > Credentials, then click Create credential.
Set the credential type to Storage credential, choose Azure Managed Identity as the authentication type, and give the credential a name.
Paste the access connector Resource ID you copied earlier into the Access connector ID field.
Click Create.
Create an external location
The external location is the Unity Catalog object that points at the container path through the storage credential.
Create three Unity Catalog external locations named dev_ext_loc = abfss://<container>@<storage_account>.dfs.core.windows.net/<directory>, qa_ext_loc = abfss://<container>@<storage_account>.dfs.core.windows.net/<directory>, and prod_ext_loc = abfss://<container>@<storage_account>.dfs.core.windows.net/<directory>. Use my existing Azure storage credential and ADLS Gen2 paths under the same container, one directory per environment (dev, qa, prod). Show the CREATE EXTERNAL LOCATION SQL for all three.
Create an external location and assign it to your workspace.
In Databricks, go to Catalog > External Data > External Locations and click Create external location. Fill in the form:
- External location name: a descriptive name (for example
dev-starter-journey-external-location). - Storage type: Azure Data Lake Storage.
- URL:
abfss://<container>@<storage_account>.dfs.core.windows.net/<directory> - Storage credential: the credential you created earlier.
Click Create.
The external location's Overview tab shows its URL and credential.
Open the Permissions tab and click Assign to workspaces.
Assign the external location to your workspace.
If this location holds source data you only read from, mark it read-only so a stray job cannot overwrite it. Follow Mark an external location as read-only.
Verify
Test the external location's connection to confirm everything is wired up. A green success status means the access connector, its storage account role, the storage credential, and the external location all line up.
Test the external location's connection from Unity Catalog and confirm every check passes.
In the Databricks workspace, open Catalog, then click Connect > External locations to open the External Data page.
On the External Locations tab, click the external location you created.
On the location's Overview tab, click Test connection in the top-right corner. Confirm the dialog reports a success status for the read, list, write, delete, path, and file events checks, then click Done.
Reference: Manage external locations (Databricks docs)
Configure network connectivity (NCC)
You only need this part when the storage account is set to private, meaning it is not open to the public internet. In that case, serverless compute cannot reach it over the network by default, so you attach a network connectivity configuration that gives serverless an Azure Private Link endpoint to read and write the storage account. If the storage account allows public network access, skip this part.
Create a network connectivity configuration, add private endpoint rules, approve them, and attach the NCC to your workspace.
In the account console, open Security.
Go to Network connectivity configuration and create an NCC in the same region as your workspace. Give it a name, choose the region, then click Add.
Before you add a private endpoint rule, grab the storage account's resource ID. In the Azure portal, open the storage account, go to Properties, and click JSON View to copy the resource ID.
Back on the NCC, open Private endpoint rules and click Add private endpoint rule. Paste the resource ID into Azure resource ID, set the Azure subresource type to dfs, then click Add.
Repeat the previous step to add a second rule for the blob subresource.
Once both private endpoints are created, approve the connections on the storage account's Networking > Private endpoint connections tab.
Attach the NCC to your workspace in its configuration, then click Save.
Reference: Configure private connectivity to Azure resources (Databricks docs)
Connect classic compute to the storage account
Like the NCC, this part is only needed when the storage account is private. The NCC covers serverless compute. Classic compute (clusters that run inside your workspace's virtual network) reaches a private storage account through a private endpoint you create on the storage account into that virtual network. Create two, one for the dfs subresource and one for blob.
Create dfs and blob private endpoints on the storage account into the workspace virtual network.
On the storage account, open Networking > Private endpoints and click Create private endpoint.
On the Basics tab, set the subscription and resource group, name the endpoint (for example dfs_pe), and choose the same region as your virtual network. Click Next: Resource.
On the Resource tab, set Target sub-resource to dfs. Click Next: Virtual Network.
On the Virtual Network tab, select your workspace's virtual network and the subnet reserved for private endpoints (for example private-endpoints). Click Next: DNS.
On the DNS tab, set Integrate with private DNS zone to Yes so the privatelink.dfs.core.windows.net zone resolves the endpoint. Click Next: Tags.
Review the configuration and click Create.
Back on the Private endpoints tab, the dfs_pe connection shows as approved.
Now create the second private endpoint for the blob subresource. Click Create private endpoint again and, on the Basics tab, name it (for example blob_pe) with the same region. Click Next: Resource.
On the Resource tab, set Target sub-resource to blob. Click Next: Virtual Network.
On the Virtual Network tab, select the same virtual network and private-endpoints subnet. Click Next: DNS.
On the DNS tab, keep Integrate with private DNS zone set to Yes for the privatelink.blob.core.windows.net zone. Click Next: Tags.
Review the configuration and click Create.
Reference: Enable Azure Private Link for a workspace (Databricks docs)
Where people trip
Test Connection fails with permission denied
The connector's managed identity is missing the Storage Blob Data Contributor role. Check the role assignment under the storage account's Access control (IAM) section in the Azure portal. This is the Add permissions to the storage account step, and it is the most common one to skip.
External location creation fails with invalid URL
The URL has to be abfss://<container>@<storage_account>.dfs.core.windows.net/<directory>. Confirm the container exists, the storage account name is right, and hierarchical namespace is on.
Storage credential creation fails
Recopy the access connector Resource ID from the Azure portal overview page. A truncated paste is the usual cause. The connector also has to be in the same region as the workspace.
Hierarchical namespace is not available on the storage account
Hierarchical namespace can only be enabled when the storage account is created. Create a new storage account with the option checked and repeat the steps against it.
Private endpoint stays in Pending
A serverless private endpoint rule is not active until you approve the connection on the storage account. Open the storage account's Networking > Private endpoint connections tab in the Azure portal and approve the pending request.
Bonus: Create a storage account and container
If you don't already have an ADLS Gen2 storage account and container, create them here first, then go back to Create an access connector and follow the walkthrough against them.
Create a storage account
Create an ADLS Gen2 storage account with hierarchical namespace enabled.
In the Azure portal, search for Storage account and select it.
Click Create.
Fill in the Basics tab:
- Subscription: the same subscription as your Databricks workspace.
- Resource group: the same resource group as your workspace resources.
- Storage account name: globally unique, lowercase letters and numbers only.
- Region: the same region as your Databricks workspace.
- Primary service: Azure Blob Storage or Azure Data Lake Storage Gen2.
- Performance: Standard is enough for most cases.
- Redundancy: choose based on your durability needs (LRS, ZRS, GRS, or GZRS).
On the Advanced tab, check Enable hierarchical namespace and set the access tier to Hot.
Hierarchical namespace has to be enabled at creation time. You cannot turn it on for an existing storage account.
Click Review + create, then Create.
When the deployment finishes, click Go to resource to open the new storage account.
Create a container
Unity Catalog reads and writes under a directory in a container, so create one now.
Create a container in the storage account and add a directory for Unity Catalog.
On the storage account, open Containers from the left menu.
Click + Container, name it (for example dev) and click Create.
The new container appears in the containers list.
Open the container and click Add Directory. Name it (for example unity_catalog_dev) and click Save.
The directory now shows up inside the container.
Reference: Connect to ADLS Gen2 using Unity Catalog (Databricks docs)
Next
- Do next: Managed connectors
- Learn why: Unity Catalog foundations
- Reference: Connect to ADLS Gen2 using Unity Catalog