Linking Entra ID Groups to Azure Databricks Without SCIM – Using Terraform and the Public Preview Feature

Michał Pawlikowski

A common challenge for teams integrating Azure Databricks with Microsoft Entra ID (formerly Azure AD) is how to synchronize groups efficiently. While SCIM is typically used for this, there’s now a much simpler way – thanks to a public preview feature that allows you to directly reference Entra ID groups without any SCIM mechanism.

In this blog post, I’ll show you how to:

  • Enable and use this feature in Azure Databricks.

  • Add Entra ID groups to your Databricks account using Terraform.

  • Retrieve the group’s object_id via the azuread Terraform provider.

Let’s walk through the process step-by-step.

Background: Entra ID Groups in Databricks

Databricks now supports external groups via a new public preview feature. When this is enabled, you can define a Databricks group that links directly to an Entra ID group by passing its object_id as the external_id.

This approach is:

  • Clean – No need for SCIM or sync connectors.

  • Declarative – Easily done in Terraform.

  • Traceable – Group identity is linked through a stable object_id.

⚠️ This feature must be explicitly enabled in your Databricks account. At the time of writing, it is available in public preview. See here: https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/automatic-identity-management.

Step 1 – Setup Your Terraform Providers

We’ll need three providers:

  • azurerm to authenticate into Azure (The azurerm provider is typically used to authenticate into Azure or manage Azure resources, but it is not necessary for this setup, since we are only accessing Entra ID and Databricks)

  • azuread to query the Entra ID group,

  • databricks to create the external group.

Save the following as your providers.tf file:

JavaScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
terraform {
    required_providers {
        azurerm = {
            source = "hashicorp/azurerm"
            version = "~> 3.0"
        }
        databricks = {
            source = "databricks/databricks"
            version = ">= 1.34.0"
        }
        azuread = {
            source = "hashicorp/azuread"
            version = "~> 3.3.0"
        }
    }
 
    required_version = ">= 1.3.0"
}
 
provider "azurerm" {
    features {}
    subscription_id = "xxxx-xxxx-xxxx-xxxx"
    tenant_id = "yyyy-yyyy-yyyy-yyyy"
}
 
provider "databricks" {
    alias = "account"
    host = "https://accounts.azuredatabricks.net"
    account_id = "zzzz-zzzz-zzzz-zzzz"
}
 
provider "databricks" {
    alias = "workspace"
    host = "https://adb-xxxxxxxxxxxxxxxx.azuredatabricks.net"
}
 

Step 2 – Reference an Existing Entra ID Group (entra_groups.tf)

Use the azuread_group data source to locate your group by display name. This assumes the group already exists in Entra ID.

JavaScript
1
2
3
4
5
6
7
8
9
10
11
12
data "azuread_group"
"databricks_group1" {
    display_name = "Databricks_Group1"
    security_enabled = true
}
 
output "databricks_group1_object_id" {
    value = data.azuread_group.databricks_group1.object_id
    description = "Object ID of the Databricks_Group1 Entra ID group"
}
# Note: Output for informational purposes only – shows the Entra ID group's object_id
# which can be used for debugging or manual reference if needed

This fetches the group and outputs its databricks_group1.object_id, which we’ll need for the next step.

This how my group looks in Entra ID:

️ Step 3 – Create the Databricks External Group (db_external_groups.tf)

Now, we link the Entra ID group into the Databricks account-level using its object_id:

JavaScript
1
2
3
4
5
6
7
8
9
10
11
resource "databricks_group"
"databricks_group1_external" {
    provider = databricks.account
    display_name = data.azuread_group.databricks_group1.display_name
    external_id = data.azuread_group.databricks_group1.object_id
}
 
output "databricks_group1_external_id" {
    value = databricks_group.databricks_group1_external.id
    description = "ID of the Databricks external group linked to Entra ID group"
}
 

This resource creates an external group in your Databricks account, not just in a workspace. Make sure you are using the account-level Databricks provider.

Now run terraform apply, you will get:

The new external group has been successfully created in your Databricks account, and all its members are automatically included.

✅ Result: A Direct Link, No SCIM Required

With the above configuration:

  • The group is visible in the Databricks Admin Console and of course all Workspaces under your account.

  • Membership is automatically pulled from Entra ID based on the external_id.

  • You don’t need to maintain a SCIM sync connector.

This approach is ideal for centralized identity governance, especially in larger organizations that want to use Terraform to manage Databricks configuration declaratively.

Security and Maintenance Notes

  • Always ensure the Terraform Databricks account user / service principal has the required permissions to view and link groups.

  • When removing the group from Terraform, Databricks will not delete the Entra ID group – only its Databricks reference.

Summary

You now know how to:

  • Use a new public preview feature in Databricks to link Entra ID groups.

  • Implement this using Terraform, without SCIM.

  • Fetch group IDs directly from Entra ID via the azuread provider.

This gives you a clean, secure and modern way of managing access in Databricks environments.

Leave a Reply