Azure, Azure Active Directory, Azure Databricks

Linking Entra ID Groups to Azure Databricks Without SCIM – Using Terraform and the Public Preview Feature

A common challenge for teams integrating Azure Databricks with Microsoft Entra ID (formerly Azure AD) is how to synchronize groups efficiently. While SCIM is typically used for this, there’s now a much simpler way – thanks to a public preview feature that allows you to directly reference Entra ID groupsCzytaj dalej / Read more

Azure, Azure Databricks, Azure Firewall, No category

The Hidden Cost of Starting Databricks Clusters: How VMs Traffic Can Drain Your Budget

TL;DR: Key Findings Every Databricks cluster node downloads over 12 GiB of data during startup. When routed via Azure Firewall (Data Exfiltration pattern), this can lead to monthly costs exceeding €3,000+ just for control plane traffic. For one bank’s project, this behavior caused over 100 TB of unexpected traffic throughCzytaj dalej / Read more

Azure, Azure Data Factory

Variability in Azure Data Factory pipeline run times and how Azure region load affects them?

Azure Data Factory is a powerful tool, offering a vast range of possibilities, though it can be a bit intricate in terms of pricing and architecture. It’s very dear to me, as I appreciate it for many improvements and conveniences compared to its ancestor, SQL Server Integration Services. However, yearsCzytaj dalej / Read more

Azure Synapse, No category

Exporting data to PARQUET file and ADLS Gen2 using Azure Synapse Serverless

Exporting query data is quite simple as one-two-three: One: define your file format

  Two: define your file location (note: you should have read/write/list permission the path)

    Three: Create external table in particular location, using format and path from previous steps:

  Took 43sec toCzytaj dalej / Read more

Azure, Azure Bastion

Azure Bastion – creating proper NSG rules

Just in case someone came across a problem defining proper nsg rules for Azure Bastion… Well, here they are: Works like a charm 😀 And below the ARM for it: