102 thoughts on “Connecting to Azure Data Lake Storage Gen2 from PowerShell using REST API – a step-by-step guide”

1. this was brilliant. Thank you so much.

   In case anyone is wondering how to not get an error if an item exists (took me some time to figure out): uncomment $stringToSign += “*” + “$n” # If-None-Match + “\n” + and remove it (or uncomment) from the addHeader part

   Reply

2. Hi Michael,

   This is really a very nice post. I still need to implement what you have listed in this post though. While searching online I came up across this page https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-powershell so it looks like Microsoft is releasing a new module to make things easier.

   Reply

   1. Hi Sirini.
      Thanks. Yes, I’m aware of new things since MPA (Multi Protocol Access) is now in general availability.
      To be honest, In my opinion they (Micorosoft, but also other vendors, developers, companies) were waiting for MPA availability since the biggining of ADLS Gen2… They did not release anything and I can say that I understand that. Why should I develop a new branch of my code, just to align it with new protocol specification (ADLS Gen2 REST API) when there is a confirmed feature from Microsoft that will bring a 100% compatibility using BLOB API?
      So now, after almost a year, you can just use BLOB API connector, for example in PowerBI and use it to connect to ADLS Gen2. Pretty cool, but a year to late..

      Anyway, I need to update my article and I will do it soon.
      Thanks.

      Reply

3. Hi Michael,
   Thank you for the post and it is what I needed. I used your LIST directories script and it worked perfectly fine from my desktop (Powershell ISE), then I copied and pasted the same code in Azure Automation Runbook and unfortunately it fails with following error:-
   Invoke-RestMethod : {“error”:{“code”:”AuthorizationFailure”,”message”:”This request is not authorized to perform this
   operation.\nRequestId:0c0b4062-901f-0032-2630-ef9d2f000000\nTime:2020-02-29T18:46:36.9060927Z”}}

   I have used the same code from my desktop and it works absolutely fine. Following is the code I used ( I copied the AccessKey from the Keys on the storage account Key1):-

   # Storage account name
   $StorageAccountName = “XXLLLSSS”
   $FilesystemName =”$logs"
$AccessKey = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXekG6f6J0iNstoZLJghXg=="
$Directory ="queue/2020/02/29"
$MaxResults = "5000"
$Recursive = "true"

   # Rest documentation:
   # https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/list

   $date = [System.DateTime]::UtcNow.ToString("R")

   $n = "n”
   $method = “GET”

   # $stringToSign = “GETnnnnnnnnnnnn”
   $stringToSign = “$method$n” #VERB
   $stringToSign += “$n” # Content-Encoding + “\n” +
   $stringToSign += “$n” # Content-Language + “\n” +
   $stringToSign += “$n” # Content-Length + “\n” +
   $stringToSign += “$n” # Content-MD5 + “\n” +
   $stringToSign += “$n” # Content-Type + “\n” +
   $stringToSign += “$n” # Date + “\n” +
   $stringToSign += “$n” # If-Modified-Since + “\n” +
   $stringToSign += “$n” # If-Match + “\n” +
   $stringToSign += “$n” # If-None-Match + “\n” +
   $stringToSign += “$n” # If-Unmodified-Since + “\n” +
   $stringToSign += “$n” # Range + “\n” +
   $stringToSign +=

   “x-ms-date:$date” + $n +
   “x-ms-version:2018-11-09” + $n #

   $stringToSign +=

   “/$StorageAccountName/$FilesystemName” + $n +
   “directory:” + $Directory + $n +
   “maxresults:” + $MaxResults + $n +
   “recursive:$Recursive” + $n +
   “resource:filesystem”#

   $sharedKey = [System.Convert]::FromBase64String($AccessKey)
   $hasher = New-Object System.Security.Cryptography.HMACSHA256
   $hasher.Key = $sharedKey

   $signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))

   $authHeader = “SharedKey ${StorageAccountName}:$signedSignature”

   $headers = @{“x-ms-date”=$date}
   $headers.Add(“x-ms-version”,”2018-11-09″)
   $headers.Add(“Authorization”,$authHeader)

   $URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + “?directory=$Directory&maxresults=$MaxResults&recursive=$Recursive&resource=filesystem”

   $result = Invoke-RestMethod -method $method -Uri $URI -Headers $headers

   $result

   Reply

   1. Sounds like a firewall issue. Are you sure that your ADLS does not have vnet and firewall filtering enabled? When this kind of filtering is enabled your ADLS can return ‘this request is not authorised’. It is a PaaS service so it does not prevent you from accessing the API, but you always will get this error message until you allow your (or Automation) public IP or allow trusted ms services (not aure how Automation is handled here)

      Reply

      1. Perfect sir, I just resolved that issue and I saw your email. That was the case on Storage Account (Firewall/Network Settings were set in such a way to not allow all networks but selected networks). Thank you again for your prompt response.

         Reply

         1. Cool 🙂 , I’m glad it worked out. Good luck!

            Reply

4. Hi Michal,

   I am trying to rename a file using ADLGen2 Rest API and can’t seem to figure out the mistake. I am trying it on fiddler and it always throws 400 error when i use
   x-ms-rename-source : in header section

   if remove it and run, it creates file successfully

   Put
   https://{storageaccountname.dfs.core.windows.net}/filesystemname/filename?resource=file
   Header –
   Authorization: Bearer {AccessToken}
   Host: {storageaccountname}.dfs.core.windows.net
   Content-Length: 0
   Content-type: application/json

   Reply

   1. Hmm, I didn’t use fiddler, does it work with powershell code?
      There is already a working “rename” script in the article.

      Adding or removing x-ms-rename-source in header section is not the only thing you need to do. There is also a requirement to calculate proper signature for canonicalized headers.
      If you will decide to remove them from headers, you need to remove them from signature too and recalculate Base64 signature.

      Reply

5. Hi Michal, thanks for posting example script. they are very useful to understand ADLSgen2.

   I am using one of your scripts – 3.8 Get permissions from filesystem or path.
   I am successfully able to run and get permissions for particular FilesystemName and path.

   I want to get permission for all the FilesystemName and all the directories, folder and files inside the Storage Accounts ADLS gen2. Can you please suggest what modification required in your example script. Please let me know if it is possible by REST API.
   For e.g.
   project-
   project1- file1.txt
   project1.1 – file.txt
   project2- file1.txt
   project1.1 – file.txt
   project3- file1.txt
   project1.1 – file.txt
   raw
   project1- file1.txt
   project1.1 – file.txt
   project2- file1.txt
   project1.1 – file.txt
   project3- file1.txt
   project1.1 – file.txt
   Project and raw are FilesystemName. project1 is path and inside it is file and folders. Need to get acl of all them
   You help would be much appreciated

   Reply

   1. Hi Adam.
      Well, it will not be a one-step-solution to make this possible. You need to create an algorithm to properly get all information in the desired order:

      1. Get all filesystems and iterate over them
      https://docs.microsoft.com/pl-pl/rest/api/storageservices/datalakestoragegen2/filesystem/list

      2. As a nested step: Get properties of a filesystem
      https://docs.microsoft.com/pl-pl/rest/api/storageservices/datalakestoragegen2/filesystem/getproperties

      3. Then Iterate over a root path listed objects, possibly retrieved with a “recursive” set to true, to get the list of all items… BUT it depends on your needs (can be huge in a large DL). Just remember that the list is limited to 5000 items and you need to implement and use “continuation” parameter
      https://docs.microsoft.com/pl-pl/rest/api/storageservices/datalakestoragegen2/filesystem/list

      4. As a nested iterator you need to traverse your path and for every particular item, directory or file, get properties using
      https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/getproperties

      I know this sounds a lot of requests to send and retrieve, but this is how atomic nature looks for REST API 😐 You cannot query for a bucket of objects in most scenarios, you need to fire them sequentially OR implement as multithreaded operation if youre using PowerShell 7.0 (so to run REST API calls for a bunch of items in parallel, see ForEach-Object -Parallel cmdlet)

      Reply

      1. Thanks for your response Michał ,

         for Get permissions from filesystem or path sample script.

         i try to export the output in the CSV format.
         $result.Headers.’x-ms-acl’ | export-csv \home\adam\permi.csv
         Output result
         Length 204

         can you suggest what i need to change to output to CSV

         Reply

         1. Hi Adam.
            Export-Csv or ConvertTo-Csv operates on a different level of variable type.
            When you handle $result.Headers.’x-ms-acl’ and try to output it, then it will be parsed as a string anywhere that expects that value as a string. But in fact $result.Headers.’x-ms-acl’ can be also converted easily to an object and CSV cmdlets expect the input to be an object. Therefore, there is an implicit conversion from string to object, where object properties are also converted (like the length of the string)

            Ok, that’s for the reasons why it worked like this.
            Now to handle it properly, I suggest to wrap your expected schema and values into PSObject and return it to CSV cmdlet, like this:

            $wrap = New-Object PSObject -Property @{ ‘Filesystem’ = $FilesystemName; ‘x-ms-acl’ = $result.Headers.’x-ms-acl’; ‘x-ms-permissions’ = $result.Headers.’x-ms-permissions’ }
            $wrap | ConvertTo-Csv -NoTypeInformation

            regards,
            m.

            Reply

6. Tysm for putting this document together. It is really helpful 🙂 I have 2 queries

   1. How to modify the last rest API to revoke the access from file system?
   2. Instead of storage account keys, if we want to use SAS token for authentication, how can we do that?

   Reply