Whether using blob storage as CDN, hosting a static website, or any other purpose, the Azure file copy task can be used to upload files from Azure DevOps pipelines to blob storage in Azure. Even though these are common scenarios, there are some gotchas associated.
To start, you need to create a service connection in Azure and Azure DevOps. A Service principal requires following roles to access the storage account:
- Storage Account Contributor (required to read storage account and create containers, Contributor also works)
- Storage Blob Data Contributor (required to write data)
AzCopy is executed behind the scenes, and it uses either the Az or AzureRM modules depending on which is installed on the machine. There are known issues in older versions of these modules which might cause AzCopy to occasionally fail. If you are encountering upload failures, check the troubleshooting section.
Azure file copy task parameters
- Task version: 4.* (latest as of May 2022)
- Source: (path/to/folder/*)
- Azure Subscription: (service connection)
- Destination Type: Azure Blob
- Container Name: (container name)
- Blob Prefix: (optional, creates a parent folder for uploaded files)
- Optional Arguments: (arguments passed to AzCopy)
Tips:
- When source folder path ends with ‘/*‘, the source folder is not copied to the destination; only the files contained inside the folder are copied.
- For CDN scenarios you can use artifact version as blob prefix:
- Classic Release pipeline: $(Release.Artifacts.<ArtifactAlias>.BuildNumber)
- YAML pipeline:
- Artifact from another pipeline: $(resources.pipeline.<ArtifactAlias>.runName)
- Current pipeline: $(Build.BuildNumber)
3. You can pass multiple arguments to AzCopy. The example below copies all files, skips existing, and sets the max-age to 30 days:
--recursive=true `
--overwrite=false `
--cache-control="public, max-age=2592000"
YAML example
- task: AzureFileCopy@4
displayName: Copy files to storage account
inputs:
SourcePath: $(System.DefaultWorkingDirectory)/_artifactAlias/drop/*
azureSubscription: My Service Connection
Destination: AzureBlob
storage: mystorageaccount
ContainerName: mycontainer
BlobPrefix: myprefix
AdditionalArgumentsForBlobCopy: |
--recursive=true `
--overwrite=true
Troubleshooting
AzCopy consistently fails with non-zero exit code
Number of Transfers Failed: 1
...
Final Job Status: Failed
Upload to container: 'mycontainer' in storage account: 'mystorageaccount' with blob prefix: 'myprefix' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.'
Solution: Service principal requires “Storage Blob Data Contributor” built-in role for the storage account.
AzCopy randomly fails for some files
Number of Transfers Completed: 2
Number of Transfers Failed: 1
...
Final Job Status: CompletedWithErrors
Upload to container: 'mycontainer' in storage account: 'mystorageaccount' with blob prefix: 'myprefix' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.'
AzCopy log file might contain following:
ERR: [P#0-T#1] UPLOADFAILED: %5C%5C?\E:\Agents\1\_work\r1\a\_artifactAlias\drop\file.ext : 000 : File modified since transfer scheduled
Solution: If using a self-hosted agent, update the Az or AzureRM modules on the agent machine, as newer versions include a fix. Microsoft-hosted agents should already have the updated modules.
Alternative: If you do not have access to the agent machine, try updating file timestamps before running AzCopy:
$currentDate = Get-Date
Get-ChildItem . * -Recurse | ForEach-Object { $_.LastWriteTime = $currentDate }
Storage account not found or missing permissions to perform listKeys
Storage account: mystorageaccount not found. The selected service connection ‘Service Principal’ supports storage accounts of Azure Resource Manager type only.
or
The client ‘xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx’ with object id ‘xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx’ does not have authorization to perform action ‘Microsoft.Storage/storageAccounts/listKeys/action’ over scope ‘/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myrg/providers/Microsoft.Storage/storageAccounts/mystorageaccount’ or the scope is invalid. If access was recently granted, please refresh your credentials. (in function: Get-AzRMStorageKeys)
Solution: Service principal is missing the ‘Storage Account Contributor’ built-in role for the storage account.
