Deleting files from blob – TypeError: quote_from_bytes() expected bytes

Question:

I have some files inside a container named data:

folder1/somepath/folder2/output/folder3/my_file1.csv
folder1/somepath/folder2/output/folder3/my_file4.csv
folder1/somepath/folder2/output/folder3/my_file23.csv

I have the following code:

file_names_prefix = os.path.join('folder1/somepath/','folder2','output','folder3','my_file')
client = BlobServiceClient('https://mystoragename.blob.core.windows.net',credential=ManagedIdentityCredential()).get_container_client('data')
blob_list = client.list_blobs(name_starts_with=file_names_prefix)
file_list = [blob.name for blob in blob_list]

The code above produces the following output:

['folder1/somepath/folder2/output/folder3/my_file1.csv',
 'folder1/somepath/folder2/output/folder3/my_file4.csv',
'folder1/somepath/folder2/output/folder3/my_file23.csv']

but when trying to delete these files using:

client.delete_blobs(file_list)

There is an error:

TypeError Traceback (most recent call last)
/tmp/ipykernel_2376/712121654.py in
—-> 1 client.delete_blobs(file_list)

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/tracing/decorator.py in wrapper_use_tracer(*args, **kwargs)
81 span_impl_type = settings.tracing_implementation()
82 if span_impl_type is None:
—> 83 return func(*args, **kwargs)
84
85 # Merge span is parameter is set, but only if no explicit parent are passed

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/storage/blob/_container_client.py in delete_blobs(self, *blobs, **kwargs)
1298 return iter(list())
1299
-> 1300 reqs, options = self._generate_delete_blobs_options(*blobs, **kwargs)
1301
1302 return self._batch_send(*reqs, **options)

/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/storage/blob/_container_client.py in _generate_delete_blobs_options(self, *blobs, **kwargs)
1206 req = HttpRequest(
1207 "DELETE",
-> 1208 "/{}/{}{}".format(quote(container_name), quote(blob_name, safe=’/~’), self._query_str),
1209 headers=header_parameters
1210 )

/anaconda/envs/azureml_py38/lib/python3.8/urllib/parse.py in quote(string, safe, encoding, errors)
817 if errors is not None:
818 raise TypeError("quote() doesn’t support ‘errors’ for bytes")
–> 819 return quote_from_bytes(string, safe)
820
821 def quote_plus(string, safe=”, encoding=None, errors=None):

/anaconda/envs/azureml_py38/lib/python3.8/urllib/parse.py in quote_from_bytes(bs, safe)
842 """
843 if not isinstance(bs, (bytes, bytearray)):
–> 844 raise TypeError("quote_from_bytes() expected bytes")
845 if not bs:
846 return ”

TypeError: quote_from_bytes() expected bytes

Can someone please help?

Asked By: Obiii

||
Source

Answers:

The error is due to lack of permissions. Azure uses Shared Access Singatures[SAS] tokens and roles to protect the Azure Blob storage objects like containers, and blobs. The above code snippet uses default credentials, which has read and list access to the Blob container that is being used, however that user is not having the correct role to delete the blob. Check Azure documentation to know the RBAC roles that allows blob deletion.

In order to delete a blob, the RBAC action that needs to be present for the role is Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete.

See Azure documentation for full list of RBAC actions

Refer this SO answer.

Answered By: Anand Sowmithiran

I tried various things, but nothing worked. Ended up deleting files in a loop.

for file in file_list:
    client.delete_blob(file)
Answered By: Obiii

See https://github.com/Azure/azure-sdk-for-python/issues/25764. delete_blobs takes *blobs as its first argument. So

client.delete_blobs(*file_list)

should do the trick.

Here are the official docs for reference.

Answered By: Steven Jin