azure-blob-storage

Python dataframe filter on a specific column

Python dataframe filter on a specific column Question: I am reading 3 blobs from Azure storage , loading them into a dataframe and later filtering the dataframe. Below is the code. blob_service_client = BlobServiceClient.from_connection_string(connect_str) container_name = "" path = "/" dt = ” pth = os.path.join(path, dt) container_client = blob_service_client.get_container_client(container_name) blob_list = container_client.list_blobs(name_starts_with=pth) for blob …

Total answers: 1

Deleting files from blob – TypeError: quote_from_bytes() expected bytes

Deleting files from blob – TypeError: quote_from_bytes() expected bytes Question: I have some files inside a container named data: folder1/somepath/folder2/output/folder3/my_file1.csv folder1/somepath/folder2/output/folder3/my_file4.csv folder1/somepath/folder2/output/folder3/my_file23.csv I have the following code: file_names_prefix = os.path.join(‘folder1/somepath/’,’folder2′,’output’,’folder3′,’my_file’) client = BlobServiceClient(‘https://mystoragename.blob.core.windows.net’,credential=ManagedIdentityCredential()).get_container_client(‘data’) blob_list = client.list_blobs(name_starts_with=file_names_prefix) file_list = [blob.name for blob in blob_list] The code above produces the following output: [‘folder1/somepath/folder2/output/folder3/my_file1.csv’, ‘folder1/somepath/folder2/output/folder3/my_file4.csv’, ‘folder1/somepath/folder2/output/folder3/my_file23.csv’] but when …

Total answers: 3

Process to interact with blob storage files from Databricks notebooks

Process to interact with blob storage files from Databricks notebooks Question: Within a Azure Databricks notebook, I am attempting to perform a transformation on some csv’s which are in blob storage using the following: *import os import glob import pandas as pd os.chdir(r’wasbs://dalefactorystorage.blob.core.windows.net/dale’) allFiles = glob.glob("*.csv") # match your csvs for file in allFiles: df …

Total answers: 2

Read csv from Azure blob Storage and store in a DataFrame

Read csv from Azure blob Storage and store in a DataFrame Question: I’m trying to read multiple CSV files from blob storage using python. The code that I’m using is: blob_service_client = BlobServiceClient.from_connection_string(connection_str) container_client = blob_service_client.get_container_client(container) blobs_list = container_client.list_blobs(folder_root) for blob in blobs_list: blob_client = blob_service_client.get_blob_client(container=container, blob="blob.name") stream = blob_client.download_blob().content_as_text() I’m not sure what is …

Total answers: 5

Best way to overwrite Azure Blob in Python

Best way to overwrite Azure Blob in Python Question: If I try to overwrite an existing blob: blob_client = BlobClient.from_connection_string(connection_string, container_name, blob_name) blob_client.upload_blob(‘Some text’) I get a ResourceExistsError. I can check if the blob exists, delete it, and then upload it: try: blob_client.get_blob_properties() blob_client.delete_blob() except ResourceNotFoundError: pass blob_client.upload_blob(‘Some text’) Taking into account both what the …

Total answers: 3

AzureBlob Upload ERROR:The specified blob already exists

AzureBlob Upload ERROR:The specified blob already exists Question: I am trying to upload file to Azure container daily. I got an Error:”The specified blob already exists” when uploading file with same file( I want to overwrite the file) from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient conn_str = yml[‘AZURE_BLOB’][‘CONN_STR’] container_name = yml[‘AZURE_BLOB’][‘CONTAINER_NAME’] # Create the BlobServiceClient that …

Total answers: 2

list and restore soft deleted blobs – azure python

list and restore soft deleted blobs – azure python Question: I need to restore multiple blobs in my container.The storage account was enabled with soft delete for 10 days. I see here on how to undelete , but I have multiple blobs recursively within directories which are soft deleted However I failed to find here …

Total answers: 4

Write Python DataFrame as CSV into Azure Blob

Write Python DataFrame as CSV into Azure Blob Question: I have got two questions on reading and writing Python objects from/to Azure blob storage. Can someone tell me how to write Python dataframe as csv file directly into Azure Blob without storing it locally? I tried using the functions create_blob_from_text and create_blob_from_stream but none of …

Total answers: 4