How to open a file in a GCS bucket in append mode using Python?
Question:
I’ve looked at all related questions and all I can find are uploading to GCS and not writing. I’m not trying to upload a file or create a new one. I need to open the file located in a google cloud storage bucket in append mode i.e create it if it doesn’t exist and write to the end of it using python. If anybody could point me in the right direction, I’d be really grateful.
The upload method replaces my file each time. I’ve tried downloading, appending and then replacing but this requires the file to be initially present in the directory.
Is there a way to implement the append mode functionality in gcs?
PS: I’m trying to do this in cloud functions and not a machine equipped with the Cloud SDK
Answers:
Use gcsfuse to mount the bucket – then you will be able to access the file directly like accessing it on a folder on your machine
There’s no way to do this, because objects in GCS are immutable.
From https://cloud.google.com/storage/docs/key-terms#immutability:
Objects are immutable, which means that an uploaded object cannot change throughout its storage lifetime. An object’s storage lifetime is the time between successful object creation (upload) and successful object deletion. In practice, this means that you cannot make incremental changes to objects, such as append operations or truncate operations. However, it is possible to overwrite objects that are stored in Cloud Storage, and doing so happens atomically — until the new upload completes the old version of the object will be served to readers, and after the upload completes the new version of the object will be served to readers. So a single overwrite operation simply marks the end of one immutable object’s lifetime and the beginning of a new immutable object’s lifetime.
Please do a work around.
from google.cloud import storage
from google.oauth2 import service_account
from google.cloud import storage
credentials = service_account.Credentials.from_service_account_file("creds.json")
client = storage.Client(project='[[Project]]', credentials=credentials)
bucket = client.bucket(str("[[Bucket]]"))
blob = bucket.blob(str("[[BLOB]]"))
// read it as str
dataT=""
with destination_bucket.blob(destination_blob_name).open('r') as f:
dataT=f.read()
print(type(dataT))
// append in that str
dataT = dataT +'nHello world!'
// write that str back
with destination_bucket.blob(destination_blob_name).open('w') as f:
f.write(dataT)
I’ve looked at all related questions and all I can find are uploading to GCS and not writing. I’m not trying to upload a file or create a new one. I need to open the file located in a google cloud storage bucket in append mode i.e create it if it doesn’t exist and write to the end of it using python. If anybody could point me in the right direction, I’d be really grateful.
The upload method replaces my file each time. I’ve tried downloading, appending and then replacing but this requires the file to be initially present in the directory.
Is there a way to implement the append mode functionality in gcs?
PS: I’m trying to do this in cloud functions and not a machine equipped with the Cloud SDK
Use gcsfuse to mount the bucket – then you will be able to access the file directly like accessing it on a folder on your machine
There’s no way to do this, because objects in GCS are immutable.
From https://cloud.google.com/storage/docs/key-terms#immutability:
Objects are immutable, which means that an uploaded object cannot change throughout its storage lifetime. An object’s storage lifetime is the time between successful object creation (upload) and successful object deletion. In practice, this means that you cannot make incremental changes to objects, such as append operations or truncate operations. However, it is possible to overwrite objects that are stored in Cloud Storage, and doing so happens atomically — until the new upload completes the old version of the object will be served to readers, and after the upload completes the new version of the object will be served to readers. So a single overwrite operation simply marks the end of one immutable object’s lifetime and the beginning of a new immutable object’s lifetime.
Please do a work around.
from google.cloud import storage
from google.oauth2 import service_account
from google.cloud import storage
credentials = service_account.Credentials.from_service_account_file("creds.json")
client = storage.Client(project='[[Project]]', credentials=credentials)
bucket = client.bucket(str("[[Bucket]]"))
blob = bucket.blob(str("[[BLOB]]"))
// read it as str
dataT=""
with destination_bucket.blob(destination_blob_name).open('r') as f:
dataT=f.read()
print(type(dataT))
// append in that str
dataT = dataT +'nHello world!'
// write that str back
with destination_bucket.blob(destination_blob_name).open('w') as f:
f.write(dataT)