client.get_bucket() returns error: api_request() got an unexpected keyword argument 'extra_api_info'
Question:
I’m trying to store a newline-delimited JSON string in a GCS bucket using a cloud function, but seeing an error. I start by converting a dataframe to ndjson, then attempt to upload this to my GCS bucket as below. There is more code above this, but not relevant to my problem:
import pandas as pd
from google.cloud import storage
from google.cloud.storage import blob
df = df.to_json(orient="records", lines=True)
storage_client = storage.Client(project='my-project')
bucket = storage_client.get_bucket('my-bucket')
blob = bucket.blob('my-blob')
blob.upload_from_string(df)
When running this, I find the error below in the logs:
Exception on / [POST] Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request rv = self.handle_user_exception(e)
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request rv = self.dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/functions_framework/__init__.py", line 99, in view_func return function(request._get_current_object())
File "/workspace/main.py", line 66, in my_request bucket = storage_client.get_bucket('my-bucket')
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/client.py", line 787, in get_bucket retry=retry,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/bucket.py", line 1037, in reload retry=retry,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/_helpers.py", line 244, in reload _target_object=self,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/client.py", line 373, in _get_resource _target_object=_target_object,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 73, in api_request return call()
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func on_error=on_error,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/retry.py", line 190, in retry_target return target() TypeError: api_request() got an unexpected keyword argument 'extra_api_info'
This ‘extra_api_info’ argument appears to be the culprit, but I have no idea what this means, and never used to get this error when following exactly the same approach, so I wonder whether this is down to some change between different versions of the ‘google.cloud’ Python module.
Answers:
I worked out the answer to my own question. It was indeed a module version issue as I suspected.
Specifying google.cloud.storage==1.44.0
in my requirements.txt
file solved the problem, as my code is seemingly not compatible with the latest version of that module (for reasons that escape me).
I had the same problem; google.cloud.storage==1.44.0
fixed it.
I’m trying to store a newline-delimited JSON string in a GCS bucket using a cloud function, but seeing an error. I start by converting a dataframe to ndjson, then attempt to upload this to my GCS bucket as below. There is more code above this, but not relevant to my problem:
import pandas as pd
from google.cloud import storage
from google.cloud.storage import blob
df = df.to_json(orient="records", lines=True)
storage_client = storage.Client(project='my-project')
bucket = storage_client.get_bucket('my-bucket')
blob = bucket.blob('my-blob')
blob.upload_from_string(df)
When running this, I find the error below in the logs:
Exception on / [POST] Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request rv = self.handle_user_exception(e)
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request rv = self.dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/functions_framework/__init__.py", line 99, in view_func return function(request._get_current_object())
File "/workspace/main.py", line 66, in my_request bucket = storage_client.get_bucket('my-bucket')
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/client.py", line 787, in get_bucket retry=retry,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/bucket.py", line 1037, in reload retry=retry,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/_helpers.py", line 244, in reload _target_object=self,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/client.py", line 373, in _get_resource _target_object=_target_object,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 73, in api_request return call()
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func on_error=on_error,
File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/retry.py", line 190, in retry_target return target() TypeError: api_request() got an unexpected keyword argument 'extra_api_info'
This ‘extra_api_info’ argument appears to be the culprit, but I have no idea what this means, and never used to get this error when following exactly the same approach, so I wonder whether this is down to some change between different versions of the ‘google.cloud’ Python module.
I worked out the answer to my own question. It was indeed a module version issue as I suspected.
Specifying google.cloud.storage==1.44.0
in my requirements.txt
file solved the problem, as my code is seemingly not compatible with the latest version of that module (for reasons that escape me).
I had the same problem; google.cloud.storage==1.44.0
fixed it.