How to Auth to Google Cloud using Service Account in Python?
Question:
I’m trying to make a project that will upload Google Storage JSON file to BigQuery (just automate something that is done manually now), and I’d like to use a ‘service account’ for this as my script is going to be run on daily basis.
After reading everything I can found about using service account, I’m still struggling to authenticate.
Here is what i’ve done so far:
- Created JSON key file for service account
- Installed client libraries:
pip install --upgrade google-cloud-bigquery
- Installed Google Cloud SDK according to: https://cloud.google.com/sdk/docs/
- Run
export GOOGLE_APPLICATION_CREDENTIALS=<path_to_service_account_file> with key path specified correctly
Now I’m trying to run the following python script:
from google.cloud import bigquery
bigquery_client = bigquery.Client()
But I get this error:
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.
I’m quite new to both Python and the Google Cloud API so possbily missed something. Can someone point out where/what was wrong in my steps above or point me to clear instruction for dummys about setting up and running simple script with BigQuery using service account?
Answers:
Are you running the script in the same command line session as the one you set your environment variable in using export? If not, you might want to look into setting it for your user or system (see this question for more info).
Another option that might make things even easier and takes care of these things automatically is using the gcloud cli tool. If you look at the second option here under the How the Application Default Credentials work, you can find info on how to use it to manage the credentials for you (gcloud auth login and gcloud auth application-default login)
I usually set this variable in the python script itself, something like:
import os
from google.cloud.bigquery.client import Client
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path_to_json_file'
bq_client = Client()
If you have implemented more fine-grained control over service accounts permissions and you have an app that needs to use several of them (say one for Pub/Sub, one for storage), then you would have to set the GOOGLE_APPLICATION_CREDENTIALS environment variable before creating each client.
Instead, you can load your credentials separately and pass them to their appropriate clients like so:
import json
from google.cloud import storage
from google.oauth2 import service_account
project_id = 'my-test-project'
with open('/path/to/my/service_keys/storage_service.json') as source:
info = json.load(source)
storage_credentials = service_account.Credentials.from_service_account_info(info)
storage_client = storage.Client(project=project_id, credentials=storage_credentials)
Just make sure in your IAM console that the account has the right permissions to perform the operations you need it to do, but luckily in that case the error messages are really informative.
This is an old question however I want to add that you must create a new service account and not use an old one. A recent Google Cloud Next presentation on security stated that there is no guarantee that the default service account will exist in future, and it could be removed at any time (or its available permissions changed), so none of your applications should depend on it. Also I’ve found that there are potential issues with authentication when using the default service account and creating a new one is more likely to allow the control you need to successfully authenticate.
Refer to the following YouTube presentation from 11mins 10s in:
I simply set credentials as a variable in python code, not a good idea to show credentials directly in python code though 🙂
#!pip install google-api-python-client
from google.oauth2 import service_account
from googleapiclient import discovery
#Downloaded credentials in JSON format
gcp_sa_credentials={
"type": "service_account",
"project_id": "mygcpprojectid-1122",
"private_key_id": "myprivkeyid",
"private_key": "-----BEGIN PRIVATE KEY-----nmyprivatekeyn-----END PRIVATE KEY-----n",
"client_email": "[email protected]",
"client_id": "myclientid",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/client-mail%40mygcpprojectid-1122.iam.gserviceaccount.com"
}
project_id=gcp_sa_credentials["project_id"]
credentials = service_account.Credentials.from_service_account_info(gcp_sa_credentials)
client = dns.Client(project=project_id,credentials=credentials)
I’m trying to make a project that will upload Google Storage JSON file to BigQuery (just automate something that is done manually now), and I’d like to use a ‘service account’ for this as my script is going to be run on daily basis.
After reading everything I can found about using service account, I’m still struggling to authenticate.
Here is what i’ve done so far:
- Created JSON key file for service account
- Installed client libraries:
pip install --upgrade google-cloud-bigquery - Installed Google Cloud SDK according to: https://cloud.google.com/sdk/docs/
- Run
export GOOGLE_APPLICATION_CREDENTIALS=<path_to_service_account_file>with key path specified correctly
Now I’m trying to run the following python script:
from google.cloud import bigquery
bigquery_client = bigquery.Client()
But I get this error:
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please setGOOGLE_APPLICATION_CREDENTIALSor
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.
I’m quite new to both Python and the Google Cloud API so possbily missed something. Can someone point out where/what was wrong in my steps above or point me to clear instruction for dummys about setting up and running simple script with BigQuery using service account?
Are you running the script in the same command line session as the one you set your environment variable in using export? If not, you might want to look into setting it for your user or system (see this question for more info).
Another option that might make things even easier and takes care of these things automatically is using the gcloud cli tool. If you look at the second option here under the How the Application Default Credentials work, you can find info on how to use it to manage the credentials for you (gcloud auth login and gcloud auth application-default login)
I usually set this variable in the python script itself, something like:
import os
from google.cloud.bigquery.client import Client
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path_to_json_file'
bq_client = Client()
If you have implemented more fine-grained control over service accounts permissions and you have an app that needs to use several of them (say one for Pub/Sub, one for storage), then you would have to set the GOOGLE_APPLICATION_CREDENTIALS environment variable before creating each client.
Instead, you can load your credentials separately and pass them to their appropriate clients like so:
import json
from google.cloud import storage
from google.oauth2 import service_account
project_id = 'my-test-project'
with open('/path/to/my/service_keys/storage_service.json') as source:
info = json.load(source)
storage_credentials = service_account.Credentials.from_service_account_info(info)
storage_client = storage.Client(project=project_id, credentials=storage_credentials)
Just make sure in your IAM console that the account has the right permissions to perform the operations you need it to do, but luckily in that case the error messages are really informative.
This is an old question however I want to add that you must create a new service account and not use an old one. A recent Google Cloud Next presentation on security stated that there is no guarantee that the default service account will exist in future, and it could be removed at any time (or its available permissions changed), so none of your applications should depend on it. Also I’ve found that there are potential issues with authentication when using the default service account and creating a new one is more likely to allow the control you need to successfully authenticate.
Refer to the following YouTube presentation from 11mins 10s in:
I simply set credentials as a variable in python code, not a good idea to show credentials directly in python code though 🙂
#!pip install google-api-python-client
from google.oauth2 import service_account
from googleapiclient import discovery
#Downloaded credentials in JSON format
gcp_sa_credentials={
"type": "service_account",
"project_id": "mygcpprojectid-1122",
"private_key_id": "myprivkeyid",
"private_key": "-----BEGIN PRIVATE KEY-----nmyprivatekeyn-----END PRIVATE KEY-----n",
"client_email": "[email protected]",
"client_id": "myclientid",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/client-mail%40mygcpprojectid-1122.iam.gserviceaccount.com"
}
project_id=gcp_sa_credentials["project_id"]
credentials = service_account.Credentials.from_service_account_info(gcp_sa_credentials)
client = dns.Client(project=project_id,credentials=credentials)