Setting ["GOOGLE_APPLICATION_CREDENTIALS"] from a dict rather than file path

Question

I’m trying to set the environment variable from a dict but getting and error when connecting.

#service account pulls in airflow variable that contains the json dict with service_account credentials

service_account = Variable.get('google_cloud_credentials')

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]=str(service_account)

error

    PermissionDeniedError: Error executing an HTTP request: HTTP response code 403 with body '<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.</Details></Error>'

when reading if I use and point to file then there are no issues.

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]=/file/path/service_account.json

I’m wondering is there a way to convert the dict object to an os path like object? I don’t want to store the json file on the container and airflow/google documentation isn’t clear at all.

Asked By: Sql_Pete_Belfast

||

Source

Answer 1

The Python stringio package lets you create a file-like object backed by a string, but that won’t help here because the consumer of this environment variable is expecting a file path, not a file-like object. I don’t think it’s possible to do what you’re trying to do. Is there a reason you don’t want to just put the credentials in a file?

Answered By: Mike Schwartz

Answer 2

There is a way to do it, but the Google documentation is terrible. So I wrote a Github gist to document the recipe that I and a colleague (Imee Cuison) developed to use the key securely. Sample code below:

import json
from google.oauth2.service_account import Credentials
from google.cloud import secretmanager

def access_secret(project_id:str, secret_id:str, version_id_str="latest")->str:
    """Return the secret in string format"""
    
    # Create the Secret Manager client.
    client = secretmanager.SecretManagerServiceClient()

    # Build the resource name of the secret version.
    name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"

    # Access the secret version.
    response = client.access_secret_version(name=name)

    # Return the decoded payload.
    return response.payload.data.decode('UTF-8')


def get_credentials_from_token(token:str)->Credentials:
    """Given an authentication token, return a Credentials object"""
    
    credential_dict = json.loads(secret_payload)
    return Credentials.from_service_account_info(credential_dict)


credentials_secret = access_secret("my_project", "my_secret")
creds = get_credentials_from_token(credentials_secret)
# And now you can use the `creds` Credentials object to authenticate to an API

Answered By: ChrisFal

Answer 3

Putting the service account into the repository is not a good practice. As a best practice; You need to use authentication propagating from the default google auth within your application.

For instance, using Google Cloud Kubernetes you can use the following python code :

from google.cloud.container_v1 import ClusterManagerClient   
credentials, project = google.auth.default(
    scopes=['https://www.googleapis.com/auth/cloud-platform', ])
credentials.refresh(google.auth.transport.requests.Request())
cluster_manager = ClusterManagerClient(credentials=credentials)

Answered By: london_utku

Setting ["GOOGLE_APPLICATION_CREDENTIALS"] from a dict rather than file path

Question:

Answers: