Azure AKS/Container App can't access Key vault using managed identity

Question

I have a docker container python app deployed on a kubernetes cluster on Azure (I also tried on a container app). I’m trying to connect this app to Azure key vault to fetch some secrets. I created a managed identity and assigned it to both but the python app always fails to find the managed identity to even attempt connecting to the key vault.

The Managed Identity role assignments:

Key Vault Contributor -> on the key vault

Managed Identity Operator -> Managed Identity

Azure Kubernetes Service Contributor Role,
Azure Kubernetes Service Cluster User Role,
Managed Identity Operator -> on the resource group that includes the cluster

Also on the key vault Access policies I added the Managed Identity and gave it access to all key, secrets, and certs permissions (for now)

Python code:

 credential = ManagedIdentityCredential()
 vault_client = SecretClient(vault_url=key_vault_uri, credential=credential)
 retrieved_secret = vault_client.get_secret(secret_name)

I keep getting the error:

azure.core.exceptions.ClientAuthenticationError: Unexpected content type "text/plain; charset=utf-8"
Content: no azure identity found for request clientID

So at some point I attempted to add the managed identity clientID in the cluster secrets and load it from there and still got the same error:

Python code:

    def get_kube_secret(self, secret_name):
        kube_config.load_incluster_config()
        v1_secrets = kube_client.CoreV1Api()
        
        string_secret = str(v1_secrets.read_namespaced_secret(secret_name, "redacted_namespace_name").data).replace("'", """)
        json_secret = json.loads(string_secret)
        return json_secret
    
    def decode_base64_string(self, encoded_string):
        decoded_secret = base64.b64decode(encoded_string.strip())
        decoded_secret = decoded_secret.decode('UTF-8')
        return decoded_secret

    managed_identity_client_id_secret = self.get_kube_secret('managed-identity-credential')['clientId']
    managed_identity_client_id = self.decode_base64_string(managed_identity_client_id_secret)

Update:

I also attempted to use the secret store CSI driver, but I have a feeling I’m missing a step there. Should the python code be updated to be able to use the secret store CSI driver?

# This is a SecretProviderClass using user-assigned identity to access the key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-user-msi
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"          # Set to true for using managed identity
    userAssignedIdentityID: "$CLIENT_ID"   # Set the clientID of the user-assigned managed identity to use
    vmmanagedidentityclientid: "$CLIENT_ID"
    keyvaultName: "$KEYVAULT_NAME"        # Set to the name of your key vault
    cloudName: ""                         # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  ""
    tenantId: "$AZURE_TENANT_ID"

Deployment Yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: redacted_namespace
  labels:
    app: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: redacted_image
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          imagePullPolicy: Always
          resources:
            # You must specify requests for CPU to autoscale
            # based on CPU utilization
            requests:
              cpu: "250m"
          env:
            - name:  test-secrets
              valueFrom:
                secretKeyRef:
                  name:  test-secrets
                  key:  test-secrets
          volumeMounts:
            - name: test-secrets
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: test-secrets
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-kvname-user-msi"
      dnsPolicy: ClusterFirst

Update 16/01/2023

I followed the steps in the answers and the linked docs to the letter, even contacted Azure support and followed it step by step with them on the phone and the result is still the following error:

"failed to process mount request" err="failed to get objectType:secret, objectName:MongoUsername, objectVersion:: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://<RedactedVaultName>.vault.azure.net/secrets/<RedactedSecretName>/?api-version=2016-10-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=<RedactedClientId>&resource=https%3A%2F%2Fvault.azure.net"

Asked By: Rimon

||

Source

Answer 1

What you are referring to is called pod identity (recently deprecated for workload identity).

if the cluster is configured with managed identity, you can use workload identity.

However, for AKS I suggest configuring the secret store CSI driver to fetch secrets from KV and have them as k8s secrets. To use managed identity for secret provider, refer to this doc.

Then you can configure your pods to read those secrets.

Answered By: akathimi

Answer 2

Using the Secrets Store CSI Driver, you can configure the SecretProviderClass to use a workload identity by setting the clientID in the SecretProviderClass. You’ll need to use the client ID of your user assigned managed identity and change the usePodIdentity and useVMManagedIdentity setting to false.

With this approach, you don’t need to add any additional code in your app to retrieve the secrets. Instead, you can mount a secrets store (using CSI driver) as a volume mount in your pod and have secrets loaded as environment variables which is documented here.

This doc will walk you through setting it up on Azure, but at a high-level here is what you need to do:

Register the EnableWorkloadIdentityPreview feature using Azure CLI
Create an AKS cluster using Azure CLI with the azure-keyvault-secrets-provider add-on enabled and --enable-oidc-issuer and --enable-workload-identiy flags set
Create an Azure Key Vault and set your secrets
Create an Azure User Assigned Managed Identity and set an access policy on the key vault for the the managed identity’ client ID
Connect to the AKS cluster and create a Kubernetes ServiceAccount with annotations and labels that enable this for Azure workload identity
Create an Azure identity federated credential for the managed identity using the AKS cluster’s OIDC issuer URL and Kubernetes ServiceAccount as the subject
Create a Kubernetes SecretProviderClass using clientID to use workload identity and adding a secretObjects block to enable syncing objects as environment variables using Kubernetes secret store.
Create a Kubernetes Deployment with a label to use workload identity, the serviceAccountName set to the service account you created above, volume using CSI and the secret provider class you created above, volumeMount, and finally environment variables in your container using valueFrom and secretKeyRef syntax to mount from your secret object store.

Hope that helps.

Answered By: Paul Yu

Answer 3

I finally figured it out, I contacted microsoft support and it seams Aks Preview is a bit buggy "go figure". They recommended to revert back to a stable version of the CLI and use user assigned identity.

I did just that but this time, instead of creating my own identity that I would then assign to both the vault and the cluster as this seams to confuse it. I used the the identity the cluster automatically generates for the nodes.

Maybe not the neatest solution, but it’s the only one that worked for me without any issues.

Finally, some notes missing from the Azure docs:
Since the CSI driver mounts the secrets as files in the target folder, you still need to read those files yourself to load them as env variables.

For example in python:

def load_secrets():
    directory = '/path/to/mounted/secrets/folder'
    if not os.path.isdir(directory):
        return

    for filename in os.listdir(directory):
        file_path = os.path.join(directory, filename)
        # checking if it is a file
        if os.path.isfile(file_path):
            with open(file_path, 'r') as file:
                file_value = file.read()
                os.environ.setdefault(filename, file_value)

Answered By: Rimon

Azure AKS/Container App can't access Key vault using managed identity

Question:

Answers: