Key Vault Integration with AKS — Azure

9 min readMar 23, 2024

Azure Key Vault allows you to securely access sensitive information from within your applications → Keys, secrets, and certificates are protected without you’re having to write the code yourself, and you can easily use them from your applications.

The Kubernetes Secrets Store CSI Driver integrates secrets stores with Kubernetes through a Container Storage Interface (CSI) volume. If you integrate the Secrets Store CSI Driver with AKS enabled by Azure Arc, you can mount secrets, keys, and certificates as a volume.

The SecretProviderClass is a namespaced resource in Secrets Store CSI Provider that is used to provide configurations and provider-specific parameters to the CSI provider.

buymeacoffee ☕ 👈 Click the link

Managed identities use certificate-based authentication. Each managed identity’s credentials have an expiration of 90 days and are rolled after 45 days.

AKS uses both system-assigned and user-assigned managed identity types, and these identities are immutable.

Create Azure Resource Group

az group create --name keyvault-demo --location eastus

A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group.

Create an AKS cluster with Azure Key Vault provider for Secrets Store CSI Driver support

AKS to deploy, scale, and manage Docker containers and container-based applications across a cluster of container hosts.

az aks create --name keyvault-demo-cluster -g keyvault-demo --node-count 1 --enable-addons azure-keyvault-secrets-provider --enable-oidc-issuer --enable-workload-identity

Microsoft Azure Key Vault is a cloud-based security service offered by Microsoft as part of its Azure platform. It provides a secure and centralized storage solution for cryptographic keys and secrets, such as passwords, certificates and keys used for encryption.

Secrets Management → Azure Key Vault can be used to Securely store and tightly control access to tokens, passwords, certificates, API keys, and other secrets.

Key Management → Azure Key Vault can be used as a Key Management solution.

Verify the azure portal

Get the Kubernetes cluster credentials (Update kubeconfig)

az aks get-credentials --resource-group keyvault-demo --name keyvault-demo-cluster

Kubernetes stores cluster authentication information in a “YAML” file known as kubeconfig. The kubeconfig file grants access to clients, such as kubectl, to run commands against the cluster. By default, kubeconfig is stored in $HOME/.

The kubeconfig file is a YAML file containing groups of clusters, users, and contexts.

kubeconfig contains a list of contexts to which kubectl refers when running commands

kubectl config current-context

The context is a critical aspect of the kubeconfig file. It determines which cluster and namespace you’re currently working on and which user’s credentials to use.

By using the kubectl config use-context command, you can switch between different contexts. This is especially useful when working with multiple clusters.

Verify that each node in your cluster’s node pool has a Secrets Store CSI Driver pod and a Secrets Store Provider Azure pod running

kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)' -o wide

A node pool is a group of nodes within a cluster that all have the same configuration.

The Secrets Store CSI Driver allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.

Secrets-store → Implements the CSI Node service gRPC services described in the CSI specification. It’s responsible for mount/unmount the volumes during pod creation/deletion. This component is developed and maintained in this repo.

gRPC is an open source remote procedure call (RPC) framework that enables client and server applications to communicate with each other remotely and transparently.

Keyvault creation and configuration

Create a key vault with Azure role-based access control (Azure RBAC).

az keyvault create -n aks-demo-ibbus -g keyvault-demo -l eastus --enable-rbac-authorization

Centralizing storage of application secrets in Azure Key Vault allows you to control their distribution.

Key Vault greatly reduces the chances that secrets may be accidentally leaked. When application developers use Key Vault, they no longer need to store security information in their application

Check azure portal

Role assignments enable you to grant a principal (such as a user, a group, a managed identity, or a service principal) access to a specific Azure resource.

Add role assignment

Access Control (IAM) → Grant access to this resoule → Add role assignment → Key Vault Administrator

Add members

Create a Keys & Secret

The Azure Key Vault secrets client library allows you to securely store and control the access to tokens, passwords, API keys, and other secrets.

Connect your Azure ID to the Azure Key Vault Secrets Store CSI Driver

The Azure Identity library provides Microsoft Entra ID (formerly Azure Active Directory) token authentication support across the Azure SDK. It provides a set of TokenCredential implementations which can be used to construct Azure SDK clients which support Microsoft Entra token authentication.

Azure Key Vault provider for Secrets Store CSI Driver allows you to get secret contents stored in an Azure Key Vault instance and use the Secrets Store CSI driver interface to mount them into Kubernetes pods.

Configure workload identity

A workload identity is an identity you assign to a software workload (such as an application, service, script, or container) to authenticate and access other services and resources.

export SUBSCRIPTION_ID=f30deb63-a417-4fa4-afc1-813a7d3920bb
export RESOURCE_GROUP=keyvault-demo
export UAMI=azurekeyvaultsecretsprovider-keyvault-demo-cluster
export KEYVAULT_NAME=aks-demo-ibbus
export CLUSTER_NAME=keyvault-demo-cluster

az account set --subscription $SUBSCRIPTION_ID

Managed identities provide an automatically managed identity in Microsoft Entra ID for applications to use when connecting to resources that support Microsoft Entra authentication.

Applications can use managed identities to obtain Microsoft Entra tokens without having to manage any credentials.

Create a managed identity

az identity create --name $UAMI --resource-group $RESOURCE_GROUP

export USER_ASSIGNED_CLIENT_ID="$(az identity show -g $RESOURCE_GROUP --name $UAMI --query 'clientId' -o tsv)"
export IDENTITY_TENANT=$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query identity.tenantId -o tsv)

Azure portal

Create a role assignment that grants the workload ID access the key vault

Grant a principal (such as a user, a group, a managed identity, or a service principal) access to a specific Azure resource

export KEYVAULT_SCOPE=$(az keyvault show --name $KEYVAULT_NAME --query id -o tsv)

az role assignment create --role "Key Vault Administrator" --assignee $USER_ASSIGNED_CLIENT_ID --scope $KEYVAULT_SCOPE

Get the AKS cluster OIDC Issuer URL

OpenID Connect (OIDC) is an open authentication protocol that works on top of the OAuth 2.0 framework.

export AKS_OIDC_ISSUER="$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)"
echo $AKS_OIDC_ISSUER

OpenID Connect is commonly used for apps that are purely in the cloud, such as mobile apps, websites, and web APIs.

Create the service account for the pod

Azure service accounts is to grant permissions to resources in Azure. There are three types of Azure accounts

Service principals
Managed identities → !) user-assigned identities !!) system-assigned identities
User accounts employed as service accounts

export SERVICE_ACCOUNT_NAME="workload-identity-sa"
export SERVICE_ACCOUNT_NAMESPACE="default"

Service accounts provide a way to isolate the identity and permissions of a service from the identity of the user who is logged on to the computer.

Two types of managed identities exist → user-assigned identities and system-assigned identities.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: ${USER_ASSIGNED_CLIENT_ID}
  name: ${SERVICE_ACCOUNT_NAME}
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
EOF

A user-assigned managed identity can be created and managed through the Azure portal, Azure CLI (or) Azure PowerShell, a software development kit (SDK) (or) by making direct calls to the Azure Resource Manager REST API.

A system-assigned managed identity is automatically created when you enable the feature for an Azure resource.

You can use any interactive Azure AD user account as a service account, doing so is not recommended.

Setup Federation

OpenID Connect (OIDC) is an identity authentication protocol that is an extension of open authorization (OAuth) 2.0 to standardize the process for authenticating and authorizing users when they sign in to access digital services.

OIDC provides authentication, which means verifying that users are who they say they are

export FEDERATED_IDENTITY_NAME="aksfederatedidentity" 

az identity federated-credential create --name $FEDERATED_IDENTITY_NAME --identity-name $UAMI --resource-group $RESOURCE_GROUP --issuer ${AKS_OIDC_ISSUER} --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}

Create the Secret Provider Class

The SecretProviderClass is a namespaced resource in Secrets Store CSI Driver that is used to provide driver configurations and provider-specific parameters to the CSI driver.

cat <<EOF | kubectl apply -f -
# This is a SecretProviderClass example using workload identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-wi # needs to be unique per namespace
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "${USER_ASSIGNED_CLIENT_ID}" # Setting this to use workload identity
    keyvaultName: ${KEYVAULT_NAME}       # Set to the name of your key vault
    cloudName: ""                         # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: secret1             # Set to the name of your secret
          objectType: secret              # object types: secret, key, or cert
          objectVersion: ""               # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: key1                # Set to the name of your key
          objectType: key
          objectVersion: ""
    tenantId: "${IDENTITY_TENANT}"        # The tenant ID of the key vault
EOF

Apply the configuration

Verify Keyvault AKS Integration

The Azure Key Vault provider for Secrets Store CSI Driver allows for the integration of an Azure Key Vault as a secret store with an Azure Kubernetes Service (AKS) cluster via a CSI volume

Create a sample pod to mount the secrets

cat <<EOF | kubectl apply -f -
# This is a sample pod definition for using SecretProviderClass and workload identity to access your key vault
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline-wi
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: "workload-identity-sa"
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.29-4
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname-wi"
EOF

Check container running

kubectl get pods

Creating a container

Got a error

kubectl describe pods

Analyse the pod

Roles permission mismatched

Assign a correct roles finally container is running

Run the container

List the contents of the volume

kubectl exec busybox-secrets-store-inline-wi -- ls /mnt/secrets-store/

Verify the contents in the file

kubectl exec busybox-secrets-store-inline-wi -- cat /mnt/secrets-store/secret1

Once done, Finally delete the AKS

az group delete --name keyvault-demo

Thank you 🙏 for taking the time to read our blog.