In today's cloud-native landscape, securely managing secrets is essential for protecting sensitive data and ensuring compliance. Combining External Secrets Operator (ESO) with Azure Key Vault and Workload Identity authentication provides a robust solution that enhances security while simplifying secret management.

By securely storing secrets in Azure Key Vault and using ESO to synchronize them with Kubernetes, we can eliminate the need for storing sensitive credentials within out cluster. Workload Identity further strengthens security by enabling seamless authentication to Azure services without hardcoding credentials, making this approach an effective way to safeguard our cloud-native applications.

What is External Secrets Operator (ESO)?

The External Secrets Operator (ESO) is a Kubernetes operator that fetches secrets from external systems and injects them into Kubernetes secrets. It allows us to securely manage sensitive data outside of Kubernetes but still makes it available to our applications running in the Kubernetes cluster.

What is Azure Key Vault ?

Azure Key Vault is a cloud service provided by Microsoft Azure designed to securely store and manage sensitive information such as secrets, encryption keys, and certificates. It helps protect and control access to critical data that applications and services rely on, ensuring security, compliance, and ease of management.

How ESO Works with Azure Key Vault ?

1. Secret Management: ESO retrieves secrets from Azure Key Vault and automatically injects them as Kubernetes secrets.
2. Synchronization: ESO keeps Kubernetes secrets in sync with the secrets stored in Azure Key Vault. If a secret in Azure Key Vault changes, ESO updates the corresponding Kubernetes secret automatically.
3. Custom Resource Definitions (CRDs): ESO uses Kubernetes CRDs to define how secrets should be fetched and managed. These CRDs specify the source of the secret (in this case, Azure Key Vault) and how it should be mapped to a Kubernetes secret.

What is Azure Workload Identity?

Azure Workload Identity in the context of Kubernetes and Azure allows Kubernetes workloads to authenticate to Azure Active Directory (Azure AD) using their Kubernetes service account, rather than using secrets stored in Kubernetes. This identity-based authentication is more secure and eliminates the need for long-lived credentials.

Why integrate External Secrets Operator with Azure Key Vault using Azure Workload Identity authentication?

Imagine we're deploying a microservices-based application on Azure Kubernetes Service (AKS), where each microservice requires access to various secrets, such as database credentials and API keys. Managing these secrets directly within Kubernetes can be complex, particularly when addressing security, key rotation, and access control.

By leveraging External Secrets Operator (ESO) and Azure Key Vault, we can:

• Securely store all our secrets in Azure Key Vault.
• ESO using Azure Workload Identity authenticates to Azure Key Vault, eliminating the need to store credentials within Kubernetes.
• Automatically retrieve and sync these secrets into Kubernetes using ESO whenever needed.

This approach enhances security, simplifies the management of secrets, and ensures that our microservices always have access to the most up-to-date and secure secrets.

How Azure Workload Identity Works with ESO and Azure Key Vault ?

When using Workload Identity with ESO and Azure Key Vault, the process looks something like this:

1. Workload Identity Setup:
   1. Azure Setup: We create an Azure Managed Identity that has the necessary permissions to access Azure Key Vault.
   2. Kubernetes Setup: We configure Kubernetes to associate a specific Kubernetes service account with the Azure Managed Identity using Azure Workload Identity.
2. Service Account Binding: We bind the Kubernetes service account used by the External Secrets Operator to the Azure Managed Identity. This setup ensures that whenever the ESO needs to access Azure Key Vault, it automatically authenticates using the managed identity, eliminating the need for explicit credentials like API keys or client secrets.
3. External Secrets Operator Integration: ESO is configured to use the service account associated with the Azure Managed Identity. When it needs to retrieve secrets from Azure Key Vault, it uses the identity to request an OAuth2 token from Azure AD, which is then used to access Key Vault.
4. Secret Synchronization: With Workload Identity in place, ESO securely retrieves secrets from Azure Key Vault and syncs them into Kubernetes secrets without the need for any static credentials stored in the cluster.

In this quick guide, we will integrate External Secret Operator with Azure Key Vault to help fetch secret data using workload identity authentication from Azure Key Vault into a Kubernetes secret.

Prerequisite

• An Azure account with an active subscription, we can create an account for free.
• To run the commands in this quick guide, we can use the Bash environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.
• If we prefer we can run them locally in a Bash environment, with the following installed -
  1. Azure CLI version 2.47.0 or later, run az --version to see the currently installed version. To install or upgrade see How to install the Azure CLI | Microsoft Learn.
  2. Sign in to the Azure CLI using the az login command, for other sign in options, we refer to Sign in with Azure CLI — Login and Authentication | Microsoft Learn.
  3. Kubectl can be the same version as our Kubernetes cluster or within one minor version difference, either earlier or later. For example, if our cluster is running version 1.29, we can use kubectl versions 1.28, 1.29, or 1.30. To check our current kubectl version, we run kubectl version. For instructions on installing or updating kubectl, we refer to Installing or updating kubectl.
  4. Install Helm
• If we have multiple Azure subscriptions, we use the az account set command to select the appropriate subscription ID where resources should be billed. For more details, we refer to How to manage Azure subscriptions – Azure CLI.

We create the following environment variables for use throughout this quick guide, we will be deploying all resources in the same resource group.

# Create enivronment variables
export LOCATION="<location>"
export RESOURCE_GROUP="<ourResourceGroup>"  
export AKS_CLUSTER="<ourAKSClusterName>"  
export AKV_NAME="<ourKeyVaultName>"  
export AKV_SECRET_NAME="<ourKeyVaultSecretName>"   
export USER_ASSIGNED_IDENTITY_NAME="<ourManagedIdentityName>"  
export SERVICE_ACCOUNT_NAMESPACE="<ourServicesAccountNamespace>" 
export SERVICE_ACCOUNT_NAME="<ourServicesAccountName>"  
export FEDERATED_IDENTITY_CREDENTIAL_NAME="<ourFederatedIdentityName>"  



# Create some more environment variables
export AZURE_SUBSCRIPTION_ID="$(az account list --query "[?name=='my case sensitive subscription full name'].id" --output tsv)"   # Retrieves and sets the default Azure Subscription ID
export AZURE_TENANT_ID="$(az account show -s ${AZURE_SUBSCRIPTION_ID} --query tenantId -otsv)"   # Retrieves and sets the Azure Tenant ID associated with the subscription

Set up an AKS cluster with OIDC issuer enabled

Using Azure CLI we are going to set up an a resource group and an AKS instance with OIDC issuer enabled and retrieve the OIDC Issuer URL for use later.

# Create a resource group
az group create -l "${LOCATION}" -n ${RESOURCE_GROUP}
# Create a an AKS cluster with OIDC issuer enabled
az aks create -g ${RESOURCE_GROUP} -n ${AKS_CLUSTER} --node-count 1 --enable-oidc-issuer --generate-ssh-keys
# Retreive OIDC Issuer URL 
export SERVICE_ACCOUNT_ISSUER="$(az aks show -n $AKS_CLUSTER -g $RESOURCE_GROUP --query "oidcIssuerProfile.issuerUrl" -otsv)"

Connect to the AKS cluster

The az aks get-credentials command lets us get the access credentials for an AKS cluster and merges them into the kubeconfig file.

az aks get-credentials --resource-group ${RESOURCE_GROUP} --name ${AKS_CLUSTER}

Install External Secret Operator and Azure AD Workload Identity on the AKS cluster

Using Helm we will install the External Secret Operator and Azure AD Workload Identity. Azure AD Workload Identity uses a mutating admission webhook to project a signed service account token to our workload’s volume.

# Adds the 'external-secrets' and 'azure-workload-identity' Helm repository to Helm configuration
helm repo add external-secrets https://charts.external-secrets.io   
helm repo add azure-workload-identity https://azure.github.io/azure-workload-identity/charts
# Updates local Helm repository cache with the latest information from all added repositories
helm repo update  



# Installs a Helm chart with the release name 'external-secrets' from the 'external-secrets' repository
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--create-namespace \
#--set installCRDs=true 



# Installs a Helm chart with the release name 'workload-identity-webhook' from the 'azure-workload-identity' repository
helm install workload-identity-webhook azure-workload-identity/workload-identity-webhook \
   --namespace azure-workload-identity-system \
   --create-namespace \
   --set azureTenantID="${AZURE_TENANT_ID}"

Set up an Azure Key Vault with a secret

Here we create an Azure Key Vault resource with Role Based Access Control enabled, we also created a secret in the Key Vault.

az keyvault create --resource-group "${RESOURCE_GROUP}" \
   --location "${LOCATION}" \
   --name "${AKV_NAME}" \
   --enable-rbac-authorization


# Create some more environment variable
export AKV_ID="$(az keyvault show -n ${AKV_NAME} -g ${RESOURCE_GROUP} --query "id" -otsv)"

Assign the Key Vault Officer role to the account we are signed into the Azure CLI with, this role assignment will be at the Key Vault resource scope which will allow the account to be able to create secrets in the newly created key vault
N.B. we will be replacing userEmail with the email address of the account we are signed into the Azure CLI with.

az role assignment create --assignee "userEmail"  \
    --role "Key Vault Secrets Officer" \
    --scope "subscriptions/${AZURE_SUBSCRIPTION_ID}/resourcegroups/${RESOURCE_GROUP}/providers/microsoft.keyvault/vaults/${AKV_NAME}"



# Create a secret in the Key Vault
az keyvault secret set --vault-name "${AKV_NAME}" \
   --name "${AKV_SECRET_NAME}" \
   --value "This is confidential\!"

Set up a user-assigned managed identity and grant permission to read the secret

We create a user-assigned managed identity and grant it permission to only the secret we created earlier.

# Create a user-assigned managed identity
az identity create --name "${USER_ASSIGNED_IDENTITY_NAME}" \
    --resource-group "${RESOURCE_GROUP}" \
    --location "${LOCATION}" \
    --subscription "${AZURE_SUBSCRIPTION_ID}"

# Create some more environment variables
export USER_ASSIGNED_IDENTITY_PRINCIPAL_ID="$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'principalId' --output tsv)"

export USER_ASSIGNED_IDENTITY_CLIENT_ID="$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'clientId' --output tsv)"

# Assign the managed identity the Key Vault Secrets User role to the secret we created earlier
az role assignment create --assignee-object-id "${USER_ASSIGNED_IDENTITY_PRINCIPAL_ID}" \
    --role "Key Vault Secrets User" \
    --scope "subscriptions/${AZURE_SUBSCRIPTION_ID}/resourcegroups/${RESOURCE_GROUP}/providers/microsoft.keyvault/vaults/${AKV_NAME}/secrets/${AKV_SECRET_NAME}" \
    --assignee-principal-type ServicePrincipal

Create a Kubernetes Service Account

We create a service account, annotate it with the client ID of the user-assigned managed identity and the tenant ID.

cat << EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: ${USER_ASSIGNED_IDENTITY_CLIENT_ID}
    azure.workload.identity/tenant-id: ${AZURE_TENANT_ID}
  name: "${SERVICE_ACCOUNT_NAME}"
  namespace: "${SERVICE_ACCOUNT_NAMESPACE}"
EOF

Establish trust between AKS and Azure AD

We create a federated identity credential linking the managed identity, the service account issuer, and the subject.

az identity federated-credential create \
      --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} \
      --identity-name "${USER_ASSIGNED_IDENTITY_NAME}" \
      --resource-group "${RESOURCE_GROUP}" \
      --issuer "${SERVICE_ACCOUNT_ISSUER}" \
      --subject system:serviceaccount:"${SERVICE_ACCOUNT_NAMESPACE}":"${SERVICE_ACCOUNT_NAME}" \
      --audience api://AzureADTokenExchange

Set up SecretStore and ExternalSecrets resources.

We create a SecretStore resource that establishes a connection to the Key vault we created.

# The manifest defines a SecretStore resource in Kubernetes, which specifies how to retrieve secrets 
# from an Azure Key Vault using the External Secrets Operator (ESO) with Workload Identity.
cat << EOF | kubectl create -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-secret-store
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity # Specifies that Workload Identity will be used for authentication.
      vaultUrl: "https://${AKV_NAME}.vault.azure.net" # The URL of the Azure Key Vault from which secrets will be retrieved.
      serviceAccountRef:
        name: ${SERVICE_ACCOUNT_NAME} # References the Kubernetes service account that is bound to the Azure Managed Identity.
EOF

External Secret Operator uses the ExternalSecret resource to sync the secret from the Key vault into a Kubernetes secret object.

# The manifest defines an ExternalSecret resource, which specifies how to sync a secret from Azure Key Vault 
# into a Kubernetes secret using the External Secrets Operator (ESO).
cat << EOF | kubectl create -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: azure-external-secret
spec:
  refreshInterval: 3m # Specifies how often the ESO should check for updates to the secret in Azure Key Vault.
  secretStoreRef:
    kind: SecretStore # References the SecretStore resource that defines how to connect to Azure Key Vault.
    name: azure-secret-store # The name of the SecretStore that will be used (must match the name defined in the SecretStore manifest).
  target:
    name: azure-secret-to-be-created # The name of the Kubernetes secret that will be created or updated with the external secret data.
    creationPolicy: Owner # Defines the creation policy; "Owner" means that the ExternalSecret controls the lifecycle of the Kubernetes secret.
  data:
  - secretKey: "${AKV_SECRET_NAME}" # The key under which the secret will be stored in the Kubernetes secret.
    remoteRef:
      key: "${AKV_SECRET_NAME}" # The key name in Azure Key Vault from which the secret value will be retrieved.
EOF

We verify that our Secret has been created, by running the command

kubectl get secret azure-secret-to-be-created -o jsonpath='{.data.*}' | base64 -d

Output of the command

This is confidential\! # this should match the value of the secret created in Azure Key Vault. 

We will create a pod that will consume the secret data through a volume.

cat << EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: secret-test-pod
spec:
  containers:
    - name: test-container
      image: nginx
      volumeMounts:
        # name must match the volume name below
        - name: secret-volume
          mountPath: /etc/secret-volume
          readOnly: true
  # The secret data is exposed to Containers in the Pod through a Volume.
  volumes:
    - name: secret-volume
      secret:
        secretName: azure-secret-to-be-created
EOF

Get a shell into the Container that is running in the Pod.

kubectl exec -it secret-test-pod -- sh

The secret data is exposed to the Container through a Volume mounted under /etc/secret-volume.

In our shell, let’s list the files in the /etc/secret-volume directory:

ls /etc/secret-volume
thesecret #output of the ls command - this should match the name of the sceret created in Azure Key Vault.

In our shell, let’s display the contents of the thesecret file.

cat /etc/secret-volume/thesecret 
This is confidential\! # output of the cat command, this should match the value of the secret created in Azure Key Vault. 

Clean up

az aks stop --resource-group ${RESOURCE_GROUP} --name ${AKS_CLUSTER}
az group delete -n ${RESOURCE_GROUP}

Conclusion

Using External Secrets Operator with Azure Key Vault and Azure Workload Identity is a best practice for securely managing secrets in Kubernetes environments. It provides a robust, scalable, and secure solution that aligns with modern cloud-native practices, ensuring that our applications can securely access the secrets they need without exposing sensitive credentials. This approach simplifies secret management, enhances security, and supports compliance efforts, making it an ideal solution for managing sensitive data in the cloud.