Desplegar AGIC en AKS utilizando workload identity

Recientemente se anunció una nueva forma de autenticarse desde dentro de tu clúster de AKS para interactuar con tus recursos en Azure llamada workload identity. Hasta ahora las alternativas eran service principal o Azure AD Pod Identity, para el cual era necesario desplegar algunos recursos personalizados para su funcionamiento. Con workload identity, ya en GA, conseguimos el mismo efecto con una integración nativa, haciendo uso de Service Account Token Volume Projection y mejorando en algunos puntos como se detallan en el anuncio que te comparto. En este artículo quiero mostrarte cómo utilizarlo con Application Gateway Ingress Controller, soportado desde su versión 1.7.

Desplegar un cluster de prueba

Para poder hacer uso de workload identity, es necesario crear tu clúster con un par de parámetros adicionales:

# Variables
RESOURCE_GROUP="agic-workload-identity"
AKS_NAME="agic-workload-identity"
LOCATION="northeurope"
AKS_SUBNET="aks-subnet"
VNET_NAME="aks-vnet"
# Create resource group
az group create --name $RESOURCE_GROUP --location $LOCATION
RESOURCE_GROUP_ID=$(az group show --name $RESOURCE_GROUP --query id -o tsv)
# Create AKS cluster
az network vnet create \
--resource-group $RESOURCE_GROUP \
--name $VNET_NAME \
--address-prefixes 10.0.0.0/8 \
--subnet-name $AKS_SUBNET \
--subnet-prefix 10.10.0.0/16
# Get VNET id
SUBNET_ID=$(az network vnet subnet show -g $RESOURCE_GROUP -n $AKS_SUBNET --vnet-name $VNET_NAME --query id -o tsv)
# Create a user identity for the AKS
az identity create --name $AKS_NAME-identity --resource-group $RESOURCE_GROUP
# Get managed identity ID
IDENTITY_ID=$(az identity show --name $AKS_NAME-identity --resource-group $RESOURCE_GROUP --query id -o tsv)
IDENTITY_CLIENT_ID=$(az identity show --name $AKS_NAME-identity --resource-group $RESOURCE_GROUP --query clientId -o tsv)
# Create the AKS cluster
time az aks create \
--resource-group $RESOURCE_GROUP \
--name $AKS_NAME \
--node-vm-size Standard_B4ms \
--network-plugin azure \
--vnet-subnet-id $SUBNET_ID \
--enable-managed-identity \
--assign-identity $IDENTITY_ID \
--enable-oidc-issuer \
--enable-workload-identity
# Get the OIDC Issuer URL and save it to an environmental variable
AKS_OIDC_ISSUER=$(az aks show -n $AKS_NAME -g $RESOURCE_GROUP --query "oidcIssuerProfile.issuerUrl" -o tsv)

Además de crear la red virtual por mi cuenta (opcional), hago uso de dos nuevos parámetros: –enable-oidc-issuer y –enable-workload-identity. También es posible actualizar un clúster existente con estos parámetros.

Crear un Application Gateway WAF v2

Para este ejemplo, he creado un Application Gateway WAF v2 con los siguientes comandos:

##########################################
##### Create Application Gateway ####
##########################################
APPGW_SUBNET="appgw-subnet"
## Create subnet for application gateway
az network vnet subnet create \
--resource-group $RESOURCE_GROUP \
--vnet-name $VNET_NAME \
--name $APPGW_SUBNET \
--address-prefixes 10.20.0.0/16
### Application Gateway for dev ###
APP_GW_NAME="app-gw"
# Create public ip
az network public-ip create \
--resource-group $RESOURCE_GROUP \
--name $APP_GW_NAME-public-ip \
--allocation-method Static \
--sku Standard
# Create a WAF policy
GENERAL_WAF_POLICY="general-waf-policies"
az network application-gateway waf-policy create \
--name $GENERAL_WAF_POLICY \
--resource-group $RESOURCE_GROUP \
--type OWASP \
--version 3.2
# Get WAF policy ID
WAF_POLICY_ID=$(az network application-gateway waf-policy show --name $GENERAL_WAF_POLICY --resource-group $RESOURCE_GROUP --query id -o tsv)
# Create the app gateway
time az network application-gateway create \
--resource-group $RESOURCE_GROUP \
--name $APP_GW_NAME \
--location $LOCATION \
--vnet-name $VNET_NAME \
--subnet $APPGW_SUBNET \
--public-ip-address $APP_GW_NAME-public-ip \
--sku WAF_v2 \
--capacity 1 \
--priority 1 \
--waf-policy $GENERAL_WAF_POLICY
APP_GW_ID=$(az network application-gateway show --name $APP_GW_NAME --resource-group $RESOURCE_GROUP --query id -o tsv)

De esta forma puedo probar el escenario con WAF si quisiera.

Desplegar AGIC con workload identity

Para terminar con la configuración, lo último que necesitas es desplegar AGIC utilizando workload identity como método para que este pueda hacer las peticiones a la API REST de Azure Resource Manager y así poder controlar el Application Gateway que acabas de crear:

#####################################################
######## Configure AGIC with workload identity ######
#####################################################
# Get AKS credentials
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_NAME
# Create a managed identity for AGIC
AGIC_IDENTITY_NAME="agic-identity"
az identity create \
--name $AGIC_IDENTITY_NAME \
--resource-group $RESOURCE_GROUP \
--location $LOCATION
AGIC_IDENTITY_CLIENT_ID="$(az identity show -g $RESOURCE_GROUP -n $AGIC_IDENTITY_NAME --query clientId -o tsv)"
# Assign Contributor role to the identity over the application gateway
az role assignment create \
    --role "Contributor" \
    --assignee $AGIC_IDENTITY_CLIENT_ID \
    --scope $APP_GW_ID
az role assignment create \
    --role "Reader" \
    --assignee $AGIC_IDENTITY_CLIENT_ID \
    --scope $RESOURCE_GROUP_ID
# Check role assignments for the user identities
az role assignment list --assignee $AGIC_IDENTITY_CLIENT_ID --all -o table
# Add the AGIC Helm repository
helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update
# Install Helm chart application-gateway-kubernetes-ingress with the helm-config.yaml configuration from the previous step
helm install agic -f workload-identity/config.yaml application-gateway-kubernetes-ingress/ingress-azure -n kube-system

La configuración que se aplica en el último punto es la siguiente:

# This file contains the essential configs for the ingress controller helm chart
# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3
################################################################################
# Specify which application gateway the ingress controller will manage
#
appgw:
    subscriptionId: <SUBSCRIPTION_ID>
    resourceGroup: agic-workload-identity
    name: app-gw
    usePrivateIP: false
    # Setting appgw.shared to "true" will create an AzureIngressProhibitedTarget CRD.
    # This prohibits AGIC from applying config for any host/path.
    # Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
    shared: false
################################################################################
# Specify which kubernetes namespace the ingress controller will watch
# Default value is "default"
# Leaving this variable out or setting it to blank or empty string would
# result in Ingress Controller observing all acessible namespaces.
#
# kubernetes:
#     watchNamespace: default

################################################################################
# Specify the authentication with Azure Resource Manager
#
armAuth:
    type: workloadIdentity    
    identityClientID: <VALUE_OF_AGIC_IDENTITY_CLIENT_ID>
################################################################################
# Specify if the cluster is RBAC enabled or not
rbac:
    enabled: true # true/false

En ella, como ves, se completa con los mismos parámetros que lo haríamos con un service principal o para Azure AD Pod Identity, solo que en este caso indicamos, en la sección armAuth, el tipo workloadIdentity y utilizamos identityClientID para asignar el valor de nuestra variable AGIC_IDENTITY_CLIENT_ID, recuperada anteriormente.

Una vez desplegado podemos comprobar el estado con estos comandos:

# Check helm release
helm list -n kube-system
# Check AGIC pods
kubectl get pods -n kube-system

Sin embargo, todavía nos queda un paso más, ya que de lo contrario AGIC no funcionará (si revisas sus logs verás que da error), y es asociar el service account creado por este a la managed identity y el issuer generado por AKS en el momento de su creación a través federated identities:

# Get service account in the current namespace
AGIC_SERVICE_ACCOUNT=$(kubectl get sa -n kube-system -l azure.workload.identity/use=true -o jsonpath='{.items[0].metadata.name}')
SERVICE_ACCOUNT_NAMESPACE="kube-system"
# Create the federated identity credential between the managed identity, the service account issuer, and the subject
# https://learn.microsoft.com/en-us/graph/api/resources/federatedidentitycredentials-overview?view=graph-rest-1.0
FEDERATED_IDENTITY_CREDENTIAL_NAME="federated-identity-credential"
az identity federated-credential create \
--name ${AGIC_SERVICE_ACCOUNT} \
--identity-name ${AGIC_IDENTITY_NAME} \
--resource-group ${RESOURCE_GROUP} \
--issuer ${AKS_OIDC_ISSUER} \
--subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${AGIC_SERVICE_ACCOUNT}

Para comprobar que la configuración ha sido satisfactoria, puedes revisar los logs de AGIC con el siguiente comando:

# Check AGIC logs
kubectl logs $(kubectl get pod -n kube-system -l release=agic-dev -o jsonpath='{.items[0].metadata.name}') -n kube-system

También puedes desplegar una aplicación de ejemplo, como la siguiente, para comprobar que efectivamente AGIC puede modificar nuestro Application Gateway utilizando workload identity como método:

apiVersion: v1
kind: Pod
metadata:
  name: aspnetapp
  labels:
    app: aspnetapp
spec:
  containers:
  - image: "mcr.microsoft.com/dotnet/samples:aspnetapp"
    name: aspnetapp-image
    ports:
    - containerPort: 80
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: aspnetapp
spec:
  selector:
    app: aspnetapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: aspnetapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    # appgw.ingress.kubernetes.io/backend-path-prefix: "/test" #allows the backend path specified in an ingress resource to be rewritten with prefix specified in this annotation. It allows users to expose services whose endpoints are different than endpoint names used to expose a service in an ingress resource.
    appgw.ingress.kubernetes.io/ssl-redirect: "false" #redirects HTTP requests to HTTPS
    appgw.ingress.kubernetes.io/connection-draining: "true" # This annotation allows us to specify whether to enable connection draining. Connection draining allows the Application Gateway to stop sending new requests to the backend when the backend is being deleted or scaled in. The default value is false.
    appgw.ingress.kubernetes.io/connection-draining-timeout: "60" # This annotation allows us to specify the time in seconds to wait for existing requests to drain. The default value is 60 seconds.
    appgw.ingress.kubernetes.io/cookie-based-affinity: "true" # This annotation allows us to specify whether to enable cookie-based affinity. Cookie-based affinity allows the Application Gateway to direct all requests from a client to the same backend instance. The default value is false.
    appgw.ingress.kubernetes.io/request-timeout: "30" # This annotation allows us to specify the timeout in seconds for the Application Gateway to wait for the response from the backend. The default value is 30 seconds.
    appgw.ingress.kubernetes.io/use-private-ip: "false" # This annotation allows us to specify whether to use the private IP address of the backend instances. The default value is false.
    appgw.ingress.kubernetes.io/backend-protocol: "http" # This annotation allows us to specify the protocol used to communicate with the backend. The default value is http.
    # appgw.ingress.kubernetes.io/rewrite-rule-set: "redirect-to-https" # This annotation allows us to specify the name of the rewrite rule set to use for the ingress. The default value is redirect-to-https.
    appgw.ingress.kubernetes.io/override-frontend-port: "9090" # This annotation allows us to specify the port to use for the frontend listener. The default value is 80.
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          service:
            name: aspnetapp
            port:
              number: 80
        pathType: Exact

Para finalizar, puedes recuperar la IP pública de tu Application Gateway e intentar acceder a través del puerto 9090:

# Get the public IP address of the application gateway
APP_GW_PUBLIC_IP=$(az network public-ip show --resource-group $RESOURCE_GROUP --name $APP_GW_NAME-public-ip --query ipAddress -o tsv)
# Check application
echo http://$APP_GW_PUBLIC_IP:9090

¡Saludos!