Understand Azure Policy for Kubernetes clusters

Azure Policy extends Gatekeeper v3, an admission controller webhook for Open Policy Agent (OPA), to apply at-scale enforcements and safeguards on your cluster components in a centralized, consistent manner. Cluster components include pods, containers, and namespaces.

Azure Policy makes it possible to manage and report on the compliance state of your Kubernetes cluster components from one place. By using Azure Policy's Add-on or Extension, governing your cluster components is enhanced with Azure Policy features, like the ability to use selectors and overrides for safe policy rollout and rollback.

Azure Policy for Kubernetes supports the following cluster environments:

• Azure Kubernetes Service (AKS), through Azure Policy's Add-on for AKS
• Azure Arc enabled Kubernetes, through Azure Policy's Extension for Arc

Overview

By installing Azure Policy's add-on or extension on your Kubernetes clusters, Azure Policy enacts the following functions:

• Checks with Azure Policy service for policy assignments to the cluster.
• Deploys policy definitions into the cluster as constraint template and constraint custom resources or as a mutation template resource (depending on policy definition content).
• Reports auditing and compliance details back to Azure Policy service.

To enable and use Azure Policy with your Kubernetes cluster, take the following actions:

1. Configure your Kubernetes cluster and install the Azure Kubernetes Service (AKS) add-on or Azure Policy's Extension for Arc-enabled Kubernetes clusters (depending on your cluster type).

   Note

   For common issues with installation, see Troubleshoot - Azure Policy Add-on.

2. Create or use a sample Azure Policy definition for Kubernetes

3. Assign a definition to your Kubernetes cluster

4. Wait for validation

5. Logging and troubleshooting

6. Review limitations and recommendations in our FAQ section

Install Azure Policy Add-on for AKS

The Azure Policy Add-on for AKS is part of Kubernetes version 1.27 with long term support (LTS).

Prerequisites

1. Register the resource providers and preview features.

   • Azure portal:

     Register the Microsoft.PolicyInsights resource providers. For steps, see Resource providers and types.

   • Azure CLI:

     # Log in first with az login if you're not using Cloud Shell
     
     # Provider register: Register the Azure Policy provider
     az provider register --namespace Microsoft.PolicyInsights
     

2. You need the Azure CLI version 2.12.0 or later installed and configured. To find the version, run the az --version command. If you need to install or upgrade, see How to install the Azure CLI.

3. The AKS cluster must be a supported Kubernetes version in Azure Kubernetes Service (AKS). Use the following script to validate your AKS cluster version:

   # Log in first with az login if you're not using Cloud Shell
   
   # Look for the value in kubernetesVersion
   az aks list
   

4. Open ports for the Azure Policy add-on. The Azure Policy add-on uses these domains and ports to fetch policy definitions and assignments and report compliance of the cluster back to Azure Policy.

   Domain	Port
   data.policy.core.windows.net	443
   store.policy.core.windows.net	443
   login.microsoftonline.com	443
   dc.services.visualstudio.com	443

After the prerequisites are completed, install the Azure Policy Add-on in the AKS cluster you want to manage.

• Azure portal

  1. Launch the AKS service in the Azure portal by selecting All services, then searching for and selecting Kubernetes services.

  2. Select one of your AKS clusters.

  3. Select Policies on the left side of the Kubernetes service page.

  4. In the main page, select the Enable add-on button.

• Azure CLI

  # Log in first with az login if you're not using Cloud Shell
  
  az aks enable-addons --addons azure-policy --name MyAKSCluster --resource-group MyResourceGroup
  

To validate that the add-on installation was successful and that the azure-policy and gatekeeper pods are running, run the following command:

# azure-policy pod is installed in kube-system namespace
kubectl get pods -n kube-system

# gatekeeper pod is installed in gatekeeper-system namespace
kubectl get pods -n gatekeeper-system

Lastly, verify that the latest add-on is installed by running this Azure CLI command, replacing with your resource group name and with the name of your AKS cluster: az aks show --query addonProfiles.azurepolicy -g -n . The result should look similar to the following output for clusters using service principals:

{
  "config": null,
  "enabled": true,
  "identity": null
}

And the following output for clusters using managed identity:

 {
   "config": null,
   "enabled": true,
   "identity": {
     "clientId": "########-####-####-####-############",
     "objectId": "########-####-####-####-############",
     "resourceId": ""
   }
 }

Install Azure Policy Extension for Azure Arc enabled Kubernetes

Azure Policy for Kubernetes makes it possible to manage and report on the compliance state of your Kubernetes clusters from one place. With Azure Policy's Extension for Arc-enabled Kubernetes clusters, you can govern your Arc-enabled Kubernetes cluster components, like pods and containers.

This article describes how to create, show extension status, and delete the Azure Policy for Kubernetes extension.

For an overview of the extensions platform, see Azure Arc cluster extensions.

Prerequisites

If you already deployed Azure Policy for Kubernetes on an Azure Arc cluster using Helm directly without extensions, follow the instructions to delete the Helm chart. After the deletion is done, you can then proceed.

1. Ensure your Kubernetes cluster is a supported distribution.

   Note

   Azure Policy for Arc extension is supported on the following Kubernetes distributions.

2. Ensure you met all the common prerequisites for Kubernetes extensions listed here including connecting your cluster to Azure Arc.

   Note

   Azure Policy extension is supported for Arc enabled Kubernetes clusters in these regions.

3. Open ports for the Azure Policy extension. The Azure Policy extension uses these domains and ports to fetch policy definitions and assignments and report compliance of the cluster back to Azure Policy.

   Domain	Port
   data.policy.core.windows.net	443
   store.policy.core.windows.net	443
   login.windows.net	443
   dc.services.visualstudio.com	443

4. Before you install the Azure Policy extension or enabling any of the service features, your subscription must enable the Microsoft.PolicyInsights resource providers.

   Note

   To enable the resource provider, follow the steps in Resource providers and types or run either the Azure CLI or Azure PowerShell command.

   • Azure CLI

     # Log in first with az login if you're not using Cloud Shell
     # Provider register: Register the Azure Policy provider
     az provider register --namespace 'Microsoft.PolicyInsights'
     

   • Azure PowerShell

     # Log in first with Connect-AzAccount if you're not using Cloud Shell
     
     # Provider register: Register the Azure Policy provider
     Register-AzResourceProvider -ProviderNamespace 'Microsoft.PolicyInsights'
     

Create Azure Policy extension

To create an extension instance, for your Arc enabled cluster, run the following command substituting <> with your values:

az k8s-extension create --cluster-type connectedClusters --cluster-name  --resource-group  --extension-type Microsoft.PolicyInsights --name 

Example:

az k8s-extension create --cluster-type connectedClusters --cluster-name my-test-cluster --resource-group my-test-rg --extension-type Microsoft.PolicyInsights --name azurepolicy

Example Output:

{
  "aksAssignedIdentity": null,
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": {},
  "configurationSettings": {},
  "customLocationSettings": null,
  "errorInfo": null,
  "extensionType": "microsoft.policyinsights",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-test-rg/providers/Microsoft.Kubernetes/connectedClusters/my-test-cluster/providers/Microsoft.KubernetesConfiguration/extensions/azurepolicy",
 "identity": {
    "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "tenantId": null,
    "type": "SystemAssigned"
  },
  "location": null,
  "name": "azurepolicy",
  "packageUri": null,
  "provisioningState": "Succeeded",
  "releaseTrain": "Stable",
  "resourceGroup": "my-test-rg",
  "scope": {
    "cluster": {
      "releaseNamespace": "kube-system"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": {
    "createdAt": "2021-10-27T01:20:06.834236+00:00",
    "createdBy": null,
    "createdByType": null,
    "lastModifiedAt": "2021-10-27T01:20:06.834236+00:00",
    "lastModifiedBy": null,
    "lastModifiedByType": null
  },
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": "1.1.0"
}

Show Azure Policy extension

To check the extension instance creation was successful, and inspect extension metadata, run the following command substituting <> with your values:

az k8s-extension show --cluster-type connectedClusters --cluster-name  --resource-group  --name 

Example:

az k8s-extension show --cluster-type connectedClusters --cluster-name my-test-cluster --resource-group my-test-rg --name azurepolicy

To validate that the extension installation was successful and that the azure-policy and gatekeeper pods are running, run the following command:

# azure-policy pod is installed in kube-system namespace
kubectl get pods -n kube-system

# gatekeeper pod is installed in gatekeeper-system namespace
kubectl get pods -n gatekeeper-system

Delete Azure Policy extension

To delete the extension instance, run the following command substituting <> with your values:

az k8s-extension delete --cluster-type connectedClusters --cluster-name  --resource-group  --name 

Create a policy definition

The Azure Policy language structure for managing Kubernetes follows that of existing policy definitions. There are sample definition files available to assign in Azure Policy's built-in policy library that can be used to govern your cluster components.

Azure Policy for Kubernetes also support custom definition creation at the component-level for both Azure Kubernetes Service clusters and Azure Arc-enabled Kubernetes clusters. Constraint template and mutation template samples are available in the Gatekeeper community library. Azure Policy's Visual Studio Code Extension can be used to help translate an existing constraint template or mutation template to a custom Azure Policy policy definition.

With a Resource Provider mode of Microsoft.Kubernetes.Data, the effects audit, deny, disabled, and mutate are used to manage your Kubernetes clusters.

Audit and deny must provide details properties specific to working with OPA Constraint Framework and Gatekeeper v3.

As part of the details.templateInfo or details.constraintInfo properties in the policy definition, Azure Policy passes the URI or Base64Encoded value of these CustomResourceDefinitions(CRD) to the add-on. Rego is the language that OPA and Gatekeeper support to validate a request to the Kubernetes cluster. By supporting an existing standard for Kubernetes management, Azure Policy makes it possible to reuse existing rules and pair them with Azure Policy for a unified cloud compliance reporting experience. For more information, see What is Rego?

Assign a policy definition

To assign a policy definition to your Kubernetes cluster, you must be assigned the appropriate Azure role-based access control (Azure RBAC) policy assignment operations. The Azure built-in roles Resource Policy Contributor and Owner have these operations. To learn more, see Azure RBAC permissions in Azure Policy.

Find the built-in policy definitions for managing your cluster using the Azure portal with the following steps. If using a custom policy definition, search for it by name or the category that you created it with.

1. Start the Azure Policy service in the Azure portal. Select All services in the left pane and then search for and select Policy.

2. In the left pane of the Azure Policy page, select Definitions.

3. From the Category dropdown list box, use Select all to clear the filter and then select Kubernetes.

4. Select the policy definition, then select the Assign button.

5. Set the Scope to the management group, subscription, or resource group of the Kubernetes cluster where the policy assignment applies.

   Note

   When assigning the Azure Policy for Kubernetes definition, the Scope must include the cluster resource.

6. Give the policy assignment a Name and Description that you can use to identify it easily.

7. Set the Policy enforcement to one of the following values:

   • Enabled - Enforce the policy on the cluster. Kubernetes admission requests with violations are denied.

   • Disabled - Don't enforce the policy on the cluster. Kubernetes admission requests with violations aren't denied. Compliance assessment results are still available. When you roll out new policy definitions to running clusters, Disabled option is helpful for testing the policy definition as admission requests with violations aren't denied.

8. Select Next.

9. Set parameter values

   • To exclude Kubernetes namespaces from policy evaluation, specify the list of namespaces in parameter Namespace exclusions. The recommendation is to exclude: kube-system, gatekeeper-system, and azure-arc.

10. Select Review + create.

Alternately, use the Assign a policy - Portal quickstart to find and assign a Kubernetes policy. Search for a Kubernetes policy definition instead of the sample audit vms.

Policy evaluation

The add-on checks in with Azure Policy service for changes in policy assignments every 15 minutes. During this refresh cycle, the add-on checks for changes. These changes trigger creates, updates, or deletes of the constraint templates and constraints.

In a Kubernetes cluster, if a namespace has the cluster-appropriate label, the admission requests with violations aren't denied. Compliance assessment results are still available.

• Azure Arc-enabled Kubernetes cluster: admission.policy.azure.com/ignore

Every 15 minutes, the add-on calls for a full scan of the cluster. After gathering details of the full scan and any real-time evaluations by Gatekeeper of attempted changes to the cluster, the add-on reports the results back to Azure Policy for inclusion in compliance details like any Azure Policy assignment. Only results for active policy assignments are returned during the audit cycle. Audit results can also be seen as violations listed in the status field of the failed constraint. For details on Non-compliant resources, see Component details for Resource Provider modes.

Some other considerations:

• If the cluster subscription is registered with Microsoft Defender for Cloud, then Microsoft Defender for Cloud Kubernetes policies are applied on the cluster automatically.

• When a deny policy is applied on cluster with existing Kubernetes resources, any preexisting resource that isn't compliant with the new policy continues to run. When the non-compliant resource gets rescheduled on a different node the Gatekeeper blocks the resource creation.

• When a cluster has a deny policy that validates resources, the user doesn't get a rejection message when creating a deployment. For example, consider a Kubernetes deployment that contains replicasets and pods. When a user executes kubectl describe deployment $MY_DEPLOYMENT, it doesn't return a rejection message as part of events. However, kubectl describe replicasets.apps $MY_DEPLOYMENT returns the events associated with rejection.

input_containers[c] {
   c := input.review.object.spec.initContainers[_]
}

Constraint template conflicts

If constraint templates have the same resource metadata name, but the policy definition references the source at different locations, the policy definitions are considered to be in conflict. Example: Two policy definitions reference the same template.yaml file stored at different source locations like the Azure Policy template store (store.policy.core.windows.net) and GitHub.

When policy definitions and their constraint templates are assigned but aren't already installed on the cluster and are in conflict, they're reported as a conflict and aren't installed into the cluster until the conflict is resolved. Likewise, any existing policy definitions and their constraint templates that are already on the cluster that conflicts with newly assigned policy definitions continue to function normally. If an existing assignment is updated and there's a failure to sync the constraint template, the cluster is also marked as a conflict. For all conflict messages, see AKS Resource Provider mode compliance reasons

Logging

As a Kubernetes controller/container, both the azure-policy and gatekeeper pods keep logs in the Kubernetes cluster. In general, azure-policy logs can be used to troubleshoot issues with policy ingestion onto the cluster and compliance reporting. The gatekeeper-controller-manager pod logs can be used to troubleshoot runtime denies. The gatekeeper-audit pod logs can be used to troubleshoot audits of existing resources. The logs can be exposed in the Insights page of the Kubernetes cluster. For more information, see Monitor your Kubernetes cluster performance with Azure Monitor for containers.

To view the add-on logs, use kubectl:

# Get the azure-policy pod name installed in kube-system namespace
kubectl logs  -n kube-system

# Get the gatekeeper pod name installed in gatekeeper-system namespace
kubectl logs  -n gatekeeper-system

If you're attempting to troubleshoot a particular ComplianceReasonCode that is appearing in your compliance results, you can search the azure-policy pod logs for that code to see the full accompanying error.

For more information, see Debugging Gatekeeper in the Gatekeeper documentation.

View Gatekeeper artifacts

After the add-on downloads the policy assignments and installs the constraint templates and constraints on the cluster, it annotates both with Azure Policy information like the policy assignment ID and the policy definition ID. To configure your client to view the add-on related artifacts, use the following steps:

1. Set up kubeconfig for the cluster.

   For an Azure Kubernetes Service cluster, use the following Azure CLI:

   # Set context to the subscription
   az account set --subscription 
   
   # Save credentials for kubeconfig into .kube in your home folder
   az aks get-credentials --resource-group  --name 
   

2. Test the cluster connection.

   Run the kubectl cluster-info command. A successful run has each service responding with a URL of where it's running.

View the add-on constraint templates

To view constraint templates downloaded by the add-on, run kubectl get constrainttemplates. Constraint templates that start with k8sazure are the ones installed by the add-on.

View the add-on mutation templates

To view mutation templates downloaded by the add-on, run kubectl get assign, kubectl get assignmetadata, and kubectl get modifyset.

Get Azure Policy mappings

To identify the mapping between a constraint template downloaded to the cluster and the policy definition, use kubectl get constrainttemplates