Edit

Prepare a cluster for Azure IoT Operations

You need an Azure Arc-enabled Kubernetes cluster to deploy Azure IoT Operations. This article explains how to prepare that cluster, with guidance for Ubuntu, Windows, Azure Local, vSphere Kubernetes Service (VKS), and Red Hat OpenShift Container Platform (OpenShift).

If you want to deploy Azure IoT Operations quickly and run a sample workload in a test environment, see the Quickstart: Run Azure IoT Operations in GitHub Codespaces with K3s.

Prerequisites

For multi-node deployments, use K3s on Ubuntu, AKS on Azure Local, vSphere Kubernetes Service (VKS), or OpenShift. AKS Edge Essentials on Windows supports single-node deployments only.

To prepare an Azure Arc-enabled Kubernetes cluster, you need:

  • An Azure subscription with either the Owner role or a combination of Contributor and User Access Administrator roles. You can check your access level by navigating to your subscription, selecting Access control (IAM) on the left-hand side of the Azure portal, and then selecting View my access. If you don't have an Azure subscription, create one for free before you begin.

  • The Azure CLI installed on your development machine. Check Available Azure CLI extensions for the minimum required version for the connectedk8s extension. Use az --version to check your version and az upgrade to update if necessary. For more information, see Install the Azure CLI.

  • The connectedk8s extension for the Azure CLI. Use the following command to add the extension or update it to the latest version:

    az extension add --upgrade --name connectedk8s

  • An Azure resource group. Only one Azure IoT Operations instance is supported per resource group. To create a new resource group, use the az group create command. For the list of currently supported Azure regions, see Supported regions.

    az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID

The Azure CLI examples in this article use environment variables so that you can set each value once and then copy and paste the commands as-is. If you're using the Azure IoT Operations Codespaces environment from the quickstart, these variables are already set for you and you can skip this step. Otherwise, set the following environment variables in your shell before you run the commands.

The following scripts set the most commonly used environment variables:

Environment variable Description
SUBSCRIPTION_ID The ID of the subscription that contains your Azure IoT Operations instance.
RESOURCE_GROUP The name of the resource group that contains your Azure IoT Operations instance.
AIO_INSTANCE_NAME The name of your Azure IoT Operations instance. To list your instances, run az iot ops list -o table.
CLUSTER_NAME The name of the Azure Arc-enabled Kubernetes cluster that hosts your instance.
LOCATION The Azure region to use for new resources, for example eastus. 
SUBSCRIPTION_ID=
RESOURCE_GROUP=
AIO_INSTANCE_NAME=
CLUSTER_NAME=
LOCATION=

You only need to set the variables that this article uses. This article might use additional environment variables for resource names that you choose. The article explains how to set them where they're introduced.

Create and Arc-enable a cluster

This section provides steps to create clusters in validated environments on Ubuntu, Windows, Azure Local, vSphere Kubernetes Service (VKS), and OpenShift.

To prepare a K3s Kubernetes cluster on Ubuntu:

  1. Create a single-node or multi-node K3s cluster. For examples, see the K3s quick-start guide or K3s related projects.

  2. Check that K3s installed kubectl. If not, follow the instructions to Install kubectl on Linux.

    kubectl version --client

  3. Follow the instructions to Install Helm.

  4. Create a K3s configuration YAML file in .kube/config:

    mkdir ~/.kube
sudo KUBECONFIG=~/.kube/config:/etc/rancher/k3s/k3s.yaml kubectl config view --flatten > ~/.kube/merged
mv ~/.kube/merged ~/.kube/config
chmod  0600 ~/.kube/config
export KUBECONFIG=~/.kube/config
#switch to k3s context
kubectl config use-context default
sudo chmod 644 /etc/rancher/k3s/k3s.yaml

  5. Run the following command to increase the user watch/instance limits.

    echo fs.inotify.max_user_instances=8192 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf

sudo sysctl -p

  6. For better performance, increase the file descriptor limit:

    echo fs.file-max = 100000 | sudo tee -a /etc/sysctl.conf

sudo sysctl -p

Arc-enable your K3s cluster

Connect your cluster to Azure Arc so that you can manage it remotely.

  1. From a machine that has kubectl access to your cluster, sign in to Azure CLI with your Microsoft Entra user account that has the required roles for the Azure subscription:

    az login

    If at any point you get an error that says Your device is required to be managed to access your resource, run az login again and make sure that you sign in interactively by using a browser.

  2. After you sign in, the Azure CLI shows all of your subscriptions and indicates your default subscription with an asterisk *. To continue with your default subscription, select Enter. Otherwise, type the number of the Azure subscription that you want to use.

  3. Register the required resource providers in your subscription.

    Note

    You need to run this step only once per subscription. To register resource providers, you need permission to do the /register/action operation, which subscription Contributor and Owner roles include. For more information, see Azure resource providers and types.

    az provider register -n "Microsoft.ExtendedLocation"
az provider register -n "Microsoft.Kubernetes"
az provider register -n "Microsoft.KubernetesConfiguration"
az provider register -n "Microsoft.IoTOperations"
az provider register -n "Microsoft.DeviceRegistry"
az provider register -n "Microsoft.SecretSyncController"

  4. Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it as part of your Azure resource group.

    az connectedk8s connect --name $CLUSTER_NAME -l $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID --enable-oidc-issuer --enable-workload-identity --disable-auto-upgrade

    To prevent unplanned updates to Azure Arc and the system Arc extensions that Azure IoT Operations uses as dependencies, this command disables autoupgrade. Instead, manually upgrade agents as needed.

    Important

    If your environment uses a proxy server or Azure Arc Gateway, modify the az connectedk8s connect command with your proxy information:

    1. Follow the instructions in either Connect using an outbound proxy server or Onboard Kubernetes clusters to Azure Arc with Azure Arc Gateway.
    2. Add 169.254.169.254 to the --proxy-skip-range parameter of the az connectedk8s connect command. Azure Device Registry uses this local endpoint to get access tokens for authorization.

    Azure IoT Operations doesn't support proxy servers that require a trusted certificate.

  5. Get the cluster's issuer URL.

    az connectedk8s show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query oidcIssuerProfile.issuerUrl --output tsv

    Save the output of this command to use in the next steps.

  6. Create a K3s config file.

    sudo nano /etc/rancher/k3s/config.yaml

  7. Add the following content to the config.yaml file, replacing the placeholder with your cluster's issuer URL.

    kube-apiserver-arg:
 - service-account-issuer=
 - service-account-max-token-expiration=24h

  8. Save the file and exit the nano editor.

  9. Prepare for enabling the Azure Arc service, custom location, on your Arc cluster by getting the custom location object ID and saving it as the environment variable, OBJECT_ID. You must sign in to Azure CLI with a Microsoft Entra user account, not a service principal, to run the command successfully. Run the following command exactly as written, without changing the GUID value.

    export OBJECT_ID=$(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv)

    Note

    If you receive the error: "Unable to fetch oid of 'custom-locations' app. Proceeding without enabling the feature. Insufficient privileges to complete the operation," then your service principal might lack the necessary permissions to retrieve the object ID of the custom location. Sign in to Azure CLI with a Microsoft Entra user account that meets the prerequisites. For more information, see Create and manage custom locations.

  10. Use the az connectedk8s enable-features command to enable the custom location feature on your Arc cluster. This command uses the OBJECT_ID environment variable that you saved in the previous step to set the value for the custom-locations-oid parameter. Run this command on the machine where you deployed the Kubernetes cluster:

    az connectedk8s enable-features -n $CLUSTER_NAME -g $RESOURCE_GROUP --custom-locations-oid $OBJECT_ID --features cluster-connect custom-locations

  11. Restart K3s.

    systemctl restart k3s

Configure Azure Container Storage enabled by Azure Arc

Features such as data flow local storage endpoints and the media connector optionally use Azure Container Storage enabled by Azure Arc (ACSA) to synchronize local data to the cloud. ACSA isn't installed as part of Azure IoT Operations, so you must install it separately.

To learn how to install ACSA on your Kubernetes cluster:

Next steps

Now that you have an Azure Arc-enabled Kubernetes cluster, you can deploy Azure IoT Operations.

  • Last updated on