Inside Look: How Azure Kubernetes Service supports OSSKU Migration between Ubuntu and Azure Linux

Microsoft recently announce General availability of OSSKU migration in Azure Kubernetes Service (AKS). This new capability allows users to take an existing AKS node pool and update the OSSKU for in-place migration between Ubuntu and Azure Linux. Previously, when OSSKUs were immutable, users had to create a new node pool and explicitly drain the workload, which was labor intensive and required additional VM quota.

In this blog post, we’ll dive deeper into how to use this feature, the technology stack that supports it, and some considerations to ensure a smooth upgrade.

Using OSSKU Migration

OSSKU migration is supported in az-cli, ARM/Bicep templates, and Terraform. All three options will put the affected node pool into an upgrade state, which will take a few minutes to resolve. During this time, the cluster will scale according to the maximum surge setting, and pods will be drained and scheduled on other VMs in the node pool or cluster.

• If you are using az-cli, the version should be: 2.61.0 Or, to trigger a migration with az-cli, run the following command on your node pool:

az aks nodepool update --resource-group myResourceGroup --cluster-name myAKSCluster --name myNodePool --os-sku AzureLinux

• If you are using the ARM/Bicep template, you will need to update the apiVersion. 2023-07-01 Or the latest version. Then update the ossku field in the agentPoolProfile section to “AzureLinux” and redeploy the template.
• If you are using Terraform, the azurerm provider version must be: v3.111.0 Or higher. Then update the os_sku field of the node pool to “AzureLinux” and redeploy the Terraform plan.

How it works

Figure 1: Flowchart of AKS OSSKU migration task

When sending a request to AKS (1) If we notice that the OSSKU value of a node pool has changed, we perform additional validation to ensure that the change is allowed.

• OSSKU migration cannot change node pool names.
• Only Ubuntu and AzureLinux are supported as OSSKU targets.
• Ubuntu node pools with UseGPUDedicatedVHD enabled cannot change OSSKU.
• Ubuntu node pools with CVM 20.04 enabled cannot change OSSKU.
• AzureLinux node pools with Kata enabled cannot change OSSKU.
• Windows node pools cannot change OSSKU.

Once all these conditions are met, AKS will move the node pool to the Upgrade state and select the latest available image for the newly selected OSSKU. The exact same flow will then be followed. Node Image UpgradeExpand node pool based on maximum surge value (2)Then, replace the images of the existing VMs one by one until each node is on the latest image for the newly selected OSSKU. (3). Once all VMs are upgraded to the new image, AKS removes the surge node and notifies the caller again that the upgrade is complete. (4).

Things to consider

There are two very important things to check before running an OSSKU migration on your production cluster:

• Before performing the migration on the remaining node pools, deploy node pools with the new target OSSKU in both your development and production environments to ensure that everything works as expected on the new OSSKU.
• Ensure that your workload has sufficient pod disruption budget to allow AKS to move pods between VMs during the upgrade. This is required for OSSKU migrations and all AKS node image upgrades to safely move workloads around the cluster while nodes are restarted. For more information about troubleshooting PDB errors during an upgrade, see: This document.

conclusion

With the public preview, several teams within Microsoft have leveraged OSSKU migration to seamlessly migrate workloads to Azure Linux OSSKUs without requiring large surge capacity or manual intervention within the cluster. We look forward to seeing more users experience how easy it is to update OSSKUs on existing AKS node pools.