OpenHCL: the new, open source paravisor

Intro

From the beginning of the cloud computing era, virtualization technology has enabled compute workloads to run as virtual machines (VMs) in a server environment. As hardware has become more sophisticated, and new functionality has become available, the software stack has kept VMs running seamlessly, thanks to sophisticated advances in the hypervisor and virtualization software.

Confidential computing is now a powerful technology for significantly improving the security of VMs running in the cloud. However, the trust boundary of a confidential VM imposes a barrier that prevents the hypervisor from offering the rich virtualization services that VMs normally expect. Customers desiring the benefits of confidential VMs have been forced to update the operating systems of their VMs to newer versions, which must be continually revised as confidential VM technology maintains its path of rapid evolution.

Microsoft has embraced a different approach that offers much more flexibility to customers through the use of a “paravisor”. A paravisor executes within the confidential trust boundary and provides the virtualization and device services needed by a general-purpose operating system (OS), enabling existing VM workloads to execute securely without requiring continual service of the OS to take advantage of innovative advances in confidential computing technology. As confidential computing becomes available on more hardware platforms and evolves, the software stack can keep VMs running seamlessly thanks to the paravisor, in much the same way other advances in virtualization software enabled VMs to run seamlessly on ever evolving hardware.

Microsoft developed the first paravisor in the industry, and for years, we have been enhancing the paravisor offered to Azure customers. This effort now culminates in the release of a new, open source paravisor, called OpenHCL. We plan to develop OpenHCL in the open here: microsoft/openvmm: Home of OpenVMM and OpenHCL (github.com).

OpenHCL capabilities

A paravisor is essentially an execution environment that runs within the guest VM – at a higher privilege level than the guest OS – and provides various services to the guest. A paravisor can run in both confidential environments and non-confidential environments. When running in a confidential environment, these privilege levels must be enforced by the confidential computing hardware platform.

We use virtual secure mode (VSM) to run a paravisor on Microsoft’s virtualization stack. When running in a confidential context, our architecture allows VSM to be appropriately enforced in a hardware platform-agnostic manner.

Today, OpenHCL can run on both x86-64 and ARM64 platforms, and it has support for Intel TDX and AMD SEV-SNP confidential computing platforms. OpenHCL runs in the L1 VMM of a TDX confidential VM and in the VMPL0 of an SEV-SNP confidential VM. See the OpenHCL user guide for step-by-step instructions to use it. OpenHCL offers a rich set of powerful services to both confidential and non-confidential VMs alike:

• Device emulation via standard device interfaces, essentially offering a set of emulated devices, such as vTPM and serial.
• Device translation via standard device interfaces, such as NVMe to para-virtualized SCSI, allowing assignment of hardware devices directly to VMs (accelerated IO) without requiring guest OS changes – enabling VMs to take advantage of the performance of cutting-edge devices.
• Diagnostics support, particularly useful to allow debugging confidential VMs where it is difficult to use traditional methods of debugging.
• (To confidential VMs specifically) Support for guests that are not fully enlightened – such as Windows and older versions of Linux – to run on confidential computing platforms via standard architectural interfaces.

For confidential VMs, even though OpenHCL unlocks amazing value to guests that are not fully enlightened (by enabling them), in the future OpenHCL could also provide a lot of value to fully enlightened guests by providing them with any or all its other services as different scenarios require it.

OpenHCL is used in Azure in new Azure Boost SKUs, and it will be used in future Azure confidential VM SKUs. In the past month alone, over 1.5 Million VMs were running with OpenHCL in Azure^[1].

OpenHCL architecture

OpenHCL is composed of several open-source components, the most important one being OpenVMM, the cross platform, virtual machine monitor (VMM) project written in Rust. This VMM runs several user mode processes to power OpenHCL. Running a VMM inside OpenHCL allows us to support guests with assigned devices and provide device translation support. Additionally, it allows us to share confidential and non-confidential architecture. We run the same VMM in the same environment for both confidential and non-confidential guests, and the VMM provides the same services tailored to their requirements. This avoids fragmented virtualization solutions among confidential and non-confidential VMs, moving towards closing the feature gaps of confidential VMs.

The other components of OpenHCL are a boot loader and a small, customized Linux kernel built to support the VMM, with min. Kconfig to minimize binary size and runtime RAM usage. Running a kernel to support our environment allows the VMM code to be mostly standard Rust, making it much more powerful by enabling the VMM to use the broadly supported and stable Rust toolchains and crate ecosystem.

The two approaches to running confidential VMs

There are two approaches to running a guest OS inside a confidential VM: either the guest must be fully enlightened (modified to understand and manage all aspects of running as a confidential VM), or it can rely on a paravisor to implement the confidential computing enlightenments on its behalf. When a guest runs in with a paravisor, it doesn’t seem like a confidential guest precisely because it doesn’t need to act like a confidential guest.

In Azure, we support all IaaS confidential VMs via a paravisor today. The paravisor enabled Azure to support the widest variety of guests, including Windows versions released almost a decade ago^[2] and Linux versions using kernels as old as the 5.19 kernel^[3] (and versions using even older kernels that had a small set of patches backported, such as some Ubuntu and RHEL distro versions).  This provided customers with an easier lift as well as the flexibility to gain future confidential computing advances without needing to upgrade their workloads. Customers’ legacy solutions are safe with Azure because of the approach we embraced.

Why is Windows not fully enlightened to run as a confidential guest? I.e., why does Windows rely on a paravisor?

When we developed the first confidential VM in Azure on the confidential computing hardware platforms available at the time, it was not possible to fully enlighten Windows guests for those platforms because Windows required APIC (interrupt controller) emulation to be done in a paravisor. APIC emulation, traditionally done by the hypervisor, must be done by another entity for confidential VMs, where the hypervisor is outside the trust boundary. It can be done by the paravisor or by the hardware platform if it supports APIC virtualization, which early platforms like 3rd Gen AMD EPYC™ processors, didn’t.

On those hardware platforms, APIC emulation had to be done in a paravisor for Windows guests but not necessarily for Linux guests. The architecture of Windows relies directly on the APIC for interrupt management. Some aspects of Windows interrupt management don’t flow through the kernel and are inlined in drivers, so Windows drivers rely on the interrupt management behavior offered by the APIC. The architecture of Linux, on the other hand, doesn’t rely directly on the APIC for interrupt management. Linux offers kernel service routines for handling interrupt state, so Linux drivers rely on these routines.

In addition to that, Windows relies on the presence of a TPM for security features, and one cannot implement a vTPM for a confidential VM with enlightenments alone. We chose to implement a vTPM in a paravisor. Given all the functionality we have built into the paravisor, our plan is not to fully enlighten Windows and continue supporting Windows guests via a paravisor in Azure. For future versions of Linux, we’re evaluating both approaches –fully enlightened and relying on a paravisor – and we will aim to do what is best for customers.

OpenHCL and COCONUT-SVSM

An SVSM like COCONUT-SVSM plays a very valuable role for confidential computing. It can store secrets and provide virtualization services to improve the usability of fully enlightened guests. OpenHCL solves a different problem than COCONUT-SVSM. COCONUT-SVSM aims to provide services to confidential VMs with fully enlightened guests using new interfaces. OpenHCL aims to provide services to confidential VMs using existing standard architectural interfaces.

COCONUT-SVSM provides device emulation, but OpenHCL uniquely provides this via existing standard interfaces. When running with an SVSM (like COCONUT-SVSM), the guest must establish a specific relationship with the SVSM by discovering its presence and then interact with the SVSM using a custom calling convention. Essentially, a guest needs to be specifically modified to be able to take advantage of SVSM services, including devices. With OpenHCL, devices are easier to consume because existing device interfaces just work, and the guest does not need any custom calling contract modifications to consume them. OpenHCL enables devices to be discovered over standard enumeration mechanisms, like PCI virtualization or existing vTPM device contracts.

COCONUT-SVSM could potentially be leveraged by OpenHCL in the future. The VMM of component OpenHCL is Rust based, which has strong memory safety properties, and evolving its kernel component to also be Rust based would improve the memory safety of OpenHCL. During the development of OpenHCL, we chose the Linux kernel because it was a familiar OS platform for contributors and provided the capabilities needed. Now that Rust-based COCONUT-SVSM exists, we are interested in moving to that in the future and building OpenHCL support for it if it gains the features that OpenHCL needs.

Open for collaboration

In this blog we described the value of OpenHCL for the future of computing. We still have much more we plan to do with OpenHCL, and as we develop new functionality in the open, we would love to collaborate with you. You can learn more about this project on: https://openvmm.dev. Please reach out to us if you have ideas you’d like to add to the OpenHCL roadmap or any other feedback. You can open a GitHub issue, reach out to us on Zulip, and even contribute to this project! We track the roadmap of OpenHCL in the open; below are some of its future milestones!

OpenHCL support for Intel TDX (Trust Domain Extensions)

Intel and Microsoft collaborated on and co-developed the TDX partitioning architecture so that it could be leveraged by a paravisor. The first ever TDX module with TD partitioning was an amazing co-engineering project between Intel and Microsoft, and Intel released TD partitioning as part of the TDX Module that accompanied the general availability of 5th Generation Xeon, and this has also been backported to 4th Generation Xeon. Using this TDX module, Azure launched the first generation of Azure TDX confidential VMs with the first paravisor, being the first cloud service provider to offer TDX in public preview as well as the first cloud service provider to offer Windows guest support for TDX. Intel has been contributing for the past 6+ months, and we’re close to feature completeness in OpenHCL for the next generation of Azure TDX confidential VMs!

OpenHCL support for Arm CCA (Confidential Compute Architecture)

We started engaging with Arm almost two years ago to make sure the Arm Confidential Compute Architecture (CCA) is well equipped to support paravisor stacks like OpenHCL. CCA comprises a collection of open-source software, firmware, specifications, and hardware support to bring confidential computing to the Arm architecture.  CCA provides protected environments called Realms, that can be used to host confidential VMs. Our collaboration lead to the creation of the Planes feature, which enables multiple of levels of privilege to coexist inside a Realm. Planes provide the ability to host a paravisor, and a guest VM in the same Realm, with the paravisor providing security and compatibility services to the guest. We are excited to collaborate further and in the open with Arm to build OpenHCL support for Arm CCA.

OpenHCL support for AMD SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging)

We used AMD’s VMPLs to build the first paravisor for confidential VMs in Azure. We have been engaging with AMD to ensure OpenHCL and AMD’s platform can best work together to provide great performance and security for customers. We will continue developing OpenHCL support for future generations of Azure SNP confidential VMs.

OpenHCL support for KVM as host

Today OpenHCL runs only on the MSFT hypervisor. We are excited to begin developing OpenHCL support for KVM as host in collaboration with other cloud service providers and the Linux and KVM community to enable others to leverage OpenHCL in their virtualization stacks.

– the Core OS Platform team.