General Availability: Azure confidential VMs with NVIDIA H100 Tensor Core GPUs

Today, we are announcing the general availability of Azure Confidential virtual machines (VMs) featuring NVIDIA H100 Tensor Core GPUs. These VMs combine four hardware-based data-in-use protection features.^Day Generation AMD EPYC^TM Processor-based confidential VMs with the power of NVIDIA H100 Tensor Core GPUs. By enabling confidential computing on GPUs, Azure gives customers more options and flexibility to run their workloads securely and efficiently in the cloud. These VMs are ideal for inferring, fine-tuning, or training small to medium-sized models such as Whisper, Stable diffusion, and its variants (SDXL, SSD), and language models such as Zephyr, Falcon, GPT2, MPT, Llama2, Wizard, and Xwin.

Azure NCC H100 v5 virtual machines are currently available in the US East2 and Europe West regions.

Figure 1. Simplified NCCH100 v5 architecture.

Recommended Hardware Partners

We appreciate the support and endorsement of our hardware partners.

“The expanding landscape of innovation, particularly generative AI, is creating limitless opportunities for enterprises and developers. NVIDIA’s accelerated computing platform is helping pioneers like Azure accelerate the performance of AI workloads while maintaining strong security through confidential computing.” Daniel Rohrer, vice president of software product security, architecture and research at NVIDIA.

“AMD is a pioneer in confidential computing, and has a long-standing partnership with Azure to deliver a number of confidential compute services powered by leading AMD EPYC processors. Now, we are extending our confidential compute capabilities to AI workloads with new Azure Confidential VMs featuring NVIDIA H100 Tensor Core GPUs and 4th Gen AMD EPYC CPUs, delivering the industry’s first confidential AI service. We are excited to work with Azure to expand our confidential compute offerings to meet the demands of AI workloads.” Ram Pedivotla, vice president of product management for AMD’s Cloud Business Unit.

Customer Use Cases and Feedback

Here are some examples of workloads that customers have experimented with during the preview and plan to further leverage the power of Azure NCC H100 v5 GPU virtual machines:

• Confidential inference from audio to text (Whisper model)
• Video inputs detecting abnormal behavior for accident prevention – Leveraging confidential computing to meet data privacy requirements.
• Reliable dissemination of privacy-preserving design data in the automotive industry (inference and learning)
• A multi-party clean room that performs analytics on billions of transactions and terabytes of data from financial institutions and their subsidiaries.

	Advancing AI securely is core to our mission, and we’re excited to collaborate with Azure Confidential Computing to validate and test confidential inference for the Audio-Text Whisper model on Nvidia GPUs. Matthew Knight, OpenAI Security Director
	F5 can develop and deploy GenAI models using Microsoft Azure Confidential VMs with NVIDIA H100 Tensor Core GPUs. While the AI ​​model learns from private data, the underlying information remains encrypted within the Trusted Execution Environment (TEE). This solution enables you to build advanced AI-based security solutions while ensuring the confidentiality of the data the model analyzes. This strengthens customer trust and reinforces our position as a leader in protecting secure networks. Azure Confidential Computing helps build a better, safer, and more innovative digital world. Arul Elumalai, SVP and GM, Distributed Cloud Platform and Security Services, F5, Inc.
	ServiceNow is working closely with Microsoft, NVIDIA, and Opaque to bring AI to people and deliver great experiences for both customers and employees on the Now Platform. The partnership between Opaque and Microsoft will enable you to quickly deploy and leverage the power of Azure Confidential VMs using NVIDIA H100 Tensor Core GPUs to deliver confidential AI with verifiable data privacy and security. Kellie Romack, Chief Digital Information Officer, ServiceNow
	Integrating the Opaque platform with Azure Confidential VMs featuring NVIDIA H100 Tensor Core GPUs to create Confidential AI helps eliminate data sovereignty and privacy concerns, making AI adoption faster and easier. Confidential AI is the future of AI deployments, and together with Opaque, Microsoft Azure, and NVIDIA, we are making this future a reality today. Aaron Fulkerson, CEO, Opaque Systems
	Powered by the preview of Azure Confidential VMs featuring NVIDIA H100 Tensor Core GPUs, our team has successfully integrated GPU capabilities into Constellation, a Kubernetes distribution focused on confidential computing. This enables customers to lift and shift sophisticated AI stacks to Azure Confidential Computing. With Continuum AI, we have created a framework for end-to-end confidential delivery of LLM, ensuring the highest level of privacy for your data and setting a new standard for AI inference solutions. We are particularly excited to work with Azure Confidential Computing to unlock the transformative potential of confidential computing in the era of generative AI. Felix Schuster, CEO and Co-Founder, Edgeless Systems
	Cyborg is excited to work with Azure to preview Azure Confidential VMs using NVIDIA H100 Tensor Core GPUs. This partnership will enable GPU acceleration of the Confidential Vector Search algorithm to meet the stringent performance requirements of AI applications while maintaining the highest level of security. We are eagerly awaiting the general availability of these VM SKUs as we prepare to deploy production-grade services. Nicolas Dupont, Cyborg CEO

RBC has been working very closely with Microsoft on confidential computing initiatives since the early days of technology availability in Azure. Justin Simonellis, Director of Engineering and Confidential Computing, RBC Services. “We have leveraged the benefits of confidential computing to integrate it into our own data cleanroom platform, called Arxis. As we continue to develop our platform capabilities, we fully recognize the importance of privacy-preserving machine learning inference and training to protect sensitive customer data within the GPU, and look forward to leveraging Azure Confidential VMs using NVIDIA H100 Tensor Core GPUs.”

Performance Insights

Azure Confidential VMs with NVIDIA H100 Tensor Core GPUs deliver best-in-class performance for inferencing small to medium-sized models while protecting your code and data throughout its lifecycle. We benchmarked these VMs on a variety of models using vLLM.

The table below shows the test configuration.

The figure above shows the confidential computing overhead with and without CUDA graph enabled. For most models, the overhead is negligible. For smaller models, the overhead is higher due to the increased latency of encrypting PCIe traffic and kernel calls. Increasing the batch size or input token length are viable strategies to mitigate confidential computing overhead.

Learn more