Azure AI Confidential Inferencing

Customers who need to protect sensitive and regulated data are looking for end-to-end, verifiable data privacy from service providers and cloud operators alike. Azure’s industry-leading support for Confidential Computing (ACC) extends traditional data protection beyond encryption at rest and in transit to ensure data remains private while in use, such as when processed by AI models. Customers in highly regulated industries, including multinational banking giant RBC, have integrated Azure Confidential Computing into their platforms to gain insights while maintaining customer privacy.

To preview Secret Reasoning For ~ that azure OpenAI service whisper To convert speech to text todayMicrosoft is the first cloud provider offering Confidential AI. cavitySecret Whispers End-to-end provision seclusion ~ of Prompt Including Audio And I wrote it down Text response By ~ guarantee it The prompt is only deciphered within: tearusty Eexecution EenvironmentS (tea) ~ in Azure Confidential GPU Virtual machine(VM).

These VMs provide enhanced protection for inference applications, prompts, responses, and models within VM memory and when code and data are transferred to and from the GPU. Confidential AI also enables application developers to anonymize users accessing the cloud model, protecting their identities and protecting against targeted attacks.

If you are interested in discussing confidential AI use cases and trying out confidential inference using Azure OpenAI Service Whisper models, visit here. Preview Signup Page. Read on more Details How is it confidential meInference task, what What Developers Need to Do and Our Confidential Computing Portfolio

Secret menferencing is designed for enterprises and cloud-native developers building AI applications that need to process sensitive or regulated data that must remain encrypted in the cloud. It also allows you to remotely measure and audit the code that processes the data to ensure that it only performs the expected functions and not other functions. This allows you to build AI applications that protect the privacy of users and data.

Secret menferencing leverages Azure aspirateTrusted virtual machines with NVIDIA H100 Tensor Cores Graphics cardnow Generally available. These VMs combine the SEV-SNP technology of AMD CPUs with the Confidential Computing support of the H100 GPUs to ensure the integrity and privacy of all code and data loaded within the VM and protected areas of GPU memory.

For example, SEV-SNP encrypts and integrity-protects the entire address space of a VM using hardware-managed keys. This means that all data processed within the TEE is protected from unauthorized access or modification by any code outside the environment, including privileged Microsoft code such as the virtualization host operating system and the Hyper-V hypervisor. When a VM is paired with an H100 GPU in confidential compute mode, all traffic between the VM and the GPU is encrypted and integrity-protected from advanced attackers.

Secret menferencing supports Oblivious HTTP using Hybrid Public Key Encryption (HPKE) to protect user privacy and encrypt and decrypt inference requests and responses. Enterprises and application providers can use Oblivious HTTP proxy to encrypt prompts, which are routed through Azure Front Door and Azure OpenAI service load balancers to OHTTP gateways hosted on Confidential GPU VMs in Kubernetes clusters managed by Project Forge on Azure Machine Learning. Front Door and load balancers are relays and only see the ciphertext and ID of the client and gateway, while the gateway only sees the relay ID and plaintext of the request. Private data remains encrypted.

The OHTTP Gateway generates attestation evidence in the form of a token obtained from the Microsoft Azure Attestation service to obtain the private HPKE key from KMS. This proves that all software running within the VM, including the Whisper container, has been attested.

After obtaining the private key, the gateway decrypts the encrypted HTTP request and passes it to the Whisper API container for processing. Once the response is generated, the OHTTP gateway encrypts the response and sends it back to the client.

Image: Confidential Inference Architecture

Secret menferencing leverages VM images and containers built securely from trusted sources. A software bill of materials (SBOM) is generated at build time and signed to attest to the software running in the TEE.

Can be integrated with Confidential meHost an application or enterprise OHTTP proxy that can obtain HPKE keys from KMS to perform inference, encrypt the inference data before leaving the network, and use the keys to decrypt the returned transcript. We are providing a reference implementation of such a proxy. The Whisper REST API and payload have not changed.

There is an overhead associated with supporting confidential computing, which adds additional delay to completing transcription requests compared to standard Whisper. We are working with Nvidia to reduce this overhead in future hardware and software releases.

Azure OpenAI service The whisper first azure AI middleOdell-as-a-s-a-Sservice From Microsoft With cKeep a secretl computerTing ProtectAction. As part of us Long-term investment in confidential computing well Continue to Participate Our FryVacancy-senPositive customer best Support Their unique AI Scenario. AI scenario, And let me know if there are any other models you’d like to see.

If you are interested in discussing confidential AI use cases and trying out confidential inference using Azure OpenAI Service Whisper models, visit here. Preview Signup Page.

resources:

1. Get it Dig deep Azure AI Confidential Inference
2. Learn more Confidential VM GA