How to Integrate Azure AI Speech in Unity: Step-by-Step Guide

Have you ever planned to create a virtual assistant for your games? Or is it something like the real world where players can talk to NPCs? Well, if you want to integrate the power of artificial intelligence within your game, the possibilities can be endless, and luckily, Azure AI Services provides a variety of tools to help you: utilize Within the Unity game engine. In this blog sore We’ll walk you through using Azure AI Speech in Unity by creating a simple speech-to-text function.

Prerequisites:

1. Unity 2020.3 or later.
2. Your subscription key for the Azure Speech service. (You can try Azure for free.)
3. A working microphone and access to it.

Create an Azure AI Speech resource:

1. Go to Azure AI Speech Studio Sign in with your Microsoft account.
2. Select Settings, and then select Create resource. Configure with the following settings:
   • New resource name: Please enter a unique name.
   • application: Azure subscription.
   • region: Please select a supported region.
   • Pricing tiers: Free FO (if available, otherwise select standard S0).
   • Resource group: Select or create a resource group with a unique name.
3. Select Create resource. Wait for the resource to be created, then select Use Resource. The audio welcome page is displayed.

You can explore Speech Studio and check out the different tools, but for now: Real-time voice to text.

check it out Microsoft Learn – Azure AI Voice To explore more.

Copy region and resource key

1. Click the settings icon in the top right corner.
2. Copy “Region” and “Resource Key”.

Unity project settings

1. Create a new 3D project in Unity.
2. Right click on the Hierrarchy panel and go to UI “Text – TextMeshPro”.
3. Right click on the Hierrarchy panel and go to UI “Button – TextMeshPro”.
4. Place these two components in the Scene View.
5. “text” Used to display recognized speech.
6. “button” It is used to record your speech.

Import the Speech SDK into your Unity project

1. Download the Speech SDK from: here.
2. After downloading, select Next to get the Speech SDK. asset > import package > custom package (Alternatively, you can double-click the downloaded package and open it in your current project.)
3. Make sure all files are selected and click Import.

Create a script

1. Right-click in the Assets folder and create a C# script called “Speech.cs”.
2. Copy and paste the code below –

using UnityEngine;
using UnityEngine.UI;
using Microsoft.CognitiveServices.Speech;
using TMPro;
#if PLATFORM_ANDROID
using UnityEngine.Android;
#endif
#if PLATFORM_IOS
using UnityEngine.iOS;
using System.Collections;
#endif

public class HelloWorld : MonoBehaviour
{
    // Hook up the two properties below with a Text and Button object in your UI.
    public TextMeshProUGUI outputText;

    public Button startRecoButton;

    private object threadLocker = new object();
    private bool waitingForReco;
    private string message;

    private bool micPermissionGranted = false;

#if PLATFORM_ANDROID || PLATFORM_IOS
    // Required to manifest microphone permission, cf.
    // https://docs.unity3d.com/Manual/android-manifest.html
    private Microphone mic;
#endif

    public async void ButtonClick()
    {
        // Creates an instance of a speech config with specified subscription key and service region.
        // Replace with your own subscription key and service region (e.g., "westus").
        var config = SpeechConfig.FromSubscription("", "");

        // Make sure to dispose the recognizer after use!
        using (var recognizer = new SpeechRecognizer(config))
        {
            lock (threadLocker)
            {
                waitingForReco = true;
            }

            // Starts speech recognition, and returns after a single utterance is recognized. The end of a
            // single utterance is determined by listening for silence at the end or until a maximum of 15
            // seconds of audio is processed.  The task returns the recognition text as result.
            // Note: Since RecognizeOnceAsync() returns only a single utterance, it is suitable only for single
            // shot recognition like command or query.
            // For long-running multi-utterance recognition, use StartContinuousRecognitionAsync() instead.
            var result = await recognizer.RecognizeOnceAsync().ConfigureAwait(false);

            // Checks result.
            string newMessage = string.Empty;
            if (result.Reason == ResultReason.RecognizedSpeech)
            {
                newMessage = result.Text;
            }
            else if (result.Reason == ResultReason.NoMatch)
            {
                newMessage = "NOMATCH: Speech could not be recognized.";
            }
            else if (result.Reason == ResultReason.Canceled)
            {
                var cancellation = CancellationDetails.FromResult(result);
                newMessage = $"CANCELED: Reason={cancellation.Reason} ErrorDetails={cancellation.ErrorDetails}";
            }

            lock (threadLocker)
            {
                message = newMessage;
                waitingForReco = false;
            }
        }
    }

    void Start()
    {
        if (outputText == null)
        {
            UnityEngine.Debug.LogError("outputText property is null! Assign a UI Text element to it.");
        }
        else if (startRecoButton == null)
        {
            message = "startRecoButton property is null! Assign a UI Button to it.";
            UnityEngine.Debug.LogError(message);
        }
        else
        {
            // Continue with normal initialization, Text and Button objects are present.
#if PLATFORM_ANDROID
            // Request to use the microphone, cf.
            // https://docs.unity3d.com/Manual/android-RequestingPermissions.html
            message = "Waiting for mic permission";
            if (!Permission.HasUserAuthorizedPermission(Permission.Microphone))
            {
                Permission.RequestUserPermission(Permission.Microphone);
            }
#elif PLATFORM_IOS
            if (!Application.HasUserAuthorization(UserAuthorization.Microphone))
            {
                Application.RequestUserAuthorization(UserAuthorization.Microphone);
            }
#else
            micPermissionGranted = true;
            message = "Click button to recognize speech";
#endif
            startRecoButton.onClick.AddListener(ButtonClick);
        }
    }

    void Update()
    {
#if PLATFORM_ANDROID
        if (!micPermissionGranted && Permission.HasUserAuthorizedPermission(Permission.Microphone))
        {
            micPermissionGranted = true;
            message = "Click button to recognize speech";
        }
#elif PLATFORM_IOS
        if (!micPermissionGranted && Application.HasUserAuthorization(UserAuthorization.Microphone))
        {
            micPermissionGranted = true;
            message = "Click button to recognize speech";
        }
#endif

        lock (threadLocker)
        {
            if (startRecoButton != null)
            {
                startRecoButton.interactable = !waitingForReco && micPermissionGranted;
            }
            if (outputText != null)
            {
                outputText.text = message;
            }
        }
    }
}
// 

Please enter the copied content. “Subscription Key” and “region” to line 36

Save the script and return to Unity.

3. Script Canvas GameObject.

4. Instead “Output Text” and “Start recording button” drag text and button Each component created in the scene.

test

1. Now click the play button in Unity to enter game mode.
2. in game mode button And speak into the microphone.
3. You will see your voice converted to text.

In this way, you can easily integrate Azure AI Speech and use it for Speech to Text. you can do more modify You can make your imagination a reality according to your own needs..

Learn more about the different use cases for Azure AI Speech. text to speech It’s about understanding its full potential. Resources include: