Integrated Vectorization for Azure AI Search now Generally Available

We are excited to announce the general availability of Integrated vectorization Using Azure OpenAI embeddings Azure AI Search. This represents an important milestone in our ongoing mission to simplify and accelerate data preparation and index creation. Augmented Search Generation (RAG) And traditional applications.

The RAG pipeline is simplified with integrated vectorization.

Why is vectorization important?

Vectorization is the process of converting data into embeddings (vector representations) for performing vector searches. Vector search helps identify similarities and differences in data, allowing businesses to provide more accurate and relevant search results. Vectorizing and indexing data also involves several steps, including cracking. fullness and Chunking. The way you perform each of these steps provides opportunities to make your search system more efficient and effective. Check out the blog post Improved vector search performance with hybrid search and ranking capabilities It shows which configuration works better depending on the scenario.

What is integrated vectorization?

Integrated vectorizationSimplify the indexing pipeline and RAG workflow from source files to index queries, a feature of Azure AI Search that integrates: Data Chunking Text/image vector conversion is integrated into a single flow, enabling vector search across proprietary data with minimal friction.

Unified vectorization simplifies the steps required to prepare and process data for vector search. It handles segmentation of the original document as part of the indexing pipeline.S Split into chunks and automatically generate and map embeddings with Azure OpenAI integration.S Sends the newly vectorized chunks to the Azure AI Search index. Also enables automatic vectorization of user queries sent to the AI ​​Search index.

This index can be used as a search system wherever you build RAG applications, including Azure AI Studio and Azure OpenAI Studio.

What features are generally available now?

The following features within Unified Vectorization are generally available as part of: REST API Version 2024-07-01:

• Index Forecast: This feature allows you to map a single source document to multiple chunks, improving the relevance of your search results.

• Split technology Chunking through duplication: This feature divides data into small, manageable chunks for independent processing.

• Custom vectorizer Feature: This allows connecting to other embedding endpoints besides Azure OpenAI.

• Customer Management Key For indexes with vectorizers: This feature allows for additional security and control over your data by using your own keys. When you configure a CMK on your AI Search index, vectorizer operations are also encrypted with your own keys at query time.

How do I get started with Unified Vectorization in the Azure Portal?

that Data Import and Vectorization Wizard Simplify the creation of integrated vectorization components, including document chunking, automatic Azure OpenAI embedding generation, index definition, and data mapping, in the Azure portal. The wizard now supports: Azure Data Lake Storage Gen2In addition to that Azure Blob Storage and OneLake (Preview)Facilitates data ingestion from a variety of data sources. A forthcoming wizard will also support mapping source document addition metadata to chunks, and the Azure portal provides: Debug Session For function Skill set Composed of Index Forecast.

ADLS Gen2 support in the “Import and Vectorize Data” wizard

Azure AI Search also lets you personalize your indexing pipeline. Through code And take advantage of integrated vectorization. Directly supported data sources. For example, here’s a blog post on how to achieve this for Azure SQL Server data using integrated vectorization: Vector Search with Azure SQL Database.

What’s still in public preview?

There is also support for image (multimodal) embedding and Azure AI Studio model catalog embedding, which remain in public preview. For more information about this feature, visit: Azure AI Search now supports AI Vision multimodal and AI Studio embedding models – Microsoft Communic….

Customers and Benefits

A streamlined RAG pipeline allows organizations to scale and accelerate app development. Managed embedding processing with integrated vectorization allows organizations to provide a turnkey RAG system for new projects, allowing teams to quickly build GenAI applications tailored to their datasets and needs without having to build custom deployments every time.

Client: SGS & Co.

For over 70 years, SGS & CO has been at the forefront of design, graphic services and graphic manufacturing. Marks and SGS’s expert teams work with clients around the world to ensure consistent and seamless brand experiences.

“A key priority was to provide our global team with efficient tools that streamline their workflows, starting with our sourcing and research processes. We realized that we needed a system that would allow for searchable assets without relying solely on order management inputs, which can be inconsistent or different from real-world data. These inconsistencies presented challenges for our AI module.”

“SGS AI Visual Search is a GenAI application built on Azure that enables global production teams to more effectively find sourcing and research information relevant to their projects. The most important benefit of SGS AI Visual Search is that it leverages RAG to use Azure AI Search as a search system to accurately locate and retrieve relevant assets for project planning and production.”

“RAG’s Azure AI Search vector search capabilities enable us to go beyond exact and fuzzy matching to contextual search. This allows employees to access information quickly and effectively, improving service delivery to both our internal teams and global customers.”

“AI Search’s integrated vectorization capabilities also greatly streamline our data processing workflows. By automating batching and chunking, we can index data faster and easier without the need for separate compute instances. Seamless handling of vectorization during live search on Azure saves development time and reduces deployment costs. This capability allows us to efficiently create and manage indexes for multiple clients without the need for extensive pipeline management. Furthermore, integrating this capability with other RAG applications, such as chatbots and data discovery systems, further enhances our ability to deliver comprehensive solutions across multiple platforms.”

Laura Portelli, Product Manager, SGS & Co

Customer: Denzibank

Intertech is the software house of Denzibank, one of the top five banks in Türkiye.Day Largest private bank. Builds a centralized RAG system using Azure AI Search and integrated vectorization to support multiple GenAI applications and minimize data processing and management.

“At Intertech, we were looking for a solution that would allow us to disseminate and leverage information more efficiently across our current documentation, ticketing systems, and company processes. This solution also needed to serve as a central vectorization and search solution for the various GenAI applications we were building. Azure AI Search’s integrated vectorization gave us access to the latest models available from OpenAI, including embedding-3-large, which made our work much easier, allowing us to develop a wide range of applications very quickly and easily.”

Salih Eligüzel, Head of DevOps and MLOps, Intertech

Frequently Asked Questions

How much does integrated vectorization cost?

As part of you AI Search Service Pricing You have the permission to do so Includes limitations of built-in indexers. The segmentation techniques (data chunking), native data parsing, and index projection required for unified vectorization are provided at no additional cost. Azure OpenAI embedding calls are billed to Azure OpenAI services according to the service. price model.

What customizations are possible with integrated vectorization?

The Azure Portal supports the most common scenarios through:Importing and vectorizing data” wizard. However, if your business requirements extend beyond these common scenarios and require additional customization, Azure AI Search lets you customize the indexing pipeline through code and use its integrated vectorization capabilities. Directly supported data sources.

Custom options include enabling features available via other technologies. AI Enhancement For example, you can use custom code via: Custom WebApi Technology To implement other chunking strategies: Leveraging AI Document Intelligence Calls all available functions for chunking, parsing, and preserving table structure. Built-in technology It is used for data conversion, etc. Skill set Configurations help you enhance functionality to better suit your business needs.

For a more comprehensive understanding, explore: AI Search Vector GitHub A repository that stores sample code and Azure AI Search Power Skills A repository containing examples of custom technologies. For example, This custom technology code Used to call external embedding endpoints (except Azure OpenAI) and can be invoked with custom indexing pipelines and vectorizers at query time.

Some scenarios suitable for integrated vectorization

Unified vectorization is especially useful when preparing data with AI enrichment before chunking and vectorization. Azure AI Search provides AI enrichment capabilities for: Character recognition For convenience, we perform other data transformations before putting them into the index.

Unified vectorization is ideal for RAG solutions that require rapid deployment without constant developer intervention. Identifying the patterns you need allows your team to use them for convenient RAG and continuous deployment. Examples include projects, use case scenarios with specific documentation, etc.

In essence, unified vectorization provides a promising option when trying to shorten the time to market for RAG scenarios by creating a finder with little or no code.

More news

Azure AI Search is also releasing binary quantization along with other vector relevance features for general availability today! Learn more about these new additions. Binary Quantization GA Announcement Blog Post.

What’s next?

Stay tuned for more updates on the latest features in Azure AI Search and how they can help simplify integration with RAG applications! 

Get started with Azure AI Search

• Explore the Preview Client Library.Python,.net,JavaandJavaScriptWe provide a variety of integration methods to suit different user needs.

• Learn how to build an end-to-end RAG application.Azure AI Studio.