Streamline Document Indexing in Azure Cosmos DB Using Logic Apps Automation

introduction

Effectively managing large documents is essential to maintaining modern applications, especially fast and reliable queries. With Azure Logic Apps, you can now automate document indexing to Azure Cosmos DB in addition to the existing indexing capabilities of AI Search, giving you the flexibility to use either service as your vector store.

Logic Apps provides a rich set of connectors to seamlessly integrate with a variety of document sources, including Azure Blob Storage, SharePoint, and OneDrive, enabling automated workflows for document collection from multiple locations. Whether you work with structured data files such as PDFs, Word documents, or CSVs, Logic Apps supports efficient parsing of a variety of document types.

For larger documents, Logic Apps can also implement chunking to optimize processing and indexing by dividing the file into manageable parts. This allows you to seamlessly process complex or large data sets without excessive use of system resources.

In terms of integration with Azure Cosmos DB, the Logic Apps Cosmos DB connector supports multiple authentication methods, including managed identity, shared key authentication, and Azure Active Directory OAuth, providing flexibility depending on your security requirements. Logic Apps can also meet a variety of networking needs, such as integrating with private endpoints or using VNet integration to secure communications between services.

In this post, we’ll explore a scenario where Logic Apps automates the ingestion and indexing of documents, such as PDFs, into Azure Cosmos DB. This approach not only reduces operational overhead, but also keeps the data highly accessible and queryable.

Why does Cosmos DB use Logic Apps for document indexing?

• Automated Workflow: Automating document indexing eliminates manual work and ensures that documents are indexed as soon as they are uploaded.
• scalability: As your document volume grows, the global distribution of Azure Cosmos DB ensures that your data remains scalable and available.
• Seamless integration: Logic App allows you to easily integrate with other Azure services, such as Blob storage and AI models, to enhance document indexing with intelligence and automation.

Scenario Overview

This scenario automates the collection, parsing, and indexing of document content in Azure Blob storage. Azure Cosmos DB. When a blob (such as a PDF or text document) is uploaded, a logic app workflow is triggered to process the document and store that data in a Cosmos DB container for easy searching and querying.

Prerequisites

To set up a scenario on your computer, you need to set up the following:

1. Azure CosmosDB resource to index data
2. An Azure Storage account to upload content to be indexed.

Azure CosmosDB settings

After creating the resource:

1. Go to your Azure CosmosDB resource
2. Select “Features” in the “Settings” menu.
3. Enable the “Vector search in Azure Cosmos DB for NoSQL” feature.

The steps can also be found in detail here: this blog post In Azure CosmosDB. Now that we have set up our CosmosDB resource as the index store, let’s create a new database and a container for the vector store database.

To create a new container, follow these instructions:

1. Go to “Data Explorer”.
2. Create a “New Container” with the following fields set:
   1. Database ID: This is your database ID. In our case it is ‘docs’.
   2. Container ID: The container where the document will be stored, defined as ‘Category’.
   3. Partition Key: We defined it as ‘/category’ for data distribution as there may be different indexed document categories that can be queried.
3. Container vector policy: This is where you set vector properties for ‘vector embedding 1’.
   1. Path: The location to retrieve and display the vector embedding. In our case it would be ‘/vector’.
   2. Data type: float32
   3. Distance function: Used to determine the distance between nearest neighbors. In our case we set it to ‘cosine’.
   4. Dimension: 1056
   5. Index Type: diskANN, a low-cost, scalable, high-latency option for finding Approximate Nearest Neighbors (ANN).

More information about container setup can be found here. GitHub Tutorials.

Document structure for indexing in Azure Cosmos DB

This Logic Apps workflow indexes document embeddings into Azure Cosmos DB. Below is a breakdown of the key fields we map and index.

1. content: This field contains the body of the document or the actual processed text content. For example, this could be text data extracted from documents such as contracts, invoices, or other file types.
2. document name: The name or title of the document being indexed. This field helps you identify documents based on their file names, making it easier to search and retrieve documents by their original names.
3. vector: This represents the document’s embedding vector, which is a numerical representation of the content. These vectors are used to perform similarity searches on documents, enabling AI-based insights or matches based on content similarity.
4. Document ID: A unique identifier generated for each document. This way, each document has a unique ID, which is important for querying and updating specific items in the Cosmos DB container.
5. category: This is where you specify the document type or category. In our case, we are using “Document” as the value for this field. This helps categorize and group documents and can be useful when querying for specific types of documents within a database.
6. ID: Another unique identifier that is often automatically generated or derived by concatenating values. This ID can be used to ensure that there are no duplicates and that each document is properly referenced.

When you configure the payload to pass Azure CosmosDB in your logic app workflow:

Key steps in the workflow

I am GitHub Sample For workflow projects. Here is a visual representation of the workflow:

1. Blob upload detection: The logic app starts by detecting when a new blob (document) is added or updated in Azure Blob Storage.
2. Read blob content: The workflow reads the content of the uploaded blob and prepares it for further processing.
3. Document parsing: Logic Apps parse documents to extract relevant content, including text and metadata. This may include PDF extraction or text chunking for larger documents.
4. Chunk text (if needed): For large documents, split content into manageable chunks for smooth processing and indexing.
5. Generate Embeddings Using AI: The logic app uses Azure AI to generate embeddings from document content. These inclusions enable improved data processing, classification, and structural mapping within Cosmos DB.
6. Mapping to Schema: Extracted data and embeddings are mapped to a predefined schema, ensuring consistency in how documents are indexed within Cosmos DB. The properties we index are:
7. Bulk updates in Cosmos DB: The final processed document is stored and indexed in Cosmos DB. The “Create or update many items in bulk” operation accepts a database and container ID along with the data to be indexed, where multiple items have been processed into the database in a previous operation.

conclusion

by utilizing Azure Logic App To automate document indexing Azure Cosmos DBhelps you streamline your data workflow, reduce manual intervention, and organize your data for optimal performance. This powerful integration simplifies processes, making it easier for teams to manage large volumes of documents and scale as needed.

What’s next?

Logic Apps currently supports efficient document indexing, but vector search Features for AI-based search are not yet available in Azure CosmosDB. This is a highly anticipated feature that will strengthen Cosmos DB as a powerful vector store. Stay tuned for this update!