Document Ingestion for Gen AI Applications using Logic Apps from 1000+ data sources!

Data is the heart of building any AI application, and efficient data collection is essential to success. With over 1,400 enterprise connectors, Logic Apps provides unparalleled access to a wide range of systems, applications, and databases, making it easier than ever to create powerful generative AI applications. By leveraging connectors like Azure OpenAI and Azure AI Search, enterprises can seamlessly implement the Retrieval-Augmented Generation (RAG) pattern to easily collect and search data from multiple sources.

Breaking news!

We’re excited to share a public preview of two new actions in Azure Logic Apps.Document analysis and text chunking. With these additional features, building an ingestion workflow that allows AI applications to “chat with your data” is now possible in just six steps, completely out of the box and without writing a single line of code!

This work is based on: Apache Tika Toolkit and parser librariesParse thousands of file types, including PDF, DOCX, PPT, HTML, and more, in multiple languages. Seamlessly read and parse documents from virtually any source, without the need for custom logic or configuration!

This code-free approach lets you automate complex workflows like parsing documents, chunking data, and driving generative AI models, so you can unleash the potential of your data with minimal effort.

In addition to these built-in actions, Azure Logic Apps also provides the following capabilities: pre-made Template For data collection from various common data sources.Helps you quickly build and deploy applications, including SharePoint, Azure File Storage, Blob Storage, SFTP, and more.

~ inside RAG (Reinforced Amplification Generation)The ingestion process involves several steps to ensure that documents can be effectively processed, searched, and used by generative AI models. Here are the details for each step and how to use Logic Apps.

• Document Collection – Leverage 1400+ Connector Collect relevant documents, data sets, or other information sources in Logic Apps.

• Parsing Documents – Leverage Document Analysis Convert content such as PDF documents, CSV files, PPT, etc. into tokenized strings.

• Document Chunking – Leverage Chunk text Split tokenized content into small, manageable chunks for AI models to process in subsequent steps. This operation provides options to choose chunking strategy, token size, etc., so users can organize chunks into the optimal size and configure them to fit their AI models.

• Vectorization – Leverage Azure Open AI ConnectorAnd especially Generating embeddings This task is to convert tokenized chunks into vector embeddings. Embedding Represent text in a format that AI can understand and compare for efficient retrieval.

• Collection – Preparing data for collection Choose We do this by mapping the generated embeddings to the Azure AI Search index schema. Then we use the Azure AI Search connector. Indexing multiple documents The task of storing vector embeddings in a vector database for fast and efficient similarity-based retrieval.

Below is a sample workflow that triggers when a new file is created on a SharePoint site and is ingested into Azure AI Search along with all the default actions.

Logic Apps now offers: Pre-written templates For ~ RAG intakeConnect common data sources like SharePoint, Azure File, SFTP, Azure Blob Storage, and more to get up and running quickly. These templates save you development time, allowing you to get started quickly while still maintaining the flexibility to customize the workflow to meet your specific needs. If you don’t see a template for your preferred data source, let us know and we’ll add it. You can also modify an existing template or start from scratch with a blank workflow.

And here’s a video that goes into more detail on this feature. As always, if you have any questions or feedback, please contact us.