Simplifying Migration to Fabric Real-Time Intelligence for Power BI Real Time Reports

Power BI with real-time streaming has been the preferred solution for users to visualize streaming data. Real-time streaming in Power BI is interrupted. We recommend that you begin planning the migration of your data processing pipeline. Fabric real-time intelligence.

Microsoft Fabric Real-Time Intelligence It is part of the Microsoft Fabric platform. It allows businesses to collect, process, analyze and gain insights from real-time data streams. We provide an end-to-end solution that integrates real-time data pipelines and delivers advanced analytics and data visualization in one place.

This article outlines patterns and best practices as you explore Fabric Real-Time Intelligence and the Azure ecosystem.

1. Real-time data flow architecture

Figure 1.1 below shows a typical data flow pattern for real-time data analysis and visualization in Azure.

Figure 1.1

On the left are common real-time data collection services such as Azure Event Hub, IoTHub, etc. Next we have Azure Stream Analytics, a fully managed stream processing engine designed for: analyze and process Large amounts of streaming data with sub-millisecond latency. You can also find some event-driven applications that use Power Automate or Logic Apps for simple event processing. The processed events are then sent to Power BI’s streaming dataset and some persistent storage layer.

2. Insight into fabric real-time intelligence

Fabric real-time intelligence is a powerful service that helps everyone in your organization extract insights and visualize data in motion.

2.1 Collection and processing

event stream Microsoft Fabric Real-Time Intelligence allows you to import real-time events into Fabric, transform them, and then route them to a variety of destinations without writing any code. Event Streams provides several source connectors that allow you to pull event data from a variety of sources.

If you want to connect your own application with the eventstream, you can add a custom endpoint or custom app as a source. This is explained in more detail in the next section.

The Event Processor Editor in Eventstream is a code-free environment that allows you to design event data processing logic with drag and drop. Here are links to learn more about the Event Processor Editor: Event Processor Editor. The transformed data can then be routed to various destinations.

For a real-time reporting experience, we recommend using the Fabric Eventhouse KQL database. Event House It provides solutions for processing and analyzing large amounts of data, especially in scenarios that require real-time analysis and exploration.

Figure 1.2

2.2 Delivery and visualization

You can use Power BI’s automatic page refresh feature with sources that support direct querying to build real-time reports. Power BI direct queries are supported on the Fabric Eventhouse KQL database. use this link Learn more about PowerBI’s automatic page refresh feature.

The direct query feature can be used with any of the Eventstream destinations of your choice, Eventhouse or Lakehouse. However, the Eventhouse KQL database provides a powerful solution for managing and analyzing significant amounts of real-time data.

Eventhouse is designed to scale efficiently, ensuring effective performance and resource usage. This design is useful in situations where timely insight is important. Built specifically for time-based streaming events with features such as automatic indexing and partitioning based on ingestion time.

3. Azure Stream Analysis

Azure Stream Analytics PowerBI output sends transformed data to PowerBI to build rich visualizations of your analysis results. Migrating Azure Stream Analytics (ASA) jobs that leverage the Power BI output connector involves several key considerations to ensure a smooth transition and maintain real-time data visualization capabilities.

3.1 Collection and processing

For users leveraging Stream Analytics with the Power BI Output Connector in an architecture where migrating your solution to Fabric is not possible, you can explore alternative patterns after deprecating the Power BI Output Connector.

The processing and data analysis logic implemented in Azure Stream Analytics allows you to:

Route Stream Analytics output to Fabric.
Switch to another Stream Analytics connector that supports direct query mode in Power BI.

To push Stream Analytics output data to Fabric, you can use the Eventstream custom endpoint connector. Here are links to learn how to add a custom endpoint or custom app as a source to Eventstream. Add a custom endpoint or custom app source to your event stream

Once you’ve completed setting up your Eventstream custom endpoint, your event hub namespace and connection details should be available in Eventstream. Now you can go back to Azure Stream Analytics, select the Event Hub output connector, and add customizations. With this setup, existing Steam analytics jobs can now publish data to the Fabric Eventstream. Next, you can add Eventhouse as a target in Eventstream and follow the same pattern as described above.

Figure 1.3

Power BI direct query functionality is compatible with a variety of ASA and Fabric Eventstream output destinations. Below is a list of other ASA and Eventstream output connectors that you can use to build reports in PowerBI through direct queries.

Other Azure Stream Analytics output connectors:

sql server
PostgreSQL
SynapseSQL
Azure Data Explorer

Other Eventstream destinations:

3.2 Delivery and visualization

In Power BI, you design visualizations and reports by importing data into a KQL database or by choosing one of the sources mentioned above.

If you don’t want to use PowerBI’s automatic page refresh feature, you can select a different source as your import mode.

4. Fabric, real-time dashboard

For use cases where you perform scheduled refreshes and are looking for an automatic refresh experience with very short intervals, you can also explore: Fabric Real-Time Dashboard. By default, you can export a Kusto Query Language (KQL) query as a visual to your dashboard, and then later modify the underlying query and visual format as needed. In addition to ease of data exploration, this fully integrated dashboard experience provides improved query and visualization performance.

Figure 1.4

5. A call to action

If you have any questions about migration recommendations, please contact: RTI support.
If you have any questions about Azure Stream Analytics, please contact us. Askasa@microsoft.com

1. Real-time data flow architecture