Stream Apache Kafka data to Azure services effortlessly via Confluent Connectors on Azure

Today’s A data-driven worldProcessing and Integration Real-time data Collecting data from various sources is essential for businesses. This data flows regularly between multiple services for efficient ETL, data processing, data analytics, etc.! Managing multiple first-party and third-party services, orchestrating data flows, and creating manual connections between each of these services via connection strings is all Very manual process, It takes a lot of time, And it carries a high risk human error While copying and pasting.

Join Connector

Join Connector Manages the movement of data between you and your sink (i.e. destination) SaaS service. Kafka streaming with high throughput and low latency. Now we have improved Managing Confluent Connectors Leverage your experience between your first-party Azure services and Confluent Cloud Apache Kafka® and Apache Flink® Confluent Cloud – Azure Native ISV Service. this Preview function It helps to have the connection details automatically populated in a dropdown. A smooth, error-free end-to-end creation experience – Stay in the Azure Portal everyone! Today, we are announcing preview support. Azure Blob Storage Source and Sink Connectors via the Azure Portal.

Figure 1. Confluent connector in Azure

Confluent Resource Hierarchy

The joining organization consists of: Environment, Clusters and Topics Nesting one within the other. Connector Nested under cluster and stream data. message. These messages are stored within feeds, called feeds. subject. The Confluent resource hierarchy is as follows:

Figure 2. Joining Organization Hierarchy

How to create a Confluent Connector in Azure (Preview)

You can now move data between Kafka and Azure Blob storage. Join Connector It just takes a few simple steps!

memo: To create a Confluent connector: you The organization, environment, clusters, and topics must be pre-configured.. If you haven’t created a Confluent organization in Azure yet, check out: This document For detailed instructions. Hurry up, even if you are a first time visitor. Free $1000 Credit to Give it a try!

1. From your Confluent organization’s overview page, navigate to: Data streaming And click Connector It’s under the same. Or you can click on: Connector Configuration Click the button at the bottom of the Overview page.

Figure 3. Confluent Overview Page

2. Select an environment and cluster from the drop-down menu. You will see a list of Azure connectors created within this cluster.
   1. You can also filter connectors based on their type (source or sink) and status.

Figure 4. Connector tile in Azure

3. Click here Create ConnectorIn the right pop-up blade, you can select the source and sink connectors, the plugin service you want to connect, and automatically fetch Azure application details through the drop-down.

Figure 5. Creating a new Confluent Connector

4. Complete the Configuration tab to set the data format, number of operations, etc. After reviewing your selections, click Next. make. After a few minutes you will receive a notification that the connector is up and ready!
   1. Clicking on the new connector tile will redirect you to the Confluent UI where you can view connector status, throughput, and other statistics.

What comes next?

We have some exciting additions to this preview feature coming soon!

• Connector support for additional Azure data services including ADLS Gen2, EventHub Source, Synapse, Functions, and more!
• Experience Confluent Connectors in the Azure Portal, SDK, and CLI.

resources