In the previous step, we saw how we could perform batch processing using some of the Azure resources available to us. In this step, we’ll take a look at real-time processing options in the Azure environment.

Event Hub in Azure

Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second. Data sent to an event hub can be transformed and stored by using any real-time analytics provider or batching/storage adapters.

The following are some of the scenarios where we can use event hubs:

• Anomaly detection (fraud/outliers)
• Application logging
• Analytics pipelines such as clickstreams
• Live dashboarding
• Archiving data
• Transaction processing
• User telemetry processing
• Device telemetry streaming.

Azure Stream Analytics

Azure Stream Analytics is real-time analytics and complex event-processing engine that’s designed to analyse and process high volumes of fast streaming data from multiple sources simultaneously.

Patterns and relationships can be identified in information extracted from a number of input sources including devices, sensors, clickstreams, social media feeds, and applications. These patterns can be used to trigger actions and initiate workflows such as creating alerts, feeding information to a reporting tool, or storing transformed data for later use. Also, Stream Analytics is available on Azure IoT Edge runtime, enabling the processing of data on IoT devices.

The following scenarios are examples of when we can use Azure Stream Analytics:

• Analyse real-time telemetry streams from IoT devices
• Weblogs/clickstream analytics
• Geospatial analytics for fleet management and driverless vehicles
• Remote monitoring and predictive maintenance of high-value assets
• Real-time analytics on Point of Sale data for inventory control and anomaly detection
• Output results to Azure Synapse Analytics, or Power BI.

You can try Azure Stream Analytics with a free Azure subscription.

The rest of this activity will cover processing big data with Azure HDInsight, an easy and cost-effective, enterprise-grade service for open-source analytics.

Join the discussion

Graeme used data from Twitter to demonstrate real-time data and the ‘sentiment’ feature. Would you suggest any modifications to the sentiment algorithm? Or any alternative uses?

Use the Discussion section below and let us know your thoughts. Try to respond to at least one other post and once you’re happy with your contribution, click the Mark as complete button to move on to the next step.