Q&A

Azure Messaging: Which Option to Use, Why & How

• By David Ramel
• 05/19/2023

What's an Azure cloud dev to do for messaging in that fancy new app? From messages to events, from streams to publish and subscribe, the options seem overwhelming. What's an Event Grid? How does that compare to a Service Bus? Where do Event Hubs fit in?

If you are feeling overwhelmed, Eldert Grootenboer can help lift the messaging burden.

As a senior program manager for Microsoft, Grootenboer has a unique insight on the various types of messaging a developer can use and which Azure services are best for implementing them.

Grootenboer will share his knowledge at an upcoming tech event session titled "A Guide Through The Azure Messaging Services," part of the big, in-person, five-day Visual Studio Live! developer conference set to kick off at Microsoft HQ in Redmond, Wash., on July 17.

"In this session, we will dive into the various types of messaging, and which services we can leverage in Azure to implement them," Grootenboer said. "Once finished, you the world of messaging is at your feet, and you will be able to counter any type of messaging which comes on your path."

Attendees at the 75-minute session will learn:

• All about the different Azure messaging services
• Identify what messaging services you should use for your workloads
• Understand the architecture used for the different types of Azure messaging services

Visual Studio Magazine caught up with Grootenboer to learn more about his upcoming session, part of the Cloud, Containers, and Microservices, Microsoft Sessions track for the big conference.

VisualStudioMagazine: What are the main differences between messages, events and streams in Azure messaging services?
Grootenboer: This is an important distinction, and actually one of the first things I will address in this session. It's important to understand what type of messaging is needed, and this starts by understanding the difference between messages, discrete events and streaming events.

Let's start with message. Messages are an exchange of data, where we expect a work to be done. This could be an action like creating an invoice, a request like getting some data from a specific system, or an orchestration such as going through different steps in a process. Most importantly is that the system has an expectation of something being done, even if they do not know which other system(s) will do this for them.

Next, we have discrete events. An event is different from a message, in the way that there is no explicit expectation of work being done. Instead, an event is more similar to a notification, where there might be one or multiple systems interested in this, however there is no expectancy. For example, if we upload a blob to a storage, there might be a Function that is interested in this, as it will parse the content and kick off another process. However, this is a separate process, and not tightly coupled with the workflow that sent out the event. A discrete event is not directly related to other events, but stands on its own. There is no direct expectation of more events coming for this.

Finally, we have streaming events. These are similar to discrete events, however they are part of a stream of multiple events. We expect that there will be more events in the stream, for example a stream of logs, or temperature readings. It's normally possible to do some kind of stream processing over these events, like aggregations, calculating changes over time, etc.

Many processes will have one or multiple of these types of messages and / or events, so it's important to understand the differences, and choose the right services to handle them.

How can you choose the best messaging service for your workload based on factors such as latency, reliability and scalability?
There is not one simple answer for this, as with most architectural questions. The first answer that needs to be found, is if we are working with messages, discrete events or streaming events. Next is more specific requirements, such as throughput needed and protocols that might already be in use such as Kafka or JMS. Another important factor is the capabilities that are wanted, such as duplicate detection, whether the messages / events should be processed once or if it should be possible to go back and forward over them, the need for archival of messages, in-order processing needs and so on. Azure offers three distinct messaging services, Service Bus, Event Grid and Event Hubs, which each cater to different requirements.

What are some common scenarios or use cases for each type of messaging service in Azure?
Some of these have already been described in the first question, but I will elaborate a bit more here. In general Service Bus is used for enterprise messaging, where messages need to be exchanged between different services and systems. This could be in the same process, such as communication between microservices, but also cross process and domain boundaries, such as messaging between different software systems, and even to integrate with other partners.

Event Grid focuses on discrete events, and can handle very high throughput scenarios, where events have to be routed to many different systems. We have native event producers and consumers -- such as Azure Storage, Logic Apps, Functions and many others -- that make it very easy to sent out events and process them.

Event Hubs is used for streaming events, where the events are part of a larger stream, such as logging data, analytics, IoT and so on. It's closely integrated with Azure Stream Analytics, which allows to easily process and transform the data coming in on Event Hubs.

How do you design and implement a publish and subscribe pattern using Azure messaging services?
For a pub-sub pattern you would probably look at Service Bus, where we provide this concept through topics and subscriptions. Producers can send their messages to a Service Bus topic, and messages can be routed and filtered to multiple subscriptions. Filters can be set up using a SQL like syntax, and work on the message properties, allowing to route messages to specific consumers, or to all consumers. As Service Bus implements the capabilities needed for enterprise messaging, this is most often the chosen solution. However, this pattern can also be implemented using the other messaging services, depending on the requirements. For example, if absolute order is not important, and the other enterprise messaging capabilities offered by Service Bus are not required, but high throughput is more important, Event Grid offers this as well, including the option to implement routing and filtering. If it's a stream of events that needs to be sent to multiple consumers, this is also possible through Event Hubs.

What are some best practices and tips for using Azure messaging services effectively and securely?
As with any cloud service, it's always important to understand the services that are being used, and how to leverage their strengths and weaknesses. For example, if absolute order is needed, you would probably want to look into Service Bus or Event Hubs, whereas if you have a large number of events that need to be distributed, you probably want to look into Event Grid. Also, as this is running in the cloud, it's important to think about security. This could mean leveraging AAD identities for authentication, or isolating the service to our own networks through private endpoints.

What's another questions we should have asked?
What is the best way to optimize for performance? This starts by understanding your workloads, and the nature of your messages. Do you have many small messages / events that come in intermittently? Can you batch these? Do you have a few producers and consumers, or is there many different ones? All of these will help you in identifying the optimal settings, both on the service and in your client code.

Note that those wanting to attend Grootenboer's session who register for the VSLive! event by Friday, May 26, can save up to $400, as the "super early bird" discount listed on the event pricing page has been extended to that date, according to the conference organizer.