Event-Routers and Brokers in AWS. Navigating the Event-Driven Landscape

This is the third post on the topic around Serverless and event-driven design thinking. In the last post I created a Serverless file manager. Today we are going to take a deeper look at different event-routers and event-brokers in AWS.

AWS offers a diverse array of event-routers and brokers designed for communication between services. Each service has its own strengths allowing for different messaging patterns and use cases. In this blog, we'll look at services like Amazon SQS, Amazon SNS, Amazon EventBridge, and AWS IoT Core.

What is a message-router and event-broker

Before we start let's take a look at what a message-router or event-broker is and what purpose it has in a event-driven design.

In an event-driven architectures, event-brokers enabled efficient event communication between our services. It operates based on the publish-subscribe pattern, where producers publish events to the broker, and subscribers consume these events based on their interests.

Producers post events and messages to the broker which allows for easy filtering of events.

The broker routes events to all interested subscribers / consumer services. It manages the possibility for services to register as a consumer. It then handles the complexity of routing events and messages to multiple subscribers based on the filters set by the subscribers.

Decoupling The event-broker decouples producers and consumers. Producers don't need to know who the consumers are, and consumers don't need to know where the events come from.

Let's continue by looking at some of the AWS services.

Queuing for Asynchronous Communication

SQS is not only the first and oldest service in AWS, it's also a very reliable service for handling message queuing. It's a very strong service when it comes to decoupling of application components and services. It's a virtual buffer that ensures messages can be reliably consumed.

We have two different types of SQS queues, standard and FIFO. The standard queues comes with a higher throughput but sacrifice the order of the messages. FIFO has a slightly lower throughput but guarantee the order.

When consuming SQS queues we work with polling and we can't have more than one consumer application / service / component per queue. Messages in the queue will only be delivered to one consumer, so if we have more than one all consumers will not get all messages. What we can do is combine the use of SNS to create a fan-out pattern. More on that later.

When using Lambda function as the consumer for messages in queues, the Lambda Service will poll the queue and invoke the Lambda function. To control the concurrency of Lambda invocations, in standard queue, we can use the maximum concurrency setting in Lambda. For FIFO queues the scaling works a bit different. The Lambda service will look at the message group id and will invoke one Lambda function per group id.

To learn more about maximum concurrency and group id read this post by AWS

Massive Broadcast of Notifications with Ease

SNS follows the publish-subscribe pattern, allowing messages to be broadcasted to multiple subscribers at a massive scale. Just like SQS, SNS also support a FIFO option. However FIFO topics in SNS comes with a very limited bandwidth compared to a standard topic. One of SNS strengths is the possibility to handle millions and millions of subscribers per topic, in a FIFO setup this is limited to a hundred.

SNS support several different target options like HTTP(S) endpoints, SMS, mobile push, SQS, and Lambda functions. All subscribers of a topic will be invoked simultaneously allowing for a massive scale.

when using Lambda function as the subscriber to a SNS topic. All functions are invoked in parallel as soon as a message is published to an SNS topic. This provides real-time responsiveness to an event.

If we combine SNS and SQS, setting SQS as the consumer, we can accomplish several things. We can control the concurrency, slowing down the parallel invocations. This is a good approach if you have a downstream application service that can't handle sudden burst of calls. It also allow us to use application components running in containers, Fargate for example, to consume SNS by putting an SQS in between.

Orchestration in Event-Driven Architectures

EventBridge is a very powerful event-bus service designed to handle and allow for easy integration between application components and services. It allows you to create rules that define how events should be routed to different services, like Lambda functions, StepFunctions, and other EventBridge event-buses. The rules can contain very complex logic to only route explicit messages to specific subscribers.

EventBridge is a fully managed event-broker that can handle a massive amount of events and messages. It's one of the most powerful event-routers in AWS, when we look at combination of scale and targets. It's also one of my personal favorite services and there are few design I make that don't include EventBridge. It has quickly become the go-to service in an event-driven architecture.

AWS services publish events to the default event-bus, there is one per account and region, this is events about state changes in AWS services. For example it can be EC2 instances starting, tags being changed, objects being created in S3. It's best practice to leave the default event-bus to the AWS services and create your own custom buses for your application use-cases.

EventBridge is also more than just an event-broker. I allows you to create scheduled rules that will invoke your targets on a schedule like every 5 minutes, once per day, etc. It also comes with an impressive serverless Scheduler that allows you to run tasks from one central service. It allows you to create one time invocations that occure on a specific time.

Yet another very useful feature is the Schema Registry. this is a component that allows services and applications to register event-schemas and make them discoverable by services. The registry can automate the process of detecting event-schemas and making them available.

All of these capabilities together create a very powerful service.

Choosing Between SQS, SNS, and EventBridge

Selecting the right service depends on your architecture's specific requirements. There is no clear if this then that scenario. The complexity and requirements of your architecture is what in the end determine what to use.

But I will try and give some form of guidance.

SQS, if you have a 1:1 matching between producer and consumer or you need to control the concurrency. If guaranteed message delivery and reliable processing are critical, SQS is a good choice..

SNS, broadcasting is key. You need to fan-out to millions of subscribers, handle millions of events, then SNS is a powerful option. If real-time Responses is required, scenarios where immediate response to events SNS is the way to go.

EventBridge Allows for complex orchestration. In architectures involving coordination of application components and services, EventBridge will simplify the event-driven orchestration. If scheduling of invocations is needed or you have specific target requirements like 3rd party SaaS solutions, EventBridge is the service for you.

What about connected Things?

IoT Core is also a serverless event-router. It comes with a very specific use-case and that is to handle messages from IoT Things. It's not designed to be used as an event-broker between components and services running in AWS. Instead it will route events and messages coming from hardware sensors and devices running anywhere. It does have an powerful rule based engine that can route these messages to many different AWS services, like Lambda functions, SQS, Kinesis and many more.

If you are building an IoT solution the this is the service to use.

Final Words

Navigating the landscape of event-routers and brokers in AWS requires careful consideration of your requirements. With Amazon SQS, SNS, EventBridge, and IoT Core, you have the tools needed to design and build great event-driven architectures. By understanding how each service interacts with AWS service, you can create responsive, scalable, and resilient systems that thrive on the power of events.

Don't forget to follow me on LinkedIn and Twitter for more content, and read rest of my Blogs