Serverless projects over the years

I decided to write a post about the serverless architectures I have created over the years. When I tried to pick one to write about I figured out that it was going to be impossible to pick just one.
I created my first AWS Lambda function late 2015, just months after the service became general available. At re:Invent AWS Step Functions was announced, I was there when it happened. Just months after that we were running our first Step Function in production at Sony. With the introduction of Amazon EventBridge it has just exploded. So I decided to write a brief post about some of the architectures me and my different teams has built over the years. It was impossible to pick one.

Just to be clear! Some of the architectures in this post is obsolete now due to introduction of other services and functionality, but back when they were created it was relevant. Therefor I decided to write about them anyway.

So buckle up! Now we start!

2017 - The SQS Poller

This is one of the architectures that would be obsolete.

We built and used this pattern back at Sony in 2017. We needed to poll work from a SQS queue and invoke a Lambda function to perform the actual work. This was way before SQS could invoke Lambda functions so that was not an option.

The solution consists of an "Auto Scaler" Step Function that is invoked every 5 minutes while there are messages in the queue. If number of messages increased from 0 the Auto Scalar was directly triggered. The Auto Scalar checks number of messages that was in the queue, how many instances of the worker Step Function that was running, and decided the required scaling action. To keep track of all instances of the worker Step Function it was assigned a unique ID by the Auto Scaler when invoked and started. A register of all running workers was kept in a DynamoDB table. A worker was considered alive as long as it had reported "home", updated the DynamoDB table within the latest 5 minutes.

The worker Step Function start by updating a field in the DynamoDB table, the calling "home". Next it polled a message from the SQS queue and invoked a Lambda to do the actual work. It was possible to do both synchronous and asynchronous work. Below is a picture of the overall architecture.

2020 GitLab Auto Scaled Self Hosted Runners

In the end of 2020 I decided to build something quite different. There was a need for a Auto Scaled solution to run self hosted GitLab runners. Since there was a need to build Docker images this needed to be done on plain old EC2 instances, since building Docker in Docker is not recommended. But at the same time there was some parts that was plain Node builds. For these builds it would be interesting to do that in a different way than on EC2. Fun part was that Docker support in Lambda was released during re:Invent 2020, this made a perfect match.

GitLab can call a webhook on different events. These events can be used to determine if a build need to be started, if it just finished, and things like this. The webhook was setup with direct integration between API Gateway and EventBridge. Here EventBridge was used to be able to invoke different Step Functions based on the event type.

When the webhook was called with an event that a new job needed to be started EventBridge would invoke a Step Function to determine the type of runner that was needed. Metadata attached to the job in the GiteLab pipeline would be used to either start a new EC2 instance or invoke a lambda function to do the build.

When the webhook was called with an event that a job had finished EventBridge would trigger a different Step Function that would terminate the EC2 instance, if the job was run on a EC2 instance.

The really fun part here was that building in a Lambda function was really great. Number of builds that could be run in parallel was out of this world.

The entire setup is available on GitHub and below is the overall architecture pictures.

2021 - Serverless Iot

Fast forward to present day and a serverless IoT architecture I'm working on. In this design the Iot devices post events to AWS IoT Core, that will trigger an Iot Rule. The IoT Rule will invoke a Lambda Function that will post the IoT events into EventBridge. I can say that it's a bit annoying having this Lambda function, as it doesn't do anything else than just passing the data. But at the time of writing there is not direct integration between IoT Core and EventBridge so I have no choice.

Different EventBridge rules will then invoke different StepFunction that are registered as targets. One StepFunction enrich and store the data, a different invoke an alarm if needed. EventBridge play a huge role here, it make it possible to route events to different services in a simple and managed way.

On top of everything there is an webapp that displays the data. The webapp fetch the data over GraphQL, of course AWS AppSync is used for that.

So what are these IoT Devices then? It's a BBQ thermometer that I'm building using a Raspberry Pi running AWS IoT Greengrass. More on this will come in later blog posts. Below is an overall picture of the architecture.

Favorite Service?

Do I have any favorite service among all of the serverless services AWS offers? I would say yes! Amazon EventBridge is there right at the top. Why? Easy, is enables decoupling between serverless services. A service producing data can just post event into a common EventBus and then doesn't need to know, or care, about the consumers. As an example, we have a producer service that store data in an Amazon S3 bucket. A different service need to read the created files, process the data, and send it over to a different external system. With EventBridge the producer could either push a message to EventBridge that a new file is available. Yes, you can use the default EventBus and rely on events from CloudTrail that files are created, but that adds some extra latency that need to be considered. Sure you can use SNS and SQS instead of EventBridge but I doesn't feel that it become as nice solution as with EventBridge.

So, EventBridge is by far my favorite service since it make decoupling so simple and elegant!

Summary

This was just three of the serverless architectures I have built during the last 6 years. One things is sure, there are more coming! I will never stop being serverless first. There are just so many benefits with serverless that it's impossible to stop. Hope you enjoyed the read and got some inspiration!