AWS StepFunctions HTTP Endpoint demystified

2024-02-14
This post cover image
Voice provided by Amazon Polly

I started to play around with StepFunctions new HTTP Endpoint state as soon as it was released. I found it to be a very nice addition and it worked really well, I was happy with it. At a recent re:Invent reCap I did a presentation and a demo on it, so I once again dove deep into it. I still like the functionality, not having to write a Lambda function just to call an API is very refreshing. However, there are some parts on it that I feel is just a bit misplaced. In this post I will go over the good, the bad, and the ugly with StepFunctions HTTP Endpoint State.

Solutions in this post

To demonstrate the HTTP Endpoint state we'll create two applications. The first will pull weather data from an open weather api, for a specific longitude and latitude. It will use the weather data to determine if it's shorts or not weather.

The second solution will use the Slack API to post a message to a Slack channel.

EventBridge Connection

To call an API using a HTTP Endpoint state a EventBridge Connection is required. This so we don't have to hard code or include any authorization parameters in our StateMachine.

Image showing the description of EventBridge connections.

The thought behind this is good, but I feel it get a bit strange, why is there not a StepFunction Authorization resource? Why reuse what was created for EventBridge API Destinations? I would prefer if this was part of StepFunctions or was a standard SecretsManager secret.

EventBridge Connection Secrets

Speaking of secrets. When a EventBridge Connection is created there is automatically a SecretsManager secret created, with a format like this.

Image showing the secret.

I understand that the secret might need to be in certain format that the connection understand, but I would have preferred that I supply the secret to the connection and that the connection just populate it with data. I have seen and got several questions from people, that has been using the HTTP Endpoint state, and then discovers this secret, wondering what it is and what created it.

Create EventBridge Connection

Let's create an EventBridge Connection for our open weather API. To do this, we need to navigate to the EventBridge console. Yes, it feels so strange. In the EventBridge Console there is also no "Connections" in the menu. Instead we have to navigate to "API Destinations" and select the "Connections" tab. Yes it is very well hidden. It feels that connections are not a first class citizen, I understand the placement when it was only for API Destinations. But it's not anymore, make it an option in the menu please!

Image showing where EventBridge connections are located.

In our first solution we'll use an Open Weather API from yr.no a weather service in Norway. Click Create Connection and fill in the details. Supply a name and a description and then we need to set an authentication.

Image showing the created connection.

Again this is where things get highly opinionated from AWS. We must select one of the three authentication types, it's not possible to create a connection without an authentication. But the API I'm going to use is fully open and doesn't require an authentication. Sure, a fully open API might not be the best of practices, but it does exists. Anyway, let's select API Key and just fill in some bogus values. And for that matter, I think API Key should have been named "Use Header" instead, more on that later.

After the connection is created let's build the first solution.

Shorts or not

The first solution will call an open Weather API, fetch the current conditions at a location and decide if it's short weather or not. The state machine that will be used looks like this.

Image showing the state machine for shorts or not.

Call the API, use a ResultSelector to pick out the current conditions, use a Choice state to determine if it's shorts or not weather.

Create a StepFunction and add an HTTP Endpoint state to the graph, we configure it to use YR API, make sure you read the documentation if you are going to try this. The method will be GET and we use connection created before.

Image showing the http endpoint state configuration.

Now, the API require that we'll supply a longitude and latitude as query parameters, we can't set this directly in the endpoint. Instead scroll down and expand "Advanced parameters" and set the query parameters there.

Image showing the http endpoint query parameters.

The output from this state will be a json object with the headers and the body that the endpoint responds with, Headers and Body will be in separate objects, like this.

{
"Headers" : { },
"ResponseBody" : { }
}

The data we get back from the API is an array with time series data, we are only interested in the current conditions, so here we can use a ResultSelector to get only what we need. In the output section add the following as ResultSelector.

{
"temperature.$": "$.ResponseBody.properties.timeseries[0].data.instant.details.air_temperature"
}

Image showing the result selector.

Drag in a Choice State and two Pass state. Create a condition for the shorts weather, in the choice state. Set that if temperature is above 10 degrees C it's shorts weather. Let the default state go to Not Shorts Weather.

Image showing the condition.

This mean that your state machine should now look like this.

Image showing the state machine.

Save it and let's try it out. We need to start a new execution and supply the longitude and latitude, I supply the coordinates close to where I live.

Image showing the state machine run.

After the run we can see that it's apparently not shorts weather in the middle of the winter,

Image showing the state machine finished.

Deploy it with SAM

This SAM template will create EventBridge connection and the state machine if you prefer to use that.


AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Create StepFunction, is it shorts or not weather?

Parameters:
Application:
Type: String
Description: Name of the application
Default: http-endpoint-demo

Resources:
YrWeatherApiConnection:
Type: AWS::Events::Connection
Properties:
AuthorizationType: API_KEY
AuthParameters:
ApiKeyAuthParameters:
ApiKeyName: notuse
ApiKeyValue: notuse
Description: Call YR open Weather API
Name: yr-weather-api

ShortsOrNotStateMachineStandard:
Type: AWS::Serverless::StateMachine
Properties:
DefinitionUri: statemachine/statemachine.asl.yaml
Tracing:
Enabled: true
DefinitionSubstitutions:
EBConnectionArn: !GetAtt YrWeatherApiConnection.Arn
Policies:
- Statement:
- Effect: Allow
Action:
- logs:*
Resource: "*"
- Statement:
- Effect: Allow
Action:
- secretsmanager:GetSecretValue
- secretsmanager:DescribeSecret
Resource: !GetAtt YrWeatherApiConnection.SecretArn
- Statement:
- Effect: Allow
Action:
- events:*
Resource: !GetAtt YrWeatherApiConnection.Arn
- Statement:
- Effect: Allow
Action:
- states:InvokeHTTPEndpoint
Resource: !Sub arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:*
Type: STANDARD

State machine ASL:

Comment: Shorts or not?
StartAt: Call Yr Weather API
States:
Call Yr Weather API:
Type: Task
Resource: arn:aws:states:::http:invoke
Parameters:
Authentication:
ConnectionArn: ${EBConnectionArn}
Method: GET
ApiEndpoint: https://api.met.no/weatherapi/locationforecast/2.0/compact
QueryParameters:
lat.$: $.lat
lon.$: $.lon
ResultSelector:
temperature.$: $.ResponseBody.properties.timeseries[0].data.instant.details.air_temperature
Next: Choice
Choice:
Type: Choice
Choices:
- Variable: $.temperature
NumericGreaterThan: 10
Next: Shorts Weather
Default: Not Shorts Weather
Shorts Weather:
Type: Pass
End: true
Not Shorts Weather:
Type: Pass
End: true

Post to Slack

First of all, we need to create and setup an Slack application how to do that can be found in my post Serverless and event-driven translation bot

Now we need to create an EventBridge connection for our Slack application / bot. When posting to a channel using the Slack API we need to send the token in the Authorization header, setting the value to Bearer [TOKEN]. Here it become clear that this Authorization type in EventBridge Connection should be called Header and not API Key, that would have made so much more sense I think.

Image showing the slack connection.

Just as with Shorts or Not we can create a StepFunction that use the connection and post to Slack. In the Body we need to supply the text and channel, we supply that as input parameters to the StepFunction invocation.

Image showing the slack StepFunction.

When calling the API from a StepFunction we need to set the Content-Type header, Slack doesn't parse our data and determine if it's json or not. This is done under Advanced Parameters.

Image showing the slack StepFunction headers.

We can now invoke the StepFunction and supply the parameters needed.

Image showing the slack StepFunction invocation.

Deploy with SAM

Just as before, this SAM template will create EventBridge connection and the state machine if you prefer to use that.


AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Create StepFunction calling Slack API.

Parameters:
Application:
Type: String
Description: Name of the application
Default: slack-endpoint-demo

Resources:
SlackApiConnection:
Type: AWS::Events::Connection
Properties:
AuthorizationType: API_KEY
AuthParameters:
ApiKeyAuthParameters:
ApiKeyName: Authorization
ApiKeyValue: Bearer resolve:secretsmanager:/slack/app/token
Description: Call Slack API
Name: slack-api

SlackSecret:
Type: AWS::SecretsManager::Secret
Properties:
Description: Slack Oauth Token Secret
Name: /slack/app/token

PostToSlackStateMachineStandard:
Type: AWS::Serverless::StateMachine
Properties:
DefinitionUri: statemachine/statemachine.asl.yaml
Tracing:
Enabled: true
DefinitionSubstitutions:
EBConnectionArn: !GetAtt SlackApiConnection.Arn
Policies:
- Statement:
- Effect: Allow
Action:
- logs:*
Resource: "*"
- Statement:
- Effect: Allow
Action:
- secretsmanager:GetSecretValue
- secretsmanager:DescribeSecret
Resource: !GetAtt SlackApiConnection.SecretArn
- Statement:
- Effect: Allow
Action:
- events:*
Resource: !GetAtt SlackApiConnection.Arn
- Statement:
- Effect: Allow
Action:
- states:InvokeHTTPEndpoint
Resource: !Sub arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:*
Type: STANDARD

StateMachine definition

Comment: Post to Slack
StartAt: Post to Slack Channel
States:
Post to Slack Channel:
Type: Task
Resource: arn:aws:states:::http:invoke
Parameters:
Authentication:
ConnectionArn: ${EBConnectionArn}
Method: POST
RequestBody:
channel.$: $.channel
text.$: $.text
Headers:
Content-type: application/json
ApiEndpoint: https://slack.com/api/chat.postMessage
End: true

Final Words

In this post I took a look at HTTP Endpoint state in StepFunctions. This functionality, to be able to call APIs directly, is a great feature. I will use this over and over again that is for sure, I love it. However, it doesn't come flawless, there are thing that I find really strange. But, I think this is such a great feature that I'm willing to live with the weirdness, for now!

Don't forget to follow me on LinkedIn and X for more content, and read rest of my Blogs


Post Quiz

Test what you just learned by doing this five question quiz - https://kvist.ai/845519.
Scan the QR code below or click the link above.

Powered by kvist.ai your AI generated quiz solution!