AWS StepFunctions HTTP Endpoint demystified

I started to play around with StepFunctions new HTTP Endpoint state as soon as it was released. I found it to be a very nice addition and it worked really well, I was happy with it. At a recent re:Invent reCap I did a presentation and a demo on it, so I once again dove deep into it. I still like the functionality, not having to write a Lambda function just to call an API is very refreshing. However, there are some parts on it that I feel is just a bit misplaced. In this post I will go over the good, the bad, and the ugly with StepFunctions HTTP Endpoint State.

Solutions in this post

To demonstrate the HTTP Endpoint state we'll create two applications. The first will pull weather data from an open weather api, for a specific longitude and latitude. It will use the weather data to determine if it's shorts or not weather.

The second solution will use the Slack API to post a message to a Slack channel.

EventBridge Connection

To call an API using a HTTP Endpoint state a EventBridge Connection is required. This so we don't have to hard code or include any authorization parameters in our StateMachine.

The thought behind this is good, but I feel it get a bit strange, why is there not a StepFunction Authorization resource? Why reuse what was created for EventBridge API Destinations? I would prefer if this was part of StepFunctions or was a standard SecretsManager secret.

EventBridge Connection Secrets

Speaking of secrets. When a EventBridge Connection is created there is automatically a SecretsManager secret created, with a format like this.

I understand that the secret might need to be in certain format that the connection understand, but I would have preferred that I supply the secret to the connection and that the connection just populate it with data. I have seen and got several questions from people, that has been using the HTTP Endpoint state, and then discovers this secret, wondering what it is and what created it.

Create EventBridge Connection

Let's create an EventBridge Connection for our open weather API. To do this, we need to navigate to the EventBridge console. Yes, it feels so strange. In the EventBridge Console there is also no "Connections" in the menu. Instead we have to navigate to "API Destinations" and select the "Connections" tab. Yes it is very well hidden. It feels that connections are not a first class citizen, I understand the placement when it was only for API Destinations. But it's not anymore, make it an option in the menu please!

In our first solution we'll use an Open Weather API from yr.no a weather service in Norway. Click Create Connection and fill in the details. Supply a name and a description and then we need to set an authentication.

Again this is where things get highly opinionated from AWS. We must select one of the three authentication types, it's not possible to create a connection without an authentication. But the API I'm going to use is fully open and doesn't require an authentication. Sure, a fully open API might not be the best of practices, but it does exists. Anyway, let's select API Key and just fill in some bogus values. And for that matter, I think API Key should have been named "Use Header" instead, more on that later.

After the connection is created let's build the first solution.

Shorts or not

The first solution will call an open Weather API, fetch the current conditions at a location and decide if it's short weather or not. The state machine that will be used looks like this.

Call the API, use a ResultSelector to pick out the current conditions, use a Choice state to determine if it's shorts or not weather.

Create a StepFunction and add an HTTP Endpoint state to the graph, we configure it to use YR API, make sure you read the documentation if you are going to try this. The method will be GET and we use connection created before.

Now, the API require that we'll supply a longitude and latitude as query parameters, we can't set this directly in the endpoint. Instead scroll down and expand "Advanced parameters" and set the query parameters there.

The output from this state will be a json object with the headers and the body that the endpoint responds with, Headers and Body will be in separate objects, like this.

{
  "Headers" : { },
  "ResponseBody" : { }
}

The data we get back from the API is an array with time series data, we are only interested in the current conditions, so here we can use a ResultSelector to get only what we need. In the output section add the following as ResultSelector.

{
  "temperature.$": "$.ResponseBody.properties.timeseries[0].data.instant.details.air_temperature"
}

Drag in a Choice State and two Pass state. Create a condition for the shorts weather, in the choice state. Set that if temperature is above 10 degrees C it's shorts weather. Let the default state go to Not Shorts Weather.

This mean that your state machine should now look like this.

Save it and let's try it out. We need to start a new execution and supply the longitude and latitude, I supply the coordinates close to where I live.

After the run we can see that it's apparently not shorts weather in the middle of the winter,

Deploy it with SAM

This SAM template will create EventBridge connection and the state machine if you prefer to use that.

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Create StepFunction, is it shorts or not weather?

Parameters:
  Application:
    Type: String
    Description: Name of the application
    Default: http-endpoint-demo

Resources:
  YrWeatherApiConnection:
    Type: AWS::Events::Connection
    Properties:
      AuthorizationType: API_KEY
      AuthParameters:
        ApiKeyAuthParameters:
          ApiKeyName: notuse
          ApiKeyValue: notuse
      Description: Call YR open Weather API
      Name: yr-weather-api

  ShortsOrNotStateMachineStandard:
    Type: AWS::Serverless::StateMachine
    Properties:
      DefinitionUri: statemachine/statemachine.asl.yaml
      Tracing:
        Enabled: true
      DefinitionSubstitutions:
        EBConnectionArn: !GetAtt YrWeatherApiConnection.Arn
      Policies:
        - Statement:
            - Effect: Allow
              Action:
                - logs:*
              Resource: "*"
        - Statement:
            - Effect: Allow
              Action:
                - secretsmanager:GetSecretValue
                - secretsmanager:DescribeSecret
              Resource: !GetAtt YrWeatherApiConnection.SecretArn
        - Statement:
            - Effect: Allow
              Action:
                - events:*
              Resource: !GetAtt YrWeatherApiConnection.Arn
        - Statement:
            - Effect: Allow
              Action:
                - states:InvokeHTTPEndpoint
              Resource: !Sub arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:*
      Type: STANDARD

State machine ASL:

Comment: Shorts or not?
StartAt: Call Yr Weather API
States:
    Call Yr Weather API:
        Type: Task
        Resource: arn:aws:states:::http:invoke
        Parameters:
            Authentication:
                ConnectionArn: ${EBConnectionArn}
            Method: GET
            ApiEndpoint: https://api.met.no/weatherapi/locationforecast/2.0/compact
            QueryParameters:
                lat.$: $.lat
                lon.$: $.lon
        ResultSelector:
            temperature.$: $.ResponseBody.properties.timeseries[0].data.instant.details.air_temperature
        Next: Choice
    Choice:
        Type: Choice
        Choices:
            - Variable: $.temperature
              NumericGreaterThan: 10
              Next: Shorts Weather
        Default: Not Shorts Weather
    Shorts Weather:
        Type: Pass
        End: true
    Not Shorts Weather:
        Type: Pass
        End: true

Post to Slack

First of all, we need to create and setup an Slack application how to do that can be found in my post Serverless and event-driven translation bot

Now we need to create an EventBridge connection for our Slack application / bot. When posting to a channel using the Slack API we need to send the token in the Authorization header, setting the value to Bearer [TOKEN]. Here it become clear that this Authorization type in EventBridge Connection should be called Header and not API Key, that would have made so much more sense I think.

Just as with Shorts or Not we can create a StepFunction that use the connection and post to Slack. In the Body we need to supply the text and channel, we supply that as input parameters to the StepFunction invocation.

When calling the API from a StepFunction we need to set the Content-Type header, Slack doesn't parse our data and determine if it's json or not. This is done under Advanced Parameters.

We can now invoke the StepFunction and supply the parameters needed.

Deploy with SAM

Just as before, this SAM template will create EventBridge connection and the state machine if you prefer to use that.

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Create StepFunction calling Slack API.

Parameters:
  Application:
    Type: String
    Description: Name of the application
    Default: slack-endpoint-demo

Resources:
  SlackApiConnection:
    Type: AWS::Events::Connection
    Properties:
      AuthorizationType: API_KEY
      AuthParameters:
        ApiKeyAuthParameters:
          ApiKeyName: Authorization
          ApiKeyValue: Bearer resolve:secretsmanager:/slack/app/token
      Description: Call Slack API
      Name: slack-api

  SlackSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Description: Slack Oauth Token Secret
      Name: /slack/app/token

  PostToSlackStateMachineStandard:
    Type: AWS::Serverless::StateMachine
    Properties:
      DefinitionUri: statemachine/statemachine.asl.yaml
      Tracing:
        Enabled: true
      DefinitionSubstitutions:
        EBConnectionArn: !GetAtt SlackApiConnection.Arn
      Policies:
        - Statement:
            - Effect: Allow
              Action:
                - logs:*
              Resource: "*"
        - Statement:
            - Effect: Allow
              Action:
                - secretsmanager:GetSecretValue
                - secretsmanager:DescribeSecret
              Resource: !GetAtt SlackApiConnection.SecretArn
        - Statement:
            - Effect: Allow
              Action:
                - events:*
              Resource: !GetAtt SlackApiConnection.Arn
        - Statement:
            - Effect: Allow
              Action:
                - states:InvokeHTTPEndpoint
              Resource: !Sub arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:*
      Type: STANDARD

StateMachine definition

Comment: Post to Slack
StartAt: Post to Slack Channel
States:
    Post to Slack Channel:
        Type: Task
        Resource: arn:aws:states:::http:invoke
        Parameters:
            Authentication:
                ConnectionArn: ${EBConnectionArn}
            Method: POST
            RequestBody:
                channel.$: $.channel
                text.$: $.text
            Headers:
                Content-type: application/json
            ApiEndpoint: https://slack.com/api/chat.postMessage
        End: true

Final Words

In this post I took a look at HTTP Endpoint state in StepFunctions. This functionality, to be able to call APIs directly, is a great feature. I will use this over and over again that is for sure, I love it. However, it doesn't come flawless, there are thing that I find really strange. But, I think this is such a great feature that I'm willing to live with the weirdness, for now!

Don't forget to follow me on LinkedIn and X for more content, and read rest of my Blogs