Building a CloudFormation macro

A couple of weeks ago AWS released a new feature to CloudFormation. Macros, a way to transform your template. This is the same technology that power the Serverless Application Model (SAM).
This release is a real game changer and this will make my life so much easier.

Why is this such a game changer then?
During the years that I have been using CloudFormation to create and handle my infrastructure I have written so many rows of boiler plate code. Adding the same sections to resources over and over again. Tags is one example, I always add a Name, Cost Allocation and Application Tag. Writing this over and over again is so annoying. With a special macro I would only need to add something like this:

Transform:
  Name: CommonTags
  Parameters:
    'Name' : 'MyAppName'
    'CostAllocation' : 'MyAppCostCode'

I have actually already built that macro, it's available on GitHub

In this blog I will guide you through building your own macro. What to think about and what limitations there are.

Create your own macro

Before we begin creating a macro we have to ensure that we understand how the transformation happens.
Basically we will use a Lambda function to do the actual work. CloudFormation will call our Lambda and expect it to return a fragment. The fragment is a piece of CloudFormation template in json format.

What is also important to remember is that the macro is only available within the account that created it. If we like to share a macro between several accounts we have to share the Lambda function, that powers the macro, with several accounts using cross account access. Then we have to create the macro in each account referencing the shared Lambda function.

The Lambda function

First of all we have to create our Lambda function that will power the macro. This can be done in your own favorite language and other tools.
I prefer to write my functions in Python and to deploy them using AWS SAM.

In the hello-world example, used through out this post, we will append a specified suffix to a S3 bucket name. This could be used if you like your buckets to always end with a specific keyword. With an simple change you can add it as a prefix instead.
This piece of CloudFormation will create the function.

MacroFunction:
  Type: AWS::Serverless::Function
  Properties:
    FunctionName: cfn-macro-hello-world
    Runtime: python2.7
    CodeUri: ./src
    Handler: index.handler
    Role: !GetAtt MacroFunctionRole.Arn

Input event

We will get an input event to our function with the following information.

{
    "region" : "The region the macro is deployed in", 
    "accountId" : "The ID of the account invoking the macro Lambda", 
    "fragment" : { }, 
    "transformId" : "The macro name", 
    "params" : { }, 
    "requestId" : "Request ID",
    "templateParameterValues" : { } 
}

Some of these are straight forward what they represent. However, fragment, params, and templateParameterValues are not that clear so let us go through those.

fragment
This is the template part that are availble for processing and transformation, always in json format.
If we the run the macro from the Transform section in the template the fragment will represent the entire CloudFormation template, with exception for the Transform section it self.

If we run it using the intrinsic function instead the fragment includes all of sibling nodes of the intrinsic function.
Yes that sounds a bot confusing, here is an example for clarification.

AWSTemplateFormatVersion: '2010-09-09'
Description: This is just a very simple example

Resources:
  TestBucket:
    Fn::Transform:
      Name: "HelloWorld"
      Parameters:
        'Suffix' : '-hello-world'
    # Start of fragment
    Type: AWS::S3::Bucket
    Properties:
      BucketName: this-is-my-macro-test-bucket
    # End of fragment
  TestBucket2:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: this-is-my-macro-test-bucket-again

params
This is the parameters injected in the Parameters section in macro, in the example above that would be 'Suffix' : '-hello-world'
It is really important to remember that CloudFormation do't evaluate the parameters before passing them to the macro. So we have to make sure that valid and required parameters are supplied.

The documentation state that params is only available when using the intrinsic function and would be empty if using Transform section.
I have found this not to be true and a template like the one below, from my CommonTags macro, will populate this section.

AWSTemplateFormatVersion: '2010-09-09'
Description: Common Tags macro example template
Transform:
  Name: CommonTags
  Parameters:
    'Name' : 'MyName'
    'CostAllocation' : 'Super-App'

templateParameterValues
Parameters that are defined in the template Parameters section will be made available in this section.

Returned data from the Lambda

CloudFormation expect us to return a json data structure:

{
    "requestId" : <Request ID>, 
    "status" : <Status>, 
    "fragment" : <CloudFormation fragment> 
}

requestId
This is the request ID from the input event.

status
CloudFormation expect us to return "success". Anything else will be treated as a failure and CloudFormation abort the creation. There is only one problem with this. Right now AWS don't document any way of returning a failure with a message. If we set status to "failure" CloudFormation will just give an error stating that creation failed without an message.
AWS please fix this! Because things go wrong and we like to return failure when it does.

fragment
This is the actual CloudFormation template section in json format e.g.

{
  "Type" : "AWS::S3::Bucket",
    "Properties" : {
      "BucketName" : "my-super-duper-bucket-hello-world",
    }
}

What is really important to remember is that the fragment that we get in the input event will be totally replaced with the fragment that we return. So if there is part of the input fragment that we have not modified we must make sure we return that part as well.

Python code

The code powering everything would then look something like this.

def handler(event, context):
    fragment = event['fragment']
    bucket_name = fragment['Properties']['BucketName']+event['params']['Suffix']
    fragment['Properties']['BucketName'] = bucket_name

    return {
        'requestId' : event['requestId'], 
        'status' : 'success', 
        'fragment' : fragment
    }

Missing here if is of course validation that a suffix was actually supplied. Remember CloudFormation will not evaluate the parameters in the Parameters section.

The actual macro

Now let us create the actual Macro. This is done by creating a AWS::CloudFormation::Macro resource and deploying that to CloudFormation.
We need to supply the Macro with a Id or Name, this must be unique for the region. Remember Macros is only available in the region in which it is deployed. We must also supply a FunctionName, which is not the name but the actual ARN for the Lambda function. This is the function that we created earlier that will do the actual work.
We can also supply an optional description which can be useful to understand what the Macro does. We can also supply a LogGroupName and LogRoleARN for debugging, more about that in the debugging section.

  Macro:
    Type: AWS::CloudFormation::Macro
    Properties:
      Description: My Hello World Macro
      Name: HelloWorld
      FunctionName: !GetAtt MacroFunction.Arn

Deploy and use it

Now when we have all our parts created, meaning the Lambda function and the Macro definition it's time to deploy this to CloudFormation. This can easily be done via the CLI.

#!/bin/bash

aws CloudFormation package \
    --template-file macro.yaml \
    --s3-bucket my-deployment-bucket \
    --output-template-file packaged-template.yaml

aws CloudFormation deploy \
    --template-file packaged-template.yaml \
    --stack-name macro-hello-world \
    --capabilities CAPABILITY_NAMED_IAM

When we now have our macro deployed we can use it from the region that we deployed it in. This we do by including it in our templates, referencing it by it's name.

Resources:
  TestBucket:
    Fn::Transform: 
      Name: "HelloWorld"
      Parameters:
        'Suffix' : '-hello-world'
    Type: AWS::S3::Bucket
    Properties:
      BucketName: this-is-my-macro-test-bucket

Now when we deploy this template CloudFormation will call our Lambda function for transformation prior to creating the CloudFormation change set.
It's just as easy as that.

Debugging

To have CloudFormation output logs to CloudWatch we have to proper configure the Macro. We have to supply a LogGroupName and LogRoleARN in the Macro configuration. The documentation however doesn't state if the Log Group need to exist or if it will be created by CloudFormation. In this case it actually need to exist, CloudFormation will not create the LogGroup, which other services, like Lambda, does. We therefor need to first create the Log Group before we can use it.
Below is an example template creating the Log group, policy needed, and the role. When everything is created and configured we get logs for the process in CloudWatch.

MacroLogGroup:
  Type: AWS::Logs::LogGroup
  Properties: 
    RetentionInDays: 30

MacroRole:
  Type: "AWS::IAM::Role"
  Properties: 
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement: 
        - 
          Effect: Allow
          Principal: 
            Service: 
              - CloudFormation.amazonaws.com
          Action: 
            - sts:AssumeRole
    Path: /

CloudWatchLogsPolicy:
  Type: "AWS::IAM::Policy"
  Properties:
    PolicyName: CfnMacroCloudwatchPolicy
    PolicyDocument: 
      Version: 2012-10-17
      Statement: 
        - 
          Effect: Allow
          Action: 
            - logs:*
          Resource: 'arn:aws:logs:*:*:*'
    Roles: 
      - !Ref MacroRole

Macro:
  Type: AWS::CloudFormation::Macro
  Properties:
    Name: HelloWorld
    LogGroupName: !Ref MacroLogGroup
    LogRoleARN: !GetAtt MacroRole.Arn
    FunctionName: !GetAtt Function.Arn

Logs for the Lambda function it self can also be found in CloudWatch, we just have to make sure the Lambda function has the proper permissions to create log groups, streams and write logs. Here we don't have to create the Log Group, Lambda will automatically do that. Would be nice with some form of consistency.

FunctionRole:
  Type: "AWS::IAM::Role"
  Properties: 
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement: 
        - 
          Effect: Allow
          Principal: 
            Service: 
              - lambda.amazonaws.com
          Action: 
            - sts:AssumeRole
    Path: /

CloudWatchLogsPolicy:
  Type: "AWS::IAM::Policy"
  Properties:
    PolicyName: CfnMacroHelloWorldCloudwatchPolicy
    PolicyDocument: 
      Version: 2012-10-17
      Statement: 
        - 
          Effect: Allow
          Action: 
            - logs:*
          Resource: 'arn:aws:logs:*:*:*'
    Roles: 
      - !Ref FunctionRole

Function:
  Type: AWS::Serverless::Function
  Properties:
    FunctionName: macro-helloworld
    Runtime: python2.7
    CodeUri: ./src
    Handler: hello-world.handler
    Role: !GetAtt FunctionRole.Arn

Conclusion.

Creating a CloudFormation macro is a rather easy and straight forward process. It brings really powerful extensibility to CloudFormation. Giving us the power to create functionality that is missing.
For me this is a total game changer and I can see myself using this all the time.
For more examples check out my GitHub repository and the AWS GitHub.