A couple of weeks ago AWS released a new feature to CloudFormation. Macros, a way to transform your template. This is the same technology that power the Serverless Application Model (SAM).
This release is a real game changer and this will make my life so much easier.
Why is this such a game changer then?
During the years that I have been using CloudFormation to create and handle my infrastructure I have written so many rows of boiler plate code. Adding the same sections to resources over and over again. Tags is one example, I always add a Name, Cost Allocation and Application Tag. Writing this over and over again is so annoying. With a special macro I would only need to add something like this:
I have actually already built that macro, it’s available on GitHub
In this blog I will guide you through building your own macro. What to think about and what limitations there are.
Create your own macro
Before we begin creating a macro we have to ensure that we understand how the transformation happens.
Basically we will use a Lambda function to do the actual work. CloudFormation will call our Lambda and expect it to return a fragment. The fragment is a piece of CloudFormation template in json format.
What is also important to remember is that the macro is only available within the account that created it. If we like to share a macro between several accounts we have to share the Lambda function, that powers the macro, with several accounts using cross account access. Then we have to create the macro in each account referencing the shared Lambda function.
The Lambda function
First of all we have to create our Lambda function that will power the macro. This can be done in your own favorite language and other tools.
I prefer to write my functions in Python and to deploy them using AWS SAM.
In the hello-world example, used through out this post, we will append a specified suffix to a S3 bucket name. This could be used if you like your buckets to always end with a specific keyword. With an simple change you can add it as a prefix instead.
This piece of CloudFormation will create the function.
We will get an input event to our function with the following information.
Some of these are straight forward what they represent. However, fragment, params, and templateParameterValues are not that clear so let us go through those.
This is the template part that are availble for processing and transformation, always in json format.
If we the run the macro from the Transform section in the template the fragment will represent the entire CloudFormation template, with exception for the Transform section it self.
If we run it using the intrinsic function instead the fragment includes all of sibling nodes of the intrinsic function.
Yes that sounds a bot confusing, here is an example for clarification.
This is the parameters injected in the Parameters section in macro, in the example above that would be ‘Suffix’ : ‘-hello-world’
It is really important to remember that CloudFormation do’t evaluate the parameters before passing them to the macro. So we have to make sure that valid and required parameters are supplied.
The documentation state that params is only available when using the intrinsic function and would be empty if using Transform section.
I have found this not to be true and a template like the one below, from my CommonTags macro, will populate this section.
Parameters that are defined in the template Parameters section will be made available in this section.
Returned data from the Lambda
CloudFormation expect us to return a json data structure:
This is the request ID from the input event.
CloudFormation expect us to return “success”. Anything else will be treated as a failure and CloudFormation abort the creation. There is only one problem with this. Right now AWS don’t document any way of returning a failure with a message. If we set status to “failure” CloudFormation will just give an error stating that creation failed without an message.
AWS please fix this! Because things go wrong and we like to return failure when it does.
This is the actual CloudFormation template section in json format e.g.
What is really important to remember is that the fragment that we get in the input event will be totally replaced with the fragment that we return. So if there is part of the input fragment that we have not modified we must make sure we return that part as well.
The code powering everything would then look something like this.
Missing here if is of course validation that a suffix was actually supplied. Remember CloudFormation will not evaluate the parameters in the Parameters section.
The actual macro
Now let us create the actual Macro. This is done by creating a AWS::CloudFormation::Macro resource and deploying that to CloudFormation.
We need to supply the Macro with a Id or Name, this must be unique for the region. Remember Macros is only available in the region in which it is deployed. We must also supply a FunctionName, which is not the name but the actual ARN for the Lambda function. This is the function that we created earlier that will do the actual work.
We can also supply an optional description which can be useful to understand what the Macro does. We can also supply a LogGroupName and LogRoleARN for debugging, more about that in the debugging section.
Deploy and use it
Now when we have all our parts created, meaning the Lambda function and the Macro definition it’s time to deploy this to CloudFormation. This can easily be done via the CLI.
When we now have our macro deployed we can use it from the region that we deployed it in. This we do by including it in our templates, referencing it by it’s name.
Now when we deploy this template CloudFormation will call our Lambda function for transformation prior to creating the CloudFormation change set.
It’s just as easy as that.
To have CloudFormation output logs to CloudWatch we have to proper configure the Macro. We have to supply a LogGroupName and LogRoleARN in the Macro configuration. The documentation however doesn’t state if the Log Group need to exist or if it will be created by CloudFormation. In this case it actually need to exist, CloudFormation will not create the LogGroup, which other services, like Lambda, does. We therefor need to first create the Log Group before we can use it.
Below is an example template creating the Log group, policy needed, and the role. When everything is created and configured we get logs for the process in CloudWatch.
Logs for the Lambda function it self can also be found in CloudWatch, we just have to make sure the Lambda function has the proper permissions to create log groups, streams and write logs. Here we don’t have to create the Log Group, Lambda will automatically do that. Would be nice with some form of consistency.
Creating a CloudFormation macro is a rather easy and straight forward process. It brings really powerful extensibility to CloudFormation. Giving us the power to create functionality that is missing.
For me this is a total game changer and I can see myself using this all the time.
For more examples check out my GitHub repository and the AWS GitHub.