A tale of an CloudFormation import

2021-02-13

Infrastructure as Code is one of the most important practices when it come to automation and DevOps. Even though more and more teams and companies enforce that setup of all infrastructure should be automated and defined as code, you can still run into key resources that have been setup manually. Several tools like CloudFormation or Terraform allow you to import manually created infrastructure. Is the process straight forward? Can everything just be imported out of the box? Recently I run into the need to import resources into a CloudFormation stack, this is my tale.

What to import

What I needed to import was a manually created Amazon CloudFront distribution with an S3 bucket as origin. So basically three resources, an S3 bucket with a specified bucket policy, a Origin Access Identity, and the CloudFront distribution.

Creating the template

Before we can start the import of infrastructure we must define the CloudFormation template that define the setup of the resource. The challenge is to make sure all configuration is exact, because if there is a mismatch between what is defined in the CloudFormation template and what as been configured manually a update to an imported stack can cause problems.
So how do we catch all of the config then? Former2 to the rescue! This is a great tool, it scans your AWS account for resources and you can select the one you like to include in a CloudFormation template. You can Read more about Former2 in this blog post.
Former2 created a perfect template for me with the resources I needed and all the configuration.

Starting the import

To start the import process navigate to CloudFormation part of the AWS Console and select to create stack with existing resources.

image{:class="img-responsive"}

You will be met by a good guide and you need to start by uploading your template.

image{:class="img-responsive"}

In my case I just uploaded the template I exported from Former2 containing the resources, S3 Bucket, Bucket Policy, Origin Access Identity, and the CloudFront Distribution. After hitting the next button I was met by a big red error banner.

image{:class="img-responsive"}

This was a huge gotcha, not all resources can actually be imported.

What to do

Not being able to import the resources caused a bit of a headache, sure I could add the Bucket Policy, and Origin Access Identity in a second step, that was not that big of a problem. Not being able too import the CloudFront Distribution, that was big problem. I needed a plan!

The plan

I needed a plan. The question now was, "How do I import the resources with minimal downtime?" I had already faced the facts. There was going to be downtime, now I needed to minimize it. I basically had two options. Either I recreate all resources or I import what I can and recreate the rest. The S3 bucket already contained data, sure it could be recreated but since the bucket need to be emptied before it can be deleted and recreated this would make the downtime longer.
The plan was set, I needed to import what I could and recreate what I needed.

Creating the step by step

I figured out that I needed to test and created a step by step to make sure I minimized downtime. I tested back and forth but finally I had a step by step that would cause 15-20minutes of downtime, depending how fast I was.
I decided to use SAM CLI even if there was no SAM resources involved, it was pure CloudFormation. There was a good reason for that. I could in advance prepare a samconfig.toml file with all parameters, stack name, everything. I could the just run sam deploy{:.language-bash} to deploy the resources, which would be short and fast to type.
I started with creating the full CloudFormation template and the samconfig.toml file and tested the deploy to make sure a full stack was created as it should. Everything was created as it should, and I was happy.
Since I always write my CloudFormation templates in YAML I could easily use comments to quickly add and remove resources in the template. In the end my step by step in the end looked like this.

1. Import the S3 bucket into a new CloudFormation stack.
2. Do a test deploy to make sure I could easy update the stack.
3. Delete the Bucket Policy and the Origin Access Identity.
4. Update the template and add the Bucket Policy and the Origin Access Identity, deploy an update to the stack.
5. While the stack updated delete the Route53 record and the CloudFront distribution.
6. Update the template and add the CloudFront distribution, deploy an update to the stack.
7. Update the template and add the Route53 record, deploy an update to the stack.

This worked like a charm. I decided not to delete all resources at once since that made it possible to do things in parallel. Deleting a CloudFront distribution still take around 10 minutes, and while the Bucket Policy and Origin Access Identity was deploying I could start the delete process of the CloudFront distribution.

Conclusion

The possibility to import resources, that have been created manually, into CloudFormation is really beneficial. Tools like Former2 is an amazing assistance in the process and makes everything easier!

That it's not possible to import all resource types is huge flaw. Since all resources has some form of physical ID it should be possible. I don't even ask for a fully automatic process, I could manually map a create resources to a logical ID in a CloudFormation template during the import process.
I really like CloudFormation and it's my weapon of choice for creating infrastructure in AWS. Unfortunately I keep running into problems, everything from a incomplete coverage, still there are some resources or configs missing in CloudFormation, too import not supporting everything.

This time I was able to solve the problem, and I will keep solving them as I run into them. Just hoping that AWS, sometime in the future, make sure that CloudFormation comes with full coverage.

Happy import everyone!