Introducing Lambda Layers

2018-12-05

During Re:Invent 2018 a new concept was introduced to AWS Lambda, Dr. Werner Vogels introduced Layers. This is a way to create shared libraries that can be used across many functions without the need to duplicate code.
No more pulling in libraries or shared code when deploying your function.
No more confusion what version of the shared code each function uses.
No more risk of creating a monolith.
Layers are versioned and the function references an exact version and each function can update to new versions in its own pace.

Limitations

Before we start creating a layer, and a function using that layer, there are some limitations we must be aware about.
A function can reference max 5 layers, so we need to think carefully how our layers are constructed.
The total unzipped function size, that is the combined unzipped size of all layers and the function code, can not exceed 250mb, we can't just go and create huge layers.

Accessing Layers

We can reference layers published by AWS, other AWS customers, or layers we have created our self.
Since layers support resource based policies we can restrict access to specific accounts, organizations, or all accounts (public).
Will go more into the permission model later in this post.

Creating our first layer

Now it's time to create our first Layer, as usual CloudFormation is the way to go.

Using regular CloudFormation
I started created the layer using regular CloudFormation. That is creating a AWS::Lambda::LayerVersion. That generated a template looking like this:

HelloWorldLayer:
  Type: "AWS::Lambda::LayerVersion"
  Properties:
    CompatibleRuntimes: 
      - python2.7
      - python3.6
      - python3.7
    Content: 
      S3Bucket: my-layer-bucket
      S3Key: layer_test.zip
    Description: Simple Hello World Layer Test
    LayerName: hello-world
    LicenseInfo: MIT

That looks nice right? Well there are some pitfalls to this.....
Remember that I said that layers was versioned? So what happens if we update our code and redeploy the CloudFormation template?
Since all changes to a AWS::Lambda::LayerVersion requires replacement a new version will be deployed to the layer, that is what we wanted. But what happens to the old version? That is deleted, since updates requires replacement.
Ohhhhh that's not good! Now we can't reference old versions. How do we solve that then?

First we need to add a deletion policy with retain to our template

HelloWorldLayer:
  Type: "AWS::Lambda::LayerVersion"
  DeletionPolicy: Retain
  Properties:
    .....

That's not to much work, was that all? No it wasn't. We can't use the same logical ID for our resource. So for each update with a new version we need to update the template with a new logical id.

HelloWorldLayerXXX:
  Type: "AWS::Lambda::LayerVersion"
  DeletionPolicy: Retain
  Properties:
    .....

Updating the logical id every time I found to be a bit annoying and nothing I would like to keep doing.

Using SAM transform
I started to look into the support for layers that was introduced to SAM (Serverless Application Model). The SAM documentation states:

"When a Serverless LayerVersion is transformed, SAM also transforms the logical id of the resource so that old LayerVersions are not automatically deleted by CloudFormation when the resource is updated."

That is exactly what we want, now we can just update the code and redeploy the template and a new version will be created and the old version will be retained.
I started to create a SAM template to handle the layer:

HelloWorldLayer:
  Type: "AWS::Serverless::LayerVersion"
  Properties:
    LayerName: hello-world
    Description: Simple Hello World Layer
    ContentUri: !Sub 's3://${ArtifactBucket}/${ArtifactKey}'
    CompatibleRuntimes:
      - python2.7
      - python3.6
      - python3.7
    LicenseInfo: 'Available under the MIT-0 license.'
    RetentionPolicy: Retain

Perfect, let's deploy that and test.

"Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [HelloWorldLayer4c0a2672e7] is invalid. 'ContentUri' requires Bucket and Key properties to be specified"

What wait? What's wrong? I did specify a string to a s3 resources just as the documentation states. So why didn't it work? It turns out that it's not possible to use !Sub in the ContentUri. Instead let's try a "S3 Location Object" and use Ref.

HelloWorldLayer:
  Type: "AWS::Serverless::LayerVersion"
  Properties:
    LayerName: hello-world
    Description: Simple Hello World Layer
    ContentUri:
      Bucket: !Ref ArtifactBucket
      Key: !Ref ArtifactKey
    CompatibleRuntimes:
      - python2.7
      - python3.6
      - python3.7
    LicenseInfo: 'Available under the MIT-0 license.'
    RetentionPolicy: Retain

Great, we now have a layer published with name hello-world. Every time we change the code and deploy we should have a new version published.

But wait.... It still doesn't work! Old versions are still being deleted.

I figured out that both this problem and the problem with !Sub is due to SAM not resolving the intrinsic functions prior to transform. In the !Ref case that leads to the SAM macro thinking that nothing has changed and no new logical id is created. When the values then are resolved by CloudFormation the old version is deleted and a new is created, since values are actually changed.
This has been confirmed as a bug by the SAM team.

Unfortunately this force us to for now update the template with some static value, like modifying the Description or similar.

What about permissions?
When creating a layer it's possible to set access permissions on that layer. Restricting access to a specific account, organization, or all accounts (public). That is done by creating a LayerVersionPermission. Right now there is no support from SAM to create a LayerVersionPermission. We have to use regular CloudFormation to create our LayerVersionPermission.

HelloWorldLayerPermission:
  Type: "AWS::Lambda::LayerVersionPermission"
  DeletionPolicy: Retain
  Properties:
    Action: lambda:GetLayerVersion
    LayerVersionArn: !Ref HelloWorldLayer
    Principal: !Ref AWS::AccountId

In this template we assign permissions to our own account to get the LayerVersion, if we like to make the layer public just set Principal: *

And if we like to give all accounts in our Organization permission we just have to change a few things.

HelloWorldLayerPermission:
  Type: "AWS::Lambda::LayerVersionPermission"
  DeletionPolicy: Retain
  Properties:
    Action: lambda:GetLayerVersion
    LayerVersionArn: !Ref HelloWorldLayer
    OrganizationId: "Your Organization ID"
    Principal: *

However the same problem exists for LayerVersionPermission that updates require replacement. Meaning that if the ARN for the LayerVersion updates, the LayerVersionPermission for the old LayerVersion is removed. Therefor we need to Retain the LayerVersionPermission resource and create new ones with different logical IDs.

All this make handling Lambda Layers from CloudFormation a bit tedious.

CloudFormation Conclusion
That was a lot of limitations and problems. Handling Lambda Layers with CloudFormation is not straight forward and we need to update the template before deploying new versions. We can't just change the code and redeploy to have a new stable version. Mixing SAM and normal CloudFormation is not working out either, I think. This only make things confusing. Right now I would stick to normal CloudFormation all the way and update the Logical ID on my own or via a Macro

That would give us a complete template looking like this:

HelloWorldLayerXYZ:
  Type: "AWS::Lambda::LayerVersion"
  DeletionPolicy: Retain
  Properties:
    CompatibleRuntimes: 
      - python2.7
      - python3.6
      - python3.7
    Content: 
      S3Bucket: !Ref ArtifactBucket
      S3Key: !Ref ArtifactKey
    Description: Simple Hello World Layer
    LayerName: hello-world-
    LicenseInfo: MIT

HelloWorldLayerPermissionXYZ:
  Type: "AWS::Lambda::LayerVersionPermission"
  DeletionPolicy: Retain
  Properties:
    Action: lambda:GetLayerVersion
    LayerVersionArn: !Ref HelloWorldLayer
    Principal: !Ref AWS::AccountId

Implementing the layer
What we need to understand when we create a layer is that we are not limited to only one language, we can mix python, node, etc in the same layer. If we would like to share our layer between functions written in different languages that is fully possible, without creating duplicate layers.
Therefor we need to understand how the layer code is extracted and presented to the function.
The layer code is extracted to the /ops folder in the function runtime. Each runtime then looks for the code in different locations under /ops
We should organize the code as shown below.

helloworld.zip  
└ python/helloworld  
└ nodejs/node-modules/helloworld  
└ java/lib/helloworld.jar

I will focus on python and you can experiment with more languages on your own. The zip file for python end up with the following structure.

helloworld.zip  
└ python/helloworld/__init__.py 
└ python/helloworld/helloworld.py

The code in helloworld.py is just a super basic hello world print

def print_hello_world():
    print ('Hello World')

Creating the Lambda function

Now it's time to create the actual function using our newly created Layer.
We start by defining the CloudFormation template to deploy the function, we use SAM here as well and the new property Layers which is a list of Arn:s

HelloWorldFunction:
  Type: AWS::Serverless::Function
  Properties:
    FunctionName: hello-world-layer-function
    Runtime: python2.7
    Timeout: 60
    CodeUri: ./src
    Handler: handler.handler
    Layers:
      - !Ref LayerVersionArn

The code for this function is not that complex.

# Import our print function from layer helloworld
from helloworld.helloworld import print_hello_world

def handler(event, context):
    print_hello_world()
    return "Success!"

Conclusion

That is basically it. We have now created a simple hello-world layer written in python and used that layer from a new function.
There is really good potential in layers and this will help us reduce code duplications and we can organize our functions and libraries and a much nicer way. We just have to keep the limits in mind so there is no surprises.