Recent Posts

Archive

Tags

A *REAL* Serverless + Python Build

So...Serverless eh? Awesomesauce, ammirite? Free (as in beer and speach) tool to create AWS Lambda functions hooked to whatever triggers with like 10% of the yml as CloudFormation

But, like any tool (especially one as young as Serverless) it has it's...gaps.

1) I Want Python!

First off, like any sane person I don't want to write my server-side code in node.js so when dealing with Lambda that leaves Python as the best choice (IMHO). But, Serverless is a node app so we're going to have to do a bit of finagling. Writing a Python app we're going to want a Python based build to handle dependencies, linting, tests, etc. My goto for python is PyBuilder, but to keep the build contained to a single source we'll need to weave this together with Serverless. Here's my solution to this: in build.py

then in serverless.yml add

So, basically we're using PyBuilder to build the zip for Lambda and telling Serverless to use it, then using syscalls to work Serverless into the PyBuilder build. Also, note boto3 is added as a build dependency; this is because boto3 is automatically available to Lambda functions (more on this later). Also, I know. Syscalls. Ugly AF, but it works. Here have some eye bleach.

2) Handling Dependencies

Now for everyone's favorite topic: dependency management! Lambda requires that any third-party dependencies are vendored in the application zip. Thankfully we're already in PyBuilder land, so we can add this processing to our build. I do this by modifying the build.py from above like so (warning: more ugly ahead)

This basically finds where PyBuilder installed the dependency and then copies it into the directory we zip up for Lambda.

3) How to Work on a Team

So, any real development is going to be done with a team of people. From the Serverless docs:

In larger teams, each member should use a separate AWS account and their own stage for development.

This seems...a bit overkill. Creating separate AWS accounts for each dev, rubish I say. Instead it's easy enough to use Serverless' substitution processing to let all your devs use the same account. In serverless.yml

This necessitates a small change to the deploy and remove tasks in build.py

This gives every dev their own stage to play in, while letting CI/CD builds specify a stage with the environment variable SERVERLESS_STAGE.

4) Integration Testing

Now, let's tackle integration testing. Unit tests of course can be written with any off the shelf Python test framework supported by PyBuilder as usual. Adding integration tests turns out to actually be a simple affair. The key is that we know the name of the CloudFormation stack Serverless creates. So we can use boto3 to interrogate it to get the endpoint URL then make requests to it. In the integration test class, the following snippet gets all the CloudFormation outputs in a usable dict:

Then use the (added by Serverless by default) ServiceEndpoint output to ping your APIs.

5) Using Additional CloudFormation Resources

Finally, we can use the same technique as above to utilize additional AWS resources specified in serverless.yml. As we know the stack name at build time, we can use the pybuilder filter_resources plugin to add this to a config.ini packaged into the zip. From there we can use the same boto3 code as above to get the CloudFormation stack outputs needed to interact with the resources (actual code left as exercise to the reader).

Example Project

  • Facebook
  • Twitter

© 2017 Chris Arnold. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.