How to Run Python 3.11 on AWS Lambda with a Custom Runtime

2022-12-16

I’ve been enjoying the improvements in Python 3.11. I wanted to try them out on AWS Lambda, but it doesn’t yet support version 3.11. What’s more, it still doesn’t support 3.10! Thankfully, Lambda allows custom runtimes, making it possible to run other Python versions, albeit with a lot more configuration. Here’s how I did it.

The existing AWS Lambda Python base images contain everything that’s needed, but also a whole lot more that probably isn’t, since they need to support layers, extensions, and other extra features of lambda. These images are based on a custom version of Amazon Linux 2 (AL2). If the custom runtime needs to be as similar to the official image as possible, then it needs to use AL2 as a base. Corporate security policies are probably the main reason to use AL2 in this case. Doing so means building the newer Python and OpenSSL in the new image, because AL2 is based on RedHat 7.3 which is pretty old, and RedHat is conservative about updating packages in the first place. Now it’s getting pretty complicated just to run 3.10 or 3.11 on Lambda! Here’s what I came up with.

I started with this approach because I’ve had to build Python on AL2 in the past for other reasons, but eventually realized this was not necessary just to run on Lambda. The AWS documentation confused me at first, because I started by reading the developer guide from the Lambda runtimes section, which makes it appear more involved than it has to be for Python. The right place to start is Deploy Python Lambda functions with container images. Amazon provides a runtime interface client (RIC) that handles all the details described in the Lambda runtimes section above. Under Create a Python image from an alternative base image there’s a link to an example based on Alpine. The Python RIC readme has an example using the official Python Debian-based image, which I used as a starting point. The readme briefly explains how to use the Runtime Interface Emulator locally, but it can also be built into the image (as it is in the AWS base Python images), as is more fully explained in the runtime emulator readme. I don’t like the readme’s use of Dockerfile ADD (reasoning here). Use curl instead:

RUN curl https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie -Lo /usr/local/bin/aws-lambda-rie && \
	    chmod +x /usr/local/bin/aws-lambda-rie

Here’s what an entrypoint script could look like for Python. Also, note that a --log-level can be passed to awslambdaric, which can help with debugging.

#!/bin/sh

if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
  exec /usr/local/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric --log-level "debug" "$@"
else
  exec /usr/local/bin/python -m awslambdaric "$@"
fi

It’s a shame that AWS has fallen behind on this. Both the Python Lambda RIC and Lambda RIE have few commits in the past year or so, and seem barely maintained. I hope this helps someone run a newer version of Python on AWS Lambda.