Introduction

As a developer working with AWS Lambda, I’ve always loved the idea of serverless computing - deploying code without worrying about managing servers. However, one challenge that frequently came up was handling Python dependencies in Lambda functions. Deploying a Lambda function with multiple dependencies can be cumbersome, especially when you have to include these libraries in your deployment package every time. This challenge led me to explore AWS Lambda Layers, and I discovered a way to simplify my workflow by building and deploying a Lambda Layer directly from a GitHub Actions pipeline.

In this blog post, I will walk you through the process of setting up an AWS Lambda Layer for Python using GitHub Actions, which allows you to dynamically manage dependencies without cluttering your function code.

What are AWS Lambda Layers, and Why Do You Need Them?

AWS Lambda Layers provide a way to package and share common code across multiple Lambda functions. This is particularly useful for:

  • Managing Dependencies: Instead of including all the libraries and dependencies in each Lambda function, you can package them once in a layer and share it across multiple functions. This makes your function codebase cleaner and smaller.
  • Reusability: Layers promote code reusability. If you have a common utility or library that multiple functions need, you can package it in a layer and reference it wherever needed.
  • Version Control: Each time you update a layer, AWS provides versioning, so you can manage changes and roll back if necessary.

Setting Up GitHub Actions for AWS Lambda Layers

Before we dive into setting up the pipeline, if you haven't already, I recommend checking out my previous blog post on how to securely connect GitHub Actions to AWS. This will ensure that your workflow is not only functional but also secure when interacting with AWS services.

How to Securely Connect GitHub Actions to AWS
Introduction Every DevOps engineer knows the importance of secure, streamlined workflows. A while ago, I found myself managing an array of AWS resources across multiple accounts. My goal was to automate deployments and other tasks using GitHub Actions. The catch? I wanted to avoid hardcoding AWS secret keys in my
Alexander HoseAlexander Hose

Defining the GitHub Workflow

The workflow begins with defining when the action will run. In our case, it triggers on every push or pull request to the main branch.

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

Setting Up Environment Variables

To make the workflow flexible and easy to configure, environment variables are defined at the beginning. These include the AWS region, Python version, and the libraries you want to include in your Lambda layer.

env:
  AWS_REGION: "us-east-1"
  PYTHON_VERSION_SHORT: "3.12"
  PYTHON_LIBRARIES: "requests"

Permissions and Job Setup

The permissions block is necessary for GitHub Actions to interact with AWS, especially when assuming a role and managing contents.

env:
  AWS_REGION: "us-east-1"
  PYTHON_VERSION_SHORT: "3.12"
  PYTHON_LIBRARIES: "requests"

The BuildAndDeployLambdaLayer job is where all the action happens. It runs on ubuntu-latest and consists of several steps, each playing a critical role in the workflow.

jobs:
  BuildAndDeployLambdaLayer:
    runs-on: ubuntu-latest

Cloning the Repository and Configuring AWS Credentials

The first two steps involve checking out the repository code and setting up AWS credentials. This ensures the pipeline has access to the required IAM role to create or update Lambda layers.

steps:
- name: Git clone the repository
  uses: actions/checkout@v4

- name: configure aws credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/CreateLambdaLayer
    aws-region: ${{ env.AWS_REGION }}

Installing Python and Virtualenv

To manage the Python dependencies efficiently, we use virtualenv. This ensures that the installed packages are isolated and can be bundled cleanly into a Lambda layer.

- name: Set up Python
  uses: actions/setup-python@v4
  with:
    python-version: ${{ env.PYTHON_VERSION_SHORT }}

- name: Install Virtualenv
  run: |
    python -m pip install --upgrade pip
    pip install virtualenv

Preparing the Lambda Layer

This step involves creating a virtual environment, installing the required Python libraries, and packaging them into a ZIP file that AWS Lambda can use as a layer.

- name: Prepare Lambda Layer with Dynamic Libraries
  run: |
    PYTHON_LIBRARIES="${{ env.PYTHON_LIBRARIES }}"
    mkdir -p requests/python
    cd requests/
    virtualenv venv -p python${{ env.PYTHON_VERSION_SHORT }}
    source ./venv/bin/activate
    pip install ${PYTHON_LIBRARIES} > /dev/null 2>&1
    cp -r ./venv/lib/python${{ env.PYTHON_VERSION_SHORT }}/site-packages/* ./python
    zip -r ${PYTHON_LIBRARIES}.zip ./python > /dev/null

Publishing the Lambda Layer

Finally, the script publishes the newly created Lambda layer. A neat trick here is dynamically replacing the . in the Python version with a dash (-), as AWS Lambda layer names do not support dots.

- name: Publish Lambda Layer
  run: |
    PYTHON_VERSION_SHORT="${{ env.PYTHON_VERSION_SHORT }}"
    PYTHON_VERSION_SHORT_DASH="${PYTHON_VERSION_SHORT//./-}"
    aws lambda publish-layer-version --layer-name ${{ env.PYTHON_LIBRARIES }}-python${PYTHON_VERSION_SHORT_DASH} --zip-file fileb://requests/${{ env.PYTHON_LIBRARIES }}.zip --compatible-runtimes python${{ env.PYTHON_VERSION_SHORT }}

Why This Workflow is a Game Changer

This pipeline provides a seamless way to manage and deploy AWS Lambda layers directly from your GitHub repository. With this setup, you can:

  • Automate Dependency Management: Automatically package and deploy Python dependencies as a Lambda layer without manual intervention.
  • Keep Codebases Clean: Separate your business logic from the dependency management, making your Lambda functions lighter and more maintainable.
  • Easily Update Layers: Whenever you need to update a dependency, simply modify the environment variable, and the pipeline will handle the rest.

Conclusion: The Power of AWS Lambda Layers

Integrating AWS Lambda Layers into your serverless applications can significantly simplify your workflow, especially when combined with GitHub Actions for continuous integration. This approach not only streamlines your deployment process but also promotes best practices in managing shared code and dependencies across multiple Lambda functions.

By automating the creation and deployment of Lambda Layers, you can focus more on writing code that delivers value and less on the intricacies of dependency management. Whether you are new to AWS Lambda or a seasoned pro, this setup will undoubtedly make your serverless journey smoother and more efficient.

Full Pipeline Code

To help you get started quickly, here’s the full pipeline code that you can copy and paste directly into your GitHub repository. This workflow will automate the process of building and deploying an AWS Lambda Layer with the specified Python libraries.

name: Build and Deploy AWS Lambda Layer

# Controls when the action will run. Invokes the workflow on push events but only for the main branch
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

env:
  
  AWS_REGION : "us-east-1" #Change to reflect your Region
  PYTHON_VERSION_SHORT : "3.12" #Change to python version
  PYTHON_LIBRARIES: "requests" #Change for layer library

# Permission can be added at job level or workflow level    
permissions:
      id-token: write   # This is required for requesting the JWT
      contents: read    # This is required for actions/checkout
jobs:
  BuildAndDeployLambdaLayer:
    runs-on: ubuntu-latest
    steps:
      - name: Git clone the repository
        uses: actions/checkout@v4
      - name: configure aws credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/CreateLambdaLayer #change to reflect your IAM role’s ARN
          aws-region: ${{ env.AWS_REGION }}
      # Hello from AWS: WhoAmI
      - name: Sts GetCallerIdentity
        run: |
          aws sts get-caller-identity
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION_SHORT }}
      - name: Install Virtualenv
        run: |
          python -m pip install --upgrade pip
          pip install virtualenv
      - name: Prepare Lambda Layer with Dynamic Libraries
        run: |
          PYTHON_LIBRARIES="${{ env.PYTHON_LIBRARIES }}"

          mkdir -p requests/python
          cd requests/
          virtualenv venv -p python${{ env.PYTHON_VERSION_SHORT }}
          source ./venv/bin/activate

          echo "Installing Python libraries: ${PYTHON_LIBRARIES}"
          pip install ${PYTHON_LIBRARIES} > /dev/null 2>&1

          cp -r ./venv/lib/python${{ env.PYTHON_VERSION_SHORT }}/site-packages/* ./python
          zip -r ${PYTHON_LIBRARIES}.zip ./python > /dev/null
          echo "Lambda layer with ${PYTHON_LIBRARIES} created successfully."
      - name: Publish Lambda Layer
        run: |
          PYTHON_VERSION_SHORT="${{ env.PYTHON_VERSION_SHORT }}"
          PYTHON_VERSION_SHORT_DASH="${PYTHON_VERSION_SHORT//./-}"
          echo "Attempting to publish Lambda layer..."
          aws lambda publish-layer-version --layer-name ${{ env.PYTHON_LIBRARIES }}-python${PYTHON_VERSION_SHORT_DASH} --zip-file fileb://requests/${{ env.PYTHON_LIBRARIES }}.zip --compatible-runtimes python${{ env.PYTHON_VERSION_SHORT }}

Copy this code into a new file named .github/workflows/aws-create-lambda-layer.yml in your repository. Modify the environment variables according to your needs, and you're all set to automate the creation and deployment of your AWS Lambda Layers!

Share this post