AWS Lambda Research - SimPPL/arbiter-documentation GitHub Wiki

Introduction

Currently, we are using a t2.micro EC2 instance to host our arbiter django backend–a website where we query (using DuckDB) hashtags, narratives, links, and keywords across datasets of 300GB from multiple social platforms that are stored as parquet files on AWS S3. Two API endpoints in the backend are processing large amounts of data, but limited compute and RAM of t2.micro becomes a bottleneck for it. Using a t2.medium instance solved all the bottleneck issues, but the cost of continuously running a t2.medium was not coming within our budget. To resolve the cost issue we started looking into AWS Lambda.

AWS Lambda helps reduce cost in two ways:

  1. It spins up the backend server only when API calls are made and goes back to sleep after that, thus saving server running cost.
  2. It scales up and down automatically as per the API calls being made.

So the bottleneck problem we were facing in our 2 API endpoints will be fixed, and the scaling will only happen for these two APIs, thus reducing cost as compared to t2.medium where the entire backend is continuously running on the expensive instance.

AWS Lambda

AWS Lambda API responses are limited to a maximum of 6MB, but most of our APIs returned responses over 6MB. So this was not feasible. AWS Lambda supports 20MB of streamed responses, so I started looking in this direction. It turns out Lambda by default only supports streamed responses in Node.js, but our backend is python based. To enable streamed responses in python we will have to create a custom python runtime on lambda. I started researching about it, but it was difficult for me to understand how to create one for the django application. So I backed up a bit and decided to first understand how to deploy a django application to Lambda and then figure out the streamed response part.

I found a lot of resources on this and understood that to achieve the task of deploying a django application to lambda we need to use a library called Zappa. I followed a 3 month old YouTube tutorial to deploy a django application on AWS Lambda using Zappa & GitHub Actions. But I stumbled upon a lot of errors one after the other. I was able to resolve all but one, which I am currently stuck at.

Errors

Some of the errors I faced are as follows -

  1. Python Version Error

    I configured zappa to use Python 3.9 as shown in the tutorial I was following, but the django application I was trying to deploy was using a more recent version of python. I updated zappa settings to use Python 3.12, which is the most recent stable version of Python, but Lambda does not support Python 3.12 yet. Using Python 3.11 solved this issue.

  2. Permission Error

    Make sure the AWS Access Keys you are using have Lambda & Cloudformation permissions.

  3. Size Limit Exceeded Error

    Lambda has a max limit of 260MB for the entire codebase including all the packages. Our project was exceeding this limit. To overcome this issue you can enable the slim_handler option in the zappa settings, so as to compress the project and bring it below 260MB.

  4. This is the error I am currently stuck at. The error is as follows -

    Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.
Deploying API Gateway..
Waiting for lambda function [arbiter-production] to be updated...
Error: Process completed with exit code 1.

GitHub Repository - https://github.com/SimPPL/arbiter-backend/tree/deploy/aws-lambda

Deployment Errors - https://github.com/SimPPL/arbiter-backend/actions

Inputs from Rohan

  1. Provided a yaml template to understand how lambda functions are deployed and gateways are configured.
  2. Only current solution for using streamed responses is with a Node.js x Django setup, which uses .py.vhl. Vhl gives access to endpoints. Use them in node.js to apply streamed responses further

Awaiting further inputs from Rohan

  1. How are handler files configured?
  2. How to setup Node + python environment synchronously?
  3. Understand how Zappa works

Our Inference

Our APIs return responses over 6MB, ∴ Streamed Responses to be configured. Streamed responses are compatible only with Node.js, ∴ 2 ways to go about it -

  1. Node.js x Django synchronous setup required
  2. Custom Python Environment setup required

Both of these methods are currently blockers

Given the stage the project is in, migration of backend deployment from EC2 t2.micro to AWS Lambda entails the need to setup a Node.js server from scratch, along with handler file configurations by the backend team for each specific API endpoint and modifying the integration by the frontend team, making it a team-wide endeavour. Also to be noted that this process must be followed for every new API endpoint to be created in the future.

Hence it is infeasible to be implemented within the scope of the given project.

Resources

What is AWS Lambda? - AWS Lambda

Configuring a Lambda function to stream responses

Building a custom runtime for AWS Lambda

What happens inside an AWS Lambda | AntStack

Building a custom runtime on AWS Lambda | AntStack

Streaming Amazon Bedrock with AWS Lambda on a custom python runtime | AntStack

Deploy Django projects to AWS lambda using Zappa and Github Action