Pandas in AWS lambda gives numpy error

Question or problem about Python programming:

I’ve been trying to run my code in AWS Lambda which imports pandas. So here is what I’ve done.
I have a python file which contains a simple code as follows(This file has the lambda handler)

import json
print('Loading function')
import pandas as pd
def lambda_handler(event, context):
    return "Welcome to Pandas usage in AWS Lambda"

But when I test the lambda function in AWS Lambda, I get the below error:

Unable to import module 'lambda_function': Missing required dependencies ['numpy']

I already have numpy in the zipped package but still I get this error. I tried to follow the hints given at Pandas & AWS Lambda but no luck.

Did anyone ran into the same issue. Would appreciate any hint or suggestions to solve this problem.

Thanks

How to solve the problem:

Solution 1:

EDIT: I figured out finally how to run pandas & numpy in a AWS Lambda python 3.6 runtime environment.

I have uploaded my deployment package to the following repo:

git clone https://github.com/pbegle/aws-lambda-py3.6-pandas-numpy.git

Simply add your lambda_function.py to the zip file by running:

zip -ur lambda.zip lambda_function.py

Upload to S3 and source to lambda.

ORIGINAL:

The only way I have gotten Pandas to work in a lambda function is by compiling the pandas (and numpy) libraries in an AWS Linux EC2 instance following the steps from this blog post and then using the python 2.7 runtime for my lambda function.

Solution 2:

To include numpy in your lambda zip follow the instructions on this page in the AWS docs…

How do I add Python packages with compiled binaries to my deployment package and make the package compatible with AWS Lambda?

To paraphrase the instructions using numpy as an example:

1. Open the module pages at pypi.org.
   https://pypi.org/project/numpy/

2. Choose Download files.

3. Download:

For Python 2.7, module-name-version-cp27-cp27mu-manylinux1_x86_64.whl

e.g. numpy-1.15.2-cp27-cp27m-manylinux1_x86_64.whl

For Python 3.6, module-name-version-cp36-cp36m-manylinux1_x86_64.whl

e.g. numpy-1.15.2-cp36-cp36m-manylinux1_x86_64.whl

4. Uncompress the wheel file on the /path/to/project-dir folder.
   You can use the unzip command on the command line to do this. There are other ways obviously.

unzip numpy-1.15.2-cp36-cp36m-manylinux1_x86_64.whl

When the wheel file is uncompressed, your deployment package will be compatible with Lambda.

Hope that all makes sense 😉

The end result might look something like this.
Note: you should not include the whl file in the deployment package.

Solution 3:

After doing a lot of research I was able to make it work with Lambda layers.

Create or open a clean directory and follow the steps below:

Prerequisites: Make sure you have Docker up and running

1. Create a requirements.txt file with the following:

pandas==0.23.4
pytz==2018.7

2. Create a get_layer_packages.sh file with the following:

#!/bin/bash

export PKG_DIR=”python”

rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}

docker run –rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.6 \
pip install -r requirements.txt –no-deps -t ${PKG_DIR}

3. Run the following commands in the same directory:

chmod +x get_layer_packages.sh

./get_layer_packages.sh

zip -r pandas.zip .

4. Upload the layer to a S3 bucket.

5. Upload the layer to AWS by running the command below:

aws lambda publish-layer-version –layer-name pandas-layer –description “Description of your layer”
–content S3Bucket=,S3Key=.zip
–compatible-runtimes python3.6 python3.7

6. Go to Lambda console and upload your code as a zip file or use the inline editor.

7. Click on Layers > Add a layer> Search for the layer (pandas-layer) from the Compatible layers and select the version.

8. Also add the AWSLambda-Python36-SciPy1x layer which is available by default for importing numpy.

Selecting the layer from the console

9. Test the code. It should work now!!!!

Thanks to this medium article https://medium.com/@qtangs/creating-new-aws-lambda-layer-for-python-pandas-library-348b126e9f3e

Solution 4:

AWS Lambda use Amazon Linux operating system. Idea is download Pandas and NumPy compatible with Amazon Linux. What you download using pip is specific to Windows or Mac. You need to download the compatible version for Linux, so that your Lambda function can understand it. These files are called wheel files.

Create new local directory with lambda_function.py file. Install Pandas to local directory with pip:

$ pip install -t . pandas

Navigate to https://pypi.org/project/pandas/#files. Search for and download newest *manylinux1_x86_64.whl package. In my case, I’m using Python 3.6 on my Lambda function, so I downloaded the following:

• Pandas – pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

• NumPy – numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl

Download whl files to directory with lambda_function.py. Remove pandas, numpy, and *.dist-info directories. Unzip whl files.

$ rm -r pandas numpy *.dist-info
$ unzip numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl
$ unzip pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

Remove whl files, *.dist-info, and __pycache__. Prepare zip.zip archive:

$ rm -r *.whl *.dist-info __pycache__
$ zip -r zip.zip .

Upload the zip.zip file in your Lambda function.

Source: https://medium.com/@korniichuk/lambda-with-pandas-fd81aa2ff25e

Solution 5:

To get additional libraries in Lambda we need to compile them on Amazon Linux (this is important if the underlying library is based on C or C++ like for Numpy) and package them in a ZIP file together with the python script you want to run in Lambda.

To get the Amazon Linux compiled version of the libraries. You can either find a version that someone already compiled, like the one by @pbegle, or compile it yourself. To compile it ourself there are two options:
– compile the libraries on an EC2 instance https://streetdatascience.com/2016/11/24/using-numpy-and-pandas-on-aws-lambda/
– compile the libraries on a docker version of Lambda environment
https://serverlesscode.com/post/scikitlearn-with-amazon-linux-container/

Following the last option with Docker, it is possible to make it work using the instructions in the blog post above and by adding:

pip install --use-wheel pandas

in the script to compile the libraries:

https://github.com/ryansb/sklearn-build-lambda/blob/master/build.sh#L21

Solution 6:

Slightly duplicate of Cannot find MySQL in NodeJS using AWS Lambda

You need to package your libraries with Lambda. As lambda runs on a public cloud, you cannot configure it.

Now in your case, as you are using pandas, you need to package Pandas with your zip. Get a path to pandas(for example: /Users/dummyUser/anaconda/lib/python3.6/site-packages) and copy the library to the place where you have your lambda function code. Inside your code, refer to pandas from your local copy. While uploading, zip the whole set(code + libraries), and upload as you will. It should work.

Solution 7:

The main problem is that libraries compiled within a specific OS will only work for that OS. So, if a library is compiled within macOS then it will not run in Linux environment. Why is this a problem?

One of the dependencies of Pandas is Numpy (which is compiled due to speed, see this answer on stackoverflow). AWS Lambda uses Linux hence if the Numpy compilation is done within macOS or windows, then the compilation will only be good for those specific architectures and will not work within AWS. I think those using Linux will not experience this problem.

To solve this problem and create a working AWS Lambda layer for Pandas, follow these simple steps (see the general steps on amazon forum):

mkdir awsPandasLayer #create a directory
cd awsPandasLayer #cd into the directory
pip3 install -t . Pandas #install pandas and all its dependencies 
rm -r pandas numpy *.dist-info __pycache__ #clean up the environment to remove the incompatible numpy and pandas

Then download the latest precompiled packages for Pandas and Numpy into the awsPandasLayer directory earlier created. For my case, since I am using Python 3.7, I downloaded these versions of Pandas and Numpy. Notice cp37 denotes the python version as clearly stated on the site. Then complete the installation as follows:

unzip pandas-1.0.5-cp37-cp37m-manylinux1_x86_64.whl #unzip the pandas precompiled package
unzip numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl #unzip the numpy precompiled package
pip3 install -t . openpyxl #I found this is required within AWS Lambda for excel files
rm -r *.whl *.dist-info __pycache__ #clean up unneeded files
zip -r awsPandasLayer.zip . # zip all the files

The zip file can then be uploaded as an AWS Lambda layer and it should work. Note that I am using macOS.

Solution 8:

I tried some of the solution here but most of them didn’t work. I liked the idea @Ranadeep Guha suggested of creating a container and downloading the repos over there so that’s what I did.

I worked in the dir my lambda function was located on and created the following files :

Dockerfile :

FROM python:3.8-slim
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt -t /app

requirements.txt : (those were mine)

pandas
numpy
xmltodict

Now in my gitbash I run the following command that will generate a docker image for me with all the dependencies installed :

docker build -t image_name .

Sending build context to Docker daemon  5.632kB
Step 1/4 : FROM python:3.8-slim
 ---> 56930ef6f6a2
Step 2/4 : WORKDIR /app
 ---> Using cache
 ---> ea0bf539bcad
Step 3/4 : COPY requirements.txt ./
 ---> cb4c005f53cc
Step 4/4 : RUN pip install --no-cache-dir -r requirements.txt -t /app
 ---> Running in a0d179a372b4
Collecting pandas
  Downloading pandas-1.0.3-cp38-cp38-manylinux1_x86_64.whl (10.0 MB)
Collecting numpy
  Downloading numpy-1.18.3-cp38-cp38-manylinux1_x86_64.whl (20.6 MB)
Collecting xmltodict
  Downloading xmltodict-0.12.0-py2.py3-none-any.whl (9.2 kB)
Collecting pytz>=2017.2
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting python-dateutil>=2.6.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six>=1.5
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: numpy, pytz, six, python-dateutil, pandas, xmltodict
Successfully installed numpy-1.18.3 pandas-1.0.3 python-dateutil-2.8.1 pytz-2020.1 six-1.14.0 xmltodict-0.12.0

Now just create a Docker container of that image and confirm that everything installed :

winpty docker run --name container_name -it --entrypoint bash image_name

type ls and you will see all the installs.

Now let’s copy all the installs to your local pc. You can replace the dot with any location on your pc:

 docker cp container_id:/app/* . 

Solution 9:

I’ve been struggling with a similar error while trying to use the python3.6 engine. When I switched to 2.7 it worked fine for me. I used Amazon AMI to create my zip file, but it has only python3.5, not 3.6. I guess the version mismatch was the reason. But it’s just a guess, I haven’t tried the process on a python3.6 installation yet.

Solution 10:

with the serverless framework, you can easily package and deploy your dependencies correctly.

you only need to;

1. install serverless

   npm install -g serverless 

2. create a serverless.yml in the root of your project with the following:

   service: numpy-test # define the environment of your lambda provider: name: aws runtime: python3.6 # specify the function you want to deploy functions: numpy: # path to your lambda_handler function handler: path/to/function.lambda_handler # add a plugin that allows serverless to package python libraries # specified in the requirements.txt or Pipfile plugins: - serverless-python-requirements # this section makes sure your libraries get build correctly  # for an aws lambda environment custom: pythonRequirements: dockerizePip: non-linux 

   adjust the path/to/function.lambda_handler

3. make sure docker is running and execute

   serverless deploy

once the deployment is finished, go to the AWS console look for the function numpy-test-dev-numpy and test your function.

this article explains the necessary steps in detail.

Hope this helps!