How To Locally Debug Dependencies In A Lambda Layer?

I'm creating a lambda layer from a dockerfile that installs python packages to a directory and zips the result. FROM amazonlinux WORKDIR / RUN yum update -y # Install Python 3.7

Solution 1:

The problem you have here is that when opencv is installed its dependencies aren't going to be installed to your -t target location. They're getting installed to the default pip install location of <somewhere>/site-packages/ in the Docker image.

So when you end up zipping up your target location you're missing all of the dependencies. I would solve this by not providing a target to pip when you install opencv. Install it as you would any other package.

From within your Docker image call python -m site --user-site to get the pip install location.

Modify your Docker commands to zip up that entire directory after opencv has been installed, then use that for your zip to Lambda.

Solution 2:

I was able to use the more recent versions of scikit-learn and cv2 with the following approach that would automate the deployment process, and automatically reduce the size of the packages by removing unnecessary files **/*.py[c|o], **/__pycache__*, **/*.dist-info*:

I had to package both cv2 and scipy, where the package size was a huge problem, and I came to the following solutions in the end.

Using the serverless-python-requirements package on Serverless helped me streamline this whole process and reduce the package size as well. Would definitely recommend checking it out.

This is the guide that I followed

Serverless python-requirements plugin

Make sure to leave the strip flag to false to avoid stripping binaries which leads to the problem "ELF load command address/offset not properly aligned",

This is what my final serverless.yml came out to be which gave me the results I wanted to package sklearn + cv2 as a layer:

custom:pythonRequirements:dockerizePip:trueuseDownloadCache:trueuseStaticCache:falseslim:truestrip:falselayer:name:${self:provider.stage}-cv2-sklearndescription:PythonrequirementslambdalayercompatibleRuntimes:-python3.8allowedAccounts:-'*'

requirements.txt:

opencv-python-headless==4.4.0.42scikit-learn==0.23.2