Elastic Beanstalk SciPy.* fails silently
Question:
Forgive me in advance, this issue I’m facing has a lot of little nuances that make it difficult to articulate.
First of all, the application:
The application is a simple Flask app. The default GET / route will simply return an object response (static) and makes no calls to SciPy or any other library other than Flask. There is then a POST route that if called with the right Schema will run a model (imported to the application.py from another local file). This model utilizes SciPy, Matplotlib and Numpy.
So simply put, the only thing that happens on application.py start, is import Flask and “model”.py which in turn imports matplotlib, numpy and a few sub components of scipy.
The EB environment:
It is a Amazon Linux environment on a t2 medium server running Python 2.7. There is a load balancer which sits in front of the service.
The issue:
The issue comes in the application starting when any SciPy sub component is imported. There are no errors in the logs, scipy installs fine and is satisfied. If I attempt to hit the GET / route, which itself doesn’t use scipy, it simply timesout in responding. If I remove all imports of scipy from the model the GET route will respond without issue, obviously though the POST route will fail is it doesn’t have requirements when actually executed. If i simply at import scipy the GET route will work (still model will fail as certain requirements are no longer stated).
So really the issue occurs whenever there is any from scipy import .... The application will just silently timeout, no errors in the logs.
This has me at a bit of a loss and any help or suggestions would be greatly appreciated!
Answers:
The issue is with WSGI accessing the Python compiler, see:
The consequences of attempting to use a C extension module for Python which is implemented against the simplified API for GIL state management in any sub interpreter besides the first, is that the code is likely to deadlock or crash the process. The only way around this issue is to ensure that any WSGI application which makes use of C extension modules which use this API, only runs in the very first Python sub interpreter created when Python is initialised.
You will need to add WSGIApplicationGroup %{GLOBAL} to your wsgi.conf file on the server.
I’ve posted a full overview of this issue to my blog:
http://djm.io/deploying-scipy-into-aws-elastic-beanstalk/
Forgive me in advance, this issue I’m facing has a lot of little nuances that make it difficult to articulate.
First of all, the application:
The application is a simple Flask app. The default GET / route will simply return an object response (static) and makes no calls to SciPy or any other library other than Flask. There is then a POST route that if called with the right Schema will run a model (imported to the application.py from another local file). This model utilizes SciPy, Matplotlib and Numpy.
So simply put, the only thing that happens on application.py start, is import Flask and “model”.py which in turn imports matplotlib, numpy and a few sub components of scipy.
The EB environment:
It is a Amazon Linux environment on a t2 medium server running Python 2.7. There is a load balancer which sits in front of the service.
The issue:
The issue comes in the application starting when any SciPy sub component is imported. There are no errors in the logs, scipy installs fine and is satisfied. If I attempt to hit the GET / route, which itself doesn’t use scipy, it simply timesout in responding. If I remove all imports of scipy from the model the GET route will respond without issue, obviously though the POST route will fail is it doesn’t have requirements when actually executed. If i simply at import scipy the GET route will work (still model will fail as certain requirements are no longer stated).
So really the issue occurs whenever there is any from scipy import .... The application will just silently timeout, no errors in the logs.
This has me at a bit of a loss and any help or suggestions would be greatly appreciated!
The issue is with WSGI accessing the Python compiler, see:
The consequences of attempting to use a C extension module for Python which is implemented against the simplified API for GIL state management in any sub interpreter besides the first, is that the code is likely to deadlock or crash the process. The only way around this issue is to ensure that any WSGI application which makes use of C extension modules which use this API, only runs in the very first Python sub interpreter created when Python is initialised.
You will need to add WSGIApplicationGroup %{GLOBAL} to your wsgi.conf file on the server.
I’ve posted a full overview of this issue to my blog:
http://djm.io/deploying-scipy-into-aws-elastic-beanstalk/