Python web hosting: Numpy, Matplotlib, Scientific Computing
Question:
I write scientific software in Numpy/Scipy/Matplotlib. Having developed applications on my home computer, I am now interested in writing simple web applications. Example: user uploads image or audio file, my program processes it using Numpy/Scipy, and output is displayed on the browser using Matplotlib, or perhaps the user can download a processed file.
I already pay for hosting that does have Python 2.4.3 installed, but no Numpy/Scipy. I don’t have shell access via command line, either. Just drag-and-drop FTP. Pretty limited, but I can get simple Python/CGI scripts working.
Surprisingly, a web search revealed few suitable options for web hosting with these capabilities already built in. (Please guide me if I am wrong.) I am learning about the Google App Engine, but I still don’t have a full understanding about its tools and limitations. What the web did tell me is that others have similar concerns.
Hoping for solutions, I thought I would ask these simple questions to the awesome SO community:
-
Is there a simple way of installing numpy (or any third-party package/library) onto my already hosted space? I know the Python path on my hosted space, and I know the relevant Python/Numpy directories on my home computer. Can I simply copy files over and have it work? Both local and remote systems run Ubuntu.
-
What hosting sites exist (either free or paid) which have Numpy/Matplotlib installed or, if not installed, the possibility of installing it? Are there any documented sites that you can reference with working applications, no matter how simple?
-
Can Google App Engine help me in any way? Or is it totally for something else? Have you or others used it to write scientific applications in Python/Numpy? If so, could you reference them?
Thank you for your help.
EDIT: After the useful answers below, I bought the $20 plan at Slicehost, and I love it so far! (I first tried Amazon EC2. I must be stupid, but I just couldn’t get it to work.) Setting up the Ubuntu server with Apache took mere hours (and I’m an Apache novice). It allows me to do exactly what I wanted with Python plus much more. I now have my own remote repository for version control, too. Thanks again!
EDIT 2: Nearly two years later, I tried Linode and EC2 (again). Linode is great. EC2 seemed easier this time around — maybe it’s just added experience, or maybe it’s the improvements that Amazon made to the AWS management console. For those interested in Numpy/Scipy/Matplotlib/Audiolab, here is my Ubuntu cheat sheet whenever I launch an EC2 instance:
ec2:~$ sudo aptitude install build-essential python-scipy ipython
python-matplotlib python-dev python-setuptools libsndfile-dev
libasound2-dev mysql-server python-mysqldb
Upload scikits.audiolab-0.11.0
ec2:~/scikits.audiolab-0.11.0$ sudo python setup.py install
ec2:~$ sudo rm -rf scikits.audiolab-0.11.0
ec2:~$ nano .ipython/ipy_user_conf.py
ip.ex('import matplotlib; matplotlib.use("Agg"); import scipy, pylab,
scipy.signal as sig, scipy.linalg as lin, scipy.sparse as spar,
os, sys, MySQLdb, boto; from scikits import audiolab')
import ipy_greedycompleter
import ipy_autoreload
Answers:
App Engine does not support any of numpy, scipy, or matplotlib, alas.
If you know exactly what OS and CPU your host is using, you could make an identical installation for yourself, download and install the same version of Python they’re using, download the sources of packages you require and build them into .so (or .pyd, depending on the platform) files, and upload those — sounds like a real tour de force.
Any of the many, many sites that offer normal virtual hosting (a virtual machine, typically Linux, with modest HW resources, but root privileges for you, ssh shell access, and a gcc you can use in particular) will be much easier to work with — essentially, you’ll download and install the software you need just about the same way you’d do on your own Linux workstation!
1: Installing third party packages to hosted spaces
You can indeed install third party packages to your hosted space. If it’s a pure python package, all that’s needed is to unpack it to a directory and then add that directory to your PYTHONPATH environment variable or sys.path.
This can be tiring to do often, and won’t work easily for compiled modules. If you have shell access to your python host, the excellent virtualenv package allows you to do set up a private python environment with its own libraries.
To set up your virtualenv, you’ll do something like this at the shell:
$ virtualenv $HOME/my_python
$ $HOME/my_python/bin/easy_install numpy
You can keep running easy_install for anything else you want to install in your personal python environment.
Now, when you write your python scripts, you will want to use your private python interpreter, if that is possible:
#!/home/myuser/my_python/bin/python
import numpy
# script here
If your python env cannot be specified (such as if run by mod_wsgi), you will need to add it to the import path:
import sys
sys.path.insert(0, '/home/myuser/my_python/lib/python2.5/site-packages')
import numpy
2: Hosting sites with numpy
I can’t think of any hosting sites offhand which offer numpy pre-installed. However, Dreamhost/Bluehost for sharedhosts provide SSH access, and with shell access you can install numpy using the methods I described above. Any Virtual Private Server such as Linode/Slicehost will allow you to install whatever you desire, as well.
3: AppEngine
As mentioned above, AppEngine will not allow you to install C extensions (but pure python ones do work) so it’s unlikely numpy will work for you on there, since I suspect some of its features use C speedups.
I do not have privledges yet to comment but I can provide an “answer”.
3: AppEngine
Numpy is now available on Google App Engine:
https://code.google.com/p/googleappengine/issues/detail?id=190
However, matplotlib is still waiting:
http://code.google.com/p/googleappengine/issues/detail?id=482
Perhaps more people starring this issue will make it happen.
I will note that svgfig is an option as it is pure python:
http://code.google.com/p/svgfig/
Update:
Turns out matplotlib is now available:
https://developers.google.com/appengine/docs/python/tools/libraries27#matplotlib
2 What hosting sites exist (either free or paid) which have Numpy/Matplotlib installed
PythonAnywhere offers web hosting and a simple in-browser IDE; lots of Python packages (including NumPy and Matplotlib) are pre-installed. There’s a free plan that you can use to play around, and “Premium” and “Hosting” accounts with more features are $5 and $10/month respectively.
Full disclosure: I work there…
I write scientific software in Numpy/Scipy/Matplotlib. Having developed applications on my home computer, I am now interested in writing simple web applications. Example: user uploads image or audio file, my program processes it using Numpy/Scipy, and output is displayed on the browser using Matplotlib, or perhaps the user can download a processed file.
I already pay for hosting that does have Python 2.4.3 installed, but no Numpy/Scipy. I don’t have shell access via command line, either. Just drag-and-drop FTP. Pretty limited, but I can get simple Python/CGI scripts working.
Surprisingly, a web search revealed few suitable options for web hosting with these capabilities already built in. (Please guide me if I am wrong.) I am learning about the Google App Engine, but I still don’t have a full understanding about its tools and limitations. What the web did tell me is that others have similar concerns.
Hoping for solutions, I thought I would ask these simple questions to the awesome SO community:
-
Is there a simple way of installing numpy (or any third-party package/library) onto my already hosted space? I know the Python path on my hosted space, and I know the relevant Python/Numpy directories on my home computer. Can I simply copy files over and have it work? Both local and remote systems run Ubuntu.
-
What hosting sites exist (either free or paid) which have Numpy/Matplotlib installed or, if not installed, the possibility of installing it? Are there any documented sites that you can reference with working applications, no matter how simple?
-
Can Google App Engine help me in any way? Or is it totally for something else? Have you or others used it to write scientific applications in Python/Numpy? If so, could you reference them?
Thank you for your help.
EDIT: After the useful answers below, I bought the $20 plan at Slicehost, and I love it so far! (I first tried Amazon EC2. I must be stupid, but I just couldn’t get it to work.) Setting up the Ubuntu server with Apache took mere hours (and I’m an Apache novice). It allows me to do exactly what I wanted with Python plus much more. I now have my own remote repository for version control, too. Thanks again!
EDIT 2: Nearly two years later, I tried Linode and EC2 (again). Linode is great. EC2 seemed easier this time around — maybe it’s just added experience, or maybe it’s the improvements that Amazon made to the AWS management console. For those interested in Numpy/Scipy/Matplotlib/Audiolab, here is my Ubuntu cheat sheet whenever I launch an EC2 instance:
ec2:~$ sudo aptitude install build-essential python-scipy ipython
python-matplotlib python-dev python-setuptools libsndfile-dev
libasound2-dev mysql-server python-mysqldb
Upload scikits.audiolab-0.11.0
ec2:~/scikits.audiolab-0.11.0$ sudo python setup.py install
ec2:~$ sudo rm -rf scikits.audiolab-0.11.0
ec2:~$ nano .ipython/ipy_user_conf.py
ip.ex('import matplotlib; matplotlib.use("Agg"); import scipy, pylab,
scipy.signal as sig, scipy.linalg as lin, scipy.sparse as spar,
os, sys, MySQLdb, boto; from scikits import audiolab')
import ipy_greedycompleter
import ipy_autoreload
App Engine does not support any of numpy, scipy, or matplotlib, alas.
If you know exactly what OS and CPU your host is using, you could make an identical installation for yourself, download and install the same version of Python they’re using, download the sources of packages you require and build them into .so (or .pyd, depending on the platform) files, and upload those — sounds like a real tour de force.
Any of the many, many sites that offer normal virtual hosting (a virtual machine, typically Linux, with modest HW resources, but root privileges for you, ssh shell access, and a gcc you can use in particular) will be much easier to work with — essentially, you’ll download and install the software you need just about the same way you’d do on your own Linux workstation!
1: Installing third party packages to hosted spaces
You can indeed install third party packages to your hosted space. If it’s a pure python package, all that’s needed is to unpack it to a directory and then add that directory to your PYTHONPATH environment variable or sys.path.
This can be tiring to do often, and won’t work easily for compiled modules. If you have shell access to your python host, the excellent virtualenv package allows you to do set up a private python environment with its own libraries.
To set up your virtualenv, you’ll do something like this at the shell:
$ virtualenv $HOME/my_python
$ $HOME/my_python/bin/easy_install numpy
You can keep running easy_install for anything else you want to install in your personal python environment.
Now, when you write your python scripts, you will want to use your private python interpreter, if that is possible:
#!/home/myuser/my_python/bin/python
import numpy
# script here
If your python env cannot be specified (such as if run by mod_wsgi), you will need to add it to the import path:
import sys
sys.path.insert(0, '/home/myuser/my_python/lib/python2.5/site-packages')
import numpy
2: Hosting sites with numpy
I can’t think of any hosting sites offhand which offer numpy pre-installed. However, Dreamhost/Bluehost for sharedhosts provide SSH access, and with shell access you can install numpy using the methods I described above. Any Virtual Private Server such as Linode/Slicehost will allow you to install whatever you desire, as well.
3: AppEngine
As mentioned above, AppEngine will not allow you to install C extensions (but pure python ones do work) so it’s unlikely numpy will work for you on there, since I suspect some of its features use C speedups.
I do not have privledges yet to comment but I can provide an “answer”.
3: AppEngine
Numpy is now available on Google App Engine:
https://code.google.com/p/googleappengine/issues/detail?id=190
However, matplotlib is still waiting:
http://code.google.com/p/googleappengine/issues/detail?id=482
Perhaps more people starring this issue will make it happen.
I will note that svgfig is an option as it is pure python:
http://code.google.com/p/svgfig/
Update:
Turns out matplotlib is now available:
https://developers.google.com/appengine/docs/python/tools/libraries27#matplotlib
2 What hosting sites exist (either free or paid) which have Numpy/Matplotlib installed
PythonAnywhere offers web hosting and a simple in-browser IDE; lots of Python packages (including NumPy and Matplotlib) are pre-installed. There’s a free plan that you can use to play around, and “Premium” and “Hosting” accounts with more features are $5 and $10/month respectively.
Full disclosure: I work there…