How to include third party Python libraries in Google App Engine?
Question:
How to add third party python libraries in Google App Engine, which are not provided by Google? I am trying to use BeautifulSoup in Google App Engine and unable to do so. But my question is for any library I want to use in Google App Engine.
Answers:
Just put Beautifulsoup in the root of your project and upload it all
You simply copy the folder containing the library you want to use into your app engine project.
Then when you deploy it’s uploaded with your application and is available for use.
EDIT: Jesse’s answer is how I now do this. So do that!
Actually I think this answer fits better here.
If you want to use 3rd party libraries that are not included in this list, then you’ll have to add them manually.
In order to include manually any other library you have to have them inside the directory where the app.yaml lives. So for example if you have the following structure:
hello
├── libs
│ └── bs4
├── hello.py
└── app.yaml
then in your hello.py you have to put these two lines in the beginning of the file:
import sys
sys.path.insert(0, 'libs')
After doing that you’ll be able to use any 3rd party library that you’re going to put in that libs directory.
For example:
from bs4 import BeautifulSoup
The way it worked here is:
import sys
# sys.path.insert(0, 'libs') #"Old" way, not working for me.
sys.path.append(os.path.join(os.path.dirname(__file__), "libs")) # This works!
Then import normally:
from bs4 import BeautifulSoup
Google has provided a documented way for included third-party libraries in your GAE project.
See the “Adding Third-party Packages to the Application” section of the Libraries in Python 2.7 docs.
If you want to include additional pure-python third-party packages, you can do so by setting up vendoring. Vendoring allows you to install packages to a subdirectory of your project and include them in your code. To use vendoring, create (or modify) appengine_config.py in the root of your project.
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
And then just put all your libs’ source code in your lib dir
> pip install beautifulsoup4 -t lib
So your project directory structure looks like this:
project
- lib
- bs4
- your_code.py
This will allow your project’s source files to import libs’ packages/modules as if they were added to your PYTHON_PATH. For example:
# file: your_code.py
import bs4 # no need for 'from lib import bs4'
# do stuff with bs4...
You can also easily install everything from a requirements.txt file by doing the following command
> pip install -t lib -r requirements.txt
pip install -t lib package_name
lib: the location for third party libraries
Then you are good to use this package like a normal library you use from ipython or terminal.
How to add third party python libraries in Google App Engine, which are not provided by Google? I am trying to use BeautifulSoup in Google App Engine and unable to do so. But my question is for any library I want to use in Google App Engine.
Just put Beautifulsoup in the root of your project and upload it all
You simply copy the folder containing the library you want to use into your app engine project.
Then when you deploy it’s uploaded with your application and is available for use.
EDIT: Jesse’s answer is how I now do this. So do that!
Actually I think this answer fits better here.
If you want to use 3rd party libraries that are not included in this list, then you’ll have to add them manually.
In order to include manually any other library you have to have them inside the directory where the app.yaml lives. So for example if you have the following structure:
hello
├── libs
│ └── bs4
├── hello.py
└── app.yaml
then in your hello.py you have to put these two lines in the beginning of the file:
import sys
sys.path.insert(0, 'libs')
After doing that you’ll be able to use any 3rd party library that you’re going to put in that libs directory.
For example:
from bs4 import BeautifulSoup
The way it worked here is:
import sys
# sys.path.insert(0, 'libs') #"Old" way, not working for me.
sys.path.append(os.path.join(os.path.dirname(__file__), "libs")) # This works!
Then import normally:
from bs4 import BeautifulSoup
Google has provided a documented way for included third-party libraries in your GAE project.
See the “Adding Third-party Packages to the Application” section of the Libraries in Python 2.7 docs.
If you want to include additional pure-python third-party packages, you can do so by setting up vendoring. Vendoring allows you to install packages to a subdirectory of your project and include them in your code. To use vendoring, create (or modify) appengine_config.py in the root of your project.
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
And then just put all your libs’ source code in your lib dir
> pip install beautifulsoup4 -t lib
So your project directory structure looks like this:
project
- lib
- bs4
- your_code.py
This will allow your project’s source files to import libs’ packages/modules as if they were added to your PYTHON_PATH. For example:
# file: your_code.py
import bs4 # no need for 'from lib import bs4'
# do stuff with bs4...
You can also easily install everything from a requirements.txt file by doing the following command
> pip install -t lib -r requirements.txt
pip install -t lib package_name
lib: the location for third party libraries
Then you are good to use this package like a normal library you use from ipython or terminal.