How to use virtualenv with Google App Engine SDK on Mac OS X 10.6
Question:
I am pulling my hair out trying to figure this out because I had it working until last week and somehow it broke.
When I setup a virtualenv for a Google App Engine app and start the app with dev_appserver.py, I get errors importing the standard library (like “ImportError: No module named base64”).
Here’s what I’m doing:
(Using the system Python)
virtualenv --python=python2.5 --no-site-packages ~/.virtualenv/foobar
Then I add the a gae.pth file to ~/.virtualenv/foobar/lib/python2.5/site-packages/ containing the Google App Engine libraries:
/usr/local/google_appengine
/usr/local/google_appengine/lib/antlr3
/usr/local/google_appengine/lib/cacerts
/usr/local/google_appengine/lib/django
/usr/local/google_appengine/lib/fancy_urllib
/usr/local/google_appengine/lib/ipaddr
/usr/local/google_appengine/lib/webob_1_1_1
/usr/local/google_appengine/lib/yaml/lib
(That’s based on this answer.)
Then I source my “foobar” virtualenv and try to start my app with dev_appserver.py.
The server starts but the first request errors out with the aforementioned “ImportError: No module named base64”. If I visit the admin console I get “ImportError: No module named cgi”.
If I start up python, I can load these modules.
>>> import base64
>>> base64.__file__
'/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/base64.py'
It seems that the SDK’s sandboxing is preventing these libraries from getting loaded. But like I said, I had this working until last week…something changed or I inadvertently broke my virtualenv and I can’t figure out how I got it working in the first place.
Software versions:
Google App Engine SDK 1.3.7
Mac OS X Snow Leopard 10.6.4
virtualenv 1.5.1
Update: In response to Alan Franzoni’s questions:
I am using the system Python that came with Mac OS X. I installed virtualenv via easy_install. I upgraded to virtualenv 1.5.1 today to try to fix the problem.
If I run python /usr/local/bin/dev_appserver.py with the virtualenv python, the problem persists. If I deactivate the virtualenv and run that command with the system python2.5, it works. (Also, I can use the GoogleAppEngineLauncher to start my app.)
Here is a full stack trace (this one uses the Kay framework, but the problem is the same with webapp):
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 3206, in _HandleRequest
self._Dispatch(dispatcher, self.rfile, outfile, env_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 3149, in _Dispatch
base_env_dict=env_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 525, in Dispatch
base_env_dict=base_env_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2402, in Dispatch
self._module_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2312, in ExecuteCGI
reset_modules = exec_script(handler_path, cgi_path, hook)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2208, in ExecuteOrImportScript
exec module_code in script_module.__dict__
File "/Users/look/myapp/kay/main.py", line 17, in <module>
kay.setup()
File "/Users/look/myapp/kay/__init__.py", line 122, in setup
from google.appengine.ext import db
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate
return func(self, *args, **kwargs)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1937, in load_module
return self.FindAndLoadModule(submodule, fullname, search_path)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate
return func(self, *args, **kwargs)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1839, in FindAndLoadModule
description)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate
return func(self, *args, **kwargs)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1790, in LoadModuleRestricted
description)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/db/__init__.py", line 81, in <module>
import base64
ImportError: No module named base64
Answers:
I think, since you’ve setup virtualenv with the –no-site-packages option, you need to install the SDK into the environment. –no-site-packages seperates the dev environment you’re configuring from any other Python installation on your computer so, as you seem to have it configured you’re calling a module that doesn’t exist (in the environment) which is why it works with the env deactivated (which is then running Python from the default OS installation). Try setting up the dev env without the –no-site-packages option if you want to be able to access modules outside of the env.
Google AppEngine SDK makes a lot of trick in order to pull its install into sys.path, and these tricks rely on actual file’s path. I think there might be a lot of various reasons why it fails. SDK doesn’t install itself as a real python package, virtualenv doesn’t do complete sandboxing, it just sets up an environment (obviously) and changes sys.path. And GAE SDK does this too, they both intefere, SDK is being developing rapidly and changing often, so this is extremely bumpy road to go.
Probably, it would be better if you’d explain what are you trying to achieve. My guess is that you’re trying to create a clean environment to ensure that no 3rd-party module is available to application. If this guess is correct, I’d go with installing GAE SDK into virtualenv via requirements files as described here.
It’s an issue 4339 with the GAE SDK, it’s confirmed and there are two slightly different patches available in the bug entry that make it work.
What happens is dev_appserver.py sets up a restricted python environment by disallowing access to any non-system-python modules and it does that by calculating the system python folder from the location of the os module. In a virtualenv instance the os.py gets symlinked into the virtualenv but gets compiled straight into virtualenv, and this is the path that dev_appserver uses, effectively blocking access to any module from the system python library that is not linked by virtualend, which is most of them. The solution is to “bless” both paths.
Same answer as bozzo. Here’s instructions:
This is described in Issue 4339 for GAE. Here’s how to fix it:
- Download patch here: patch
- Move the patch to google_appengine/google/appengine/tools/
- Change your working directory to the same path as above
- Type: patch -p0 < dev_appserver.patch
I’m a little late to the conversation, but I was just having the same issue and I stumbled across gae_installer, which you can install in the usual way with pip install gae_installer. This will put the google app engine (gae) sdk directly into your python path. Hope others find this useful.
I am pulling my hair out trying to figure this out because I had it working until last week and somehow it broke.
When I setup a virtualenv for a Google App Engine app and start the app with dev_appserver.py, I get errors importing the standard library (like “ImportError: No module named base64”).
Here’s what I’m doing:
(Using the system Python)
virtualenv --python=python2.5 --no-site-packages ~/.virtualenv/foobar
Then I add the a gae.pth file to ~/.virtualenv/foobar/lib/python2.5/site-packages/ containing the Google App Engine libraries:
/usr/local/google_appengine
/usr/local/google_appengine/lib/antlr3
/usr/local/google_appengine/lib/cacerts
/usr/local/google_appengine/lib/django
/usr/local/google_appengine/lib/fancy_urllib
/usr/local/google_appengine/lib/ipaddr
/usr/local/google_appengine/lib/webob_1_1_1
/usr/local/google_appengine/lib/yaml/lib
(That’s based on this answer.)
Then I source my “foobar” virtualenv and try to start my app with dev_appserver.py.
The server starts but the first request errors out with the aforementioned “ImportError: No module named base64”. If I visit the admin console I get “ImportError: No module named cgi”.
If I start up python, I can load these modules.
>>> import base64
>>> base64.__file__
'/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/base64.py'
It seems that the SDK’s sandboxing is preventing these libraries from getting loaded. But like I said, I had this working until last week…something changed or I inadvertently broke my virtualenv and I can’t figure out how I got it working in the first place.
Software versions:
Google App Engine SDK 1.3.7
Mac OS X Snow Leopard 10.6.4
virtualenv 1.5.1
Update: In response to Alan Franzoni’s questions:
I am using the system Python that came with Mac OS X. I installed virtualenv via easy_install. I upgraded to virtualenv 1.5.1 today to try to fix the problem.
If I run python /usr/local/bin/dev_appserver.py with the virtualenv python, the problem persists. If I deactivate the virtualenv and run that command with the system python2.5, it works. (Also, I can use the GoogleAppEngineLauncher to start my app.)
Here is a full stack trace (this one uses the Kay framework, but the problem is the same with webapp):
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 3206, in _HandleRequest
self._Dispatch(dispatcher, self.rfile, outfile, env_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 3149, in _Dispatch
base_env_dict=env_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 525, in Dispatch
base_env_dict=base_env_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2402, in Dispatch
self._module_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2312, in ExecuteCGI
reset_modules = exec_script(handler_path, cgi_path, hook)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2208, in ExecuteOrImportScript
exec module_code in script_module.__dict__
File "/Users/look/myapp/kay/main.py", line 17, in <module>
kay.setup()
File "/Users/look/myapp/kay/__init__.py", line 122, in setup
from google.appengine.ext import db
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate
return func(self, *args, **kwargs)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1937, in load_module
return self.FindAndLoadModule(submodule, fullname, search_path)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate
return func(self, *args, **kwargs)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1839, in FindAndLoadModule
description)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate
return func(self, *args, **kwargs)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1790, in LoadModuleRestricted
description)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/db/__init__.py", line 81, in <module>
import base64
ImportError: No module named base64
I think, since you’ve setup virtualenv with the –no-site-packages option, you need to install the SDK into the environment. –no-site-packages seperates the dev environment you’re configuring from any other Python installation on your computer so, as you seem to have it configured you’re calling a module that doesn’t exist (in the environment) which is why it works with the env deactivated (which is then running Python from the default OS installation). Try setting up the dev env without the –no-site-packages option if you want to be able to access modules outside of the env.
Google AppEngine SDK makes a lot of trick in order to pull its install into sys.path, and these tricks rely on actual file’s path. I think there might be a lot of various reasons why it fails. SDK doesn’t install itself as a real python package, virtualenv doesn’t do complete sandboxing, it just sets up an environment (obviously) and changes sys.path. And GAE SDK does this too, they both intefere, SDK is being developing rapidly and changing often, so this is extremely bumpy road to go.
Probably, it would be better if you’d explain what are you trying to achieve. My guess is that you’re trying to create a clean environment to ensure that no 3rd-party module is available to application. If this guess is correct, I’d go with installing GAE SDK into virtualenv via requirements files as described here.
It’s an issue 4339 with the GAE SDK, it’s confirmed and there are two slightly different patches available in the bug entry that make it work.
What happens is dev_appserver.py sets up a restricted python environment by disallowing access to any non-system-python modules and it does that by calculating the system python folder from the location of the os module. In a virtualenv instance the os.py gets symlinked into the virtualenv but gets compiled straight into virtualenv, and this is the path that dev_appserver uses, effectively blocking access to any module from the system python library that is not linked by virtualend, which is most of them. The solution is to “bless” both paths.
Same answer as bozzo. Here’s instructions:
This is described in Issue 4339 for GAE. Here’s how to fix it:
- Download patch here: patch
- Move the patch to google_appengine/google/appengine/tools/
- Change your working directory to the same path as above
- Type: patch -p0 < dev_appserver.patch
I’m a little late to the conversation, but I was just having the same issue and I stumbled across gae_installer, which you can install in the usual way with pip install gae_installer. This will put the google app engine (gae) sdk directly into your python path. Hope others find this useful.