What is the Python egg cache (PYTHON_EGG_CACHE)?
Question:
I’ve just upgraded from Python 2.6.1 to 2.6.4 on my development machine and upon starting a python script was presented with the following message:
Can’t extract file(s) to egg cache
The following error occurred while
trying to extract file(s) to the
Python egg cache:
[Errno 13] Permission denied:
‘/var/www/.python-eggs’
The Python egg cache directory is
currently set to:
/var/www/.python-eggs
Perhaps your account does not have
write access to this directory? You
can change the cache directory by
setting the PYTHON_EGG_CACHE
environment variable to point to an
accessible directory.
There isn’t anything in the python docs so I’m at a bit of a loss regarding best-practices on where to put this directory and what it’s used for.
Can someone explain what the Python egg cache is?
Also, can you explain why/how it is different to the site-packages directory Python uses to store eggs (as I understand it)?
Answers:
The python egg cache is simply a directory used by setuptools to store packages installed that conform to the egg specification. You can read more about setuptools here.
Additionally, as the error message states, you can specify a different egg cache directory in your environment by setting PYTHON_EGG_CACHE=/some/other/dir. The easiest way to do this is to set it in your ~/.bash_profile (assuming you’re using bash), like this:
export PYTHON_EGG_CACHE=/some/other/dir
You may need to set it in your Apache environment if you’re using a Web application.
Python eggs are zip-compressed packages containing both Python modules and metadata. The egg cache is where the extracted contents of the egg are stored so that the Python modules contained within are usable.
From my investigations it turns out that some eggs are packaged as zip files, and are saved as such in Python’s site-packages directory.
These zipped eggs need to be unzipped before they can be executed, so are expanded into the PYTHON_EGG_CACHE directory which by default is ~/.python-eggs (located in the user’s home directory). If this doesn’t exist it causes problems when trying to run applications.
There are a number of fixes:
- Create a
.python-eggs directory in the user’s home directory and make it writable for the user.
- Create a global directory for unzipping (eg.
/tmp/python-eggs) and set the environment variable PYTHON_EGG_CACHE to this directory.
- Use the
-Z switch when using easy_install to unzip the package when installing.
You can also disable the use of the .egg after it has been installed. You need to go into the site-packages directory, extract the .egg, and then move it to a hidden file (or delete it, or whatever).
Here is an example of what I did to disable the MySQLdb module .egg file which was causing this error when the python script was being run from Zabbix.
cd /usr/local/lib/python2.7/site-packages
unzip MySQL_python-1.2.3-py2.7-linux-x86_64.egg
mv MySQL_python-1.2.3-py2.7-linux-x86_64.egg .MySQL_python-1.2.3-py2.7-linux-x86_64.egg
This is a dark side-effect of using otherwise nice eggs mechanism.
Eggs are packages (a directory full of files) packed into one .egg file to simplify depolyment.
They are stored in /site-packages/ dir.
As long as the files stored in the egg are .py files it works great. Python import can import things from any file-like object just like it was an ordinary file.
But when something like .so happens to drop in there, python cannot explain to the underlying OS that it wants to load an library which doesn’t have a physical name. And the only workaround distutils authors have thought of is unzipping it into a temp dir. Naturally it is not /site-packages/ since /site-packages/ is not writable for ordinary users.
So you can either
-
set PYTHON_EGG_DIR to /tmp, or
-
give user www write permission to /var/www/.python-eggs
(so that the files don’t get unzipped every time /tmp is cleaned up) or better then
-
unzip the egg as suggested by @shalley303
(and avoid unzipping of the egg in the run-time altogether).
Phillip B Oldham’s right. You can add these lines in your code:
import os
os.environ['PYTHON_EGG_CACHE'] = '/tmp' # a writable directory
A simple fix would be to create the directory and provide www-data access to it.
$ mkdir /var/www/.python-eggs
$ chown www-data:www-data /var/www/.python-eggs
I got this error in Django when running the below command the first time.
python manage.py sql myproject
I got it to work like this:
1. In Explorer, view the folder that the error says egg cache directory is set to
2. Delete (or rename) the file mysql_python-1.2.5-py2.7-win32.egg-tmp
3. That's it. The command now works and creates a new file in there. (Haven't tested if I need to do this every time.)
adding this at the begin of my source file before any import works
import os
xyz = os.path.join('~', 'Documents', '.cache')
os.environ['PYTHON_EGG_CACHE'] = os.path.expanduser(xyz)
I’ve just upgraded from Python 2.6.1 to 2.6.4 on my development machine and upon starting a python script was presented with the following message:
Can’t extract file(s) to egg cache
The following error occurred while
trying to extract file(s) to the
Python egg cache:[Errno 13] Permission denied:
‘/var/www/.python-eggs’The Python egg cache directory is
currently set to:/var/www/.python-eggs
Perhaps your account does not have
write access to this directory? You
can change the cache directory by
setting the PYTHON_EGG_CACHE
environment variable to point to an
accessible directory.
There isn’t anything in the python docs so I’m at a bit of a loss regarding best-practices on where to put this directory and what it’s used for.
Can someone explain what the Python egg cache is?
Also, can you explain why/how it is different to the site-packages directory Python uses to store eggs (as I understand it)?
The python egg cache is simply a directory used by setuptools to store packages installed that conform to the egg specification. You can read more about setuptools here.
Additionally, as the error message states, you can specify a different egg cache directory in your environment by setting PYTHON_EGG_CACHE=/some/other/dir. The easiest way to do this is to set it in your ~/.bash_profile (assuming you’re using bash), like this:
export PYTHON_EGG_CACHE=/some/other/dir
You may need to set it in your Apache environment if you’re using a Web application.
Python eggs are zip-compressed packages containing both Python modules and metadata. The egg cache is where the extracted contents of the egg are stored so that the Python modules contained within are usable.
From my investigations it turns out that some eggs are packaged as zip files, and are saved as such in Python’s site-packages directory.
These zipped eggs need to be unzipped before they can be executed, so are expanded into the PYTHON_EGG_CACHE directory which by default is ~/.python-eggs (located in the user’s home directory). If this doesn’t exist it causes problems when trying to run applications.
There are a number of fixes:
- Create a
.python-eggsdirectory in the user’s home directory and make it writable for the user. - Create a global directory for unzipping (eg.
/tmp/python-eggs) and set the environment variablePYTHON_EGG_CACHEto this directory. - Use the
-Zswitch when usingeasy_installto unzip the package when installing.
You can also disable the use of the .egg after it has been installed. You need to go into the site-packages directory, extract the .egg, and then move it to a hidden file (or delete it, or whatever).
Here is an example of what I did to disable the MySQLdb module .egg file which was causing this error when the python script was being run from Zabbix.
cd /usr/local/lib/python2.7/site-packages unzip MySQL_python-1.2.3-py2.7-linux-x86_64.egg mv MySQL_python-1.2.3-py2.7-linux-x86_64.egg .MySQL_python-1.2.3-py2.7-linux-x86_64.egg
This is a dark side-effect of using otherwise nice eggs mechanism.
Eggs are packages (a directory full of files) packed into one .egg file to simplify depolyment.
They are stored in /site-packages/ dir.
As long as the files stored in the egg are .py files it works great. Python import can import things from any file-like object just like it was an ordinary file.
But when something like .so happens to drop in there, python cannot explain to the underlying OS that it wants to load an library which doesn’t have a physical name. And the only workaround distutils authors have thought of is unzipping it into a temp dir. Naturally it is not /site-packages/ since /site-packages/ is not writable for ordinary users.
So you can either
-
set
PYTHON_EGG_DIRto/tmp, or -
give user
wwwwrite permission to/var/www/.python-eggs
(so that the files don’t get unzipped every time /tmp is cleaned up) or better then -
unzip the egg as suggested by @shalley303
(and avoid unzipping of the egg in the run-time altogether).
Phillip B Oldham’s right. You can add these lines in your code:
import os
os.environ['PYTHON_EGG_CACHE'] = '/tmp' # a writable directory
A simple fix would be to create the directory and provide www-data access to it.
$ mkdir /var/www/.python-eggs
$ chown www-data:www-data /var/www/.python-eggs
I got this error in Django when running the below command the first time.
python manage.py sql myproject
I got it to work like this:
1. In Explorer, view the folder that the error says egg cache directory is set to
2. Delete (or rename) the file mysql_python-1.2.5-py2.7-win32.egg-tmp
3. That's it. The command now works and creates a new file in there. (Haven't tested if I need to do this every time.)
adding this at the begin of my source file before any import works
import os
xyz = os.path.join('~', 'Documents', '.cache')
os.environ['PYTHON_EGG_CACHE'] = os.path.expanduser(xyz)