How can you bundle all your python code into a single zip file?

Question:

It would be convenient when distributing applications to combine all of the eggs into a single zip file so that all you need to distribute is a single zip file and an executable (some custom binary that simply starts, loads the zip file’s main function and kicks python off or similar).

I’ve seen some talk of doing this online, but no examples of how to actually do it.

I’m aware that you can (if it is zip safe) convert eggs into zip files.

What I’m not sure about is:

Can you somehow combine all your eggs into a single zip file? If so, how?

How would you load and run code from a specific egg?

How would you ensure that the code in that egg could access all the dependencies (ie. other eggs in the zip file)?

People ask this sort of stuff a lot and get answers like; use py2exe. Yes, I get it, that’s one solution. It’s not the question I’m asking here though…

Asked By: Doug

||

Answers:

You could use a self-extracting zip file, set up to launch a Python interpreter after decompressing the eggs from within the very same .exe file that contains them.

Answered By: John Zwinck

Yes, one zip-file/egg can provide multiple modules, so you can combine them into one file. I’m however highly skeptical to that being a good idea. You still need to install that zip-file, and it may still clash with other already installed versions, etc.

So the first question to ask is what the aim is. Why do you want just one file? Is it for ease of install, or ease of distribution, or what?

Having just one file will not really make the install easier, there are other, better ways. You can let the install download and install the dependencies automatically, that’s easy to do.

And having them in one zip-file still means you need to expand that zip-file and run setup.py, which isn’t very userfriendly.

So having just one file doesn’t really solve many problems, so the question is which problem you are trying to solve.

Answered By: Lennart Regebro

Well, it’s possible to create your own “packages/eggs” in your {app-home-dir/packages} (by coping eggs there for example) and configure extra files in setup.py (setuptools) to pack it all as single distribution (What is setup.py?). Note, that before you launch your app main function you need to inform Python where exactly your external “packages/eggs” are – by adding {app-home-dir/packages} to sys.path. That is the easies way to create standalone package ..however with that goes dangers regarding dependencies and their versions, Python modules mixed with Ansi C code, etc.

Answered By: soerium

Can you somehow combine all your eggs into a single zip file? If so, how?

Yes you can. Python will load from zip archive that are added in sys.path (see PEP 273). If you put all python libraries inside an archive, the archive is treated as a directory. This is what some of the py2exe, bbfreeze, etc. tools can do to isolate the libraries.

As for the how, it really depends on how your eggs are installed: pip, easy_install, etc. The logic would be to inspect all you dependent eggs and gather their install path and then zip the eggs inside an archive.

How would you load and run code from a specific egg?

You need to define load and run. If you are talking about importing module and packages, you do not have to do anything
special. Here is an interesting blog post on the subject including some caveat Packaging Python programs as runnable ZIP files

How would you ensure that the code in that egg could access all the dependencies (ie. other eggs in the zip file)?

This is built-in as long as the eggs are not extensions (ie zip safe). See also zipimport

Answered By: Rod

You can automate most of the work with regular python tools. Let’s start with clean virtualenv.

[zart@feena ~]$ mkdir ziplib-demo
[zart@feena ~]$ cd ziplib-demo
[zart@feena ziplib-demo]$ virtualenv .
New python executable in ./bin/python
Installing setuptools.............done.
Installing pip...............done.

Now let’s install set of packages that will go into zipped library. The trick is to force installing them into specific directory.

(Note: don’t use –egg option either on command-line or in pip.conf/pip.ini because it will break file layout making it non-importable in zip)

[zart@feena ziplib-demo]$ bin/pip install --install-option --install-lib=$PWD/unpacked waitress
Downloading/unpacking waitress
  Downloading waitress-0.8.5.tar.gz (112kB): 112kB downloaded
  Running setup.py egg_info for package waitress

Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg (from waitress)
Installing collected packages: waitress
  Running setup.py install for waitress

    Installing waitress-serve script to /home/zart/ziplib-demo/bin
Successfully installed waitress
Cleaning up...

Update: pip now has -t <path> switch, that does the same thing as --install-option --install-lib=.

Now let’s pack all of them into one zip

[zart@feena ziplib-demo]$ cd unpacked
[zart@feena unpacked]$ ls
waitress  waitress-0.8.5-py2.7.egg-info
[zart@feena unpacked]$ zip -r9 ../library.zip *
  adding: waitress/ (stored 0%)
  adding: waitress/receiver.py (deflated 71%)
  adding: waitress/server.pyc (deflated 64%)
  adding: waitress/utilities.py (deflated 62%)
  adding: waitress/trigger.pyc (deflated 63%)
  adding: waitress/trigger.py (deflated 61%)
  adding: waitress/receiver.pyc (deflated 60%)
  adding: waitress/adjustments.pyc (deflated 51%)
  adding: waitress/compat.pyc (deflated 56%)
  adding: waitress/adjustments.py (deflated 60%)
  adding: waitress/server.py (deflated 68%)
  adding: waitress/channel.py (deflated 72%)
  adding: waitress/task.pyc (deflated 57%)
  adding: waitress/tests/ (stored 0%)
  adding: waitress/tests/test_regression.py (deflated 63%)
  adding: waitress/tests/test_functional.py (deflated 88%)
  adding: waitress/tests/test_parser.pyc (deflated 76%)
  adding: waitress/tests/test_trigger.pyc (deflated 73%)
  adding: waitress/tests/test_init.py (deflated 72%)
  adding: waitress/tests/test_utilities.pyc (deflated 78%)
  adding: waitress/tests/test_buffers.pyc (deflated 79%)
  adding: waitress/tests/test_trigger.py (deflated 82%)
  adding: waitress/tests/test_buffers.py (deflated 86%)
  adding: waitress/tests/test_runner.py (deflated 75%)
  adding: waitress/tests/test_init.pyc (deflated 69%)
  adding: waitress/tests/__init__.pyc (deflated 21%)
  adding: waitress/tests/support.pyc (deflated 48%)
  adding: waitress/tests/test_utilities.py (deflated 73%)
  adding: waitress/tests/test_channel.py (deflated 87%)
  adding: waitress/tests/test_task.py (deflated 87%)
  adding: waitress/tests/test_functional.pyc (deflated 82%)
  adding: waitress/tests/__init__.py (deflated 5%)
  adding: waitress/tests/test_compat.pyc (deflated 53%)
  adding: waitress/tests/test_receiver.pyc (deflated 79%)
  adding: waitress/tests/test_adjustments.py (deflated 78%)
  adding: waitress/tests/test_adjustments.pyc (deflated 74%)
  adding: waitress/tests/test_server.pyc (deflated 73%)
  adding: waitress/tests/fixtureapps/ (stored 0%)
  adding: waitress/tests/fixtureapps/filewrapper.pyc (deflated 59%)
  adding: waitress/tests/fixtureapps/getline.py (deflated 37%)
  adding: waitress/tests/fixtureapps/nocl.py (deflated 47%)
  adding: waitress/tests/fixtureapps/sleepy.pyc (deflated 44%)
  adding: waitress/tests/fixtureapps/echo.py (deflated 40%)
  adding: waitress/tests/fixtureapps/error.py (deflated 52%)
  adding: waitress/tests/fixtureapps/nocl.pyc (deflated 48%)
  adding: waitress/tests/fixtureapps/getline.pyc (deflated 32%)
  adding: waitress/tests/fixtureapps/writecb.pyc (deflated 42%)
  adding: waitress/tests/fixtureapps/toolarge.py (deflated 37%)
  adding: waitress/tests/fixtureapps/__init__.pyc (deflated 20%)
  adding: waitress/tests/fixtureapps/writecb.py (deflated 50%)
  adding: waitress/tests/fixtureapps/badcl.pyc (deflated 44%)
  adding: waitress/tests/fixtureapps/runner.pyc (deflated 58%)
  adding: waitress/tests/fixtureapps/__init__.py (stored 0%)
  adding: waitress/tests/fixtureapps/filewrapper.py (deflated 74%)
  adding: waitress/tests/fixtureapps/runner.py (deflated 41%)
  adding: waitress/tests/fixtureapps/echo.pyc (deflated 42%)
  adding: waitress/tests/fixtureapps/groundhog1.jpg (deflated 24%)
  adding: waitress/tests/fixtureapps/error.pyc (deflated 48%)
  adding: waitress/tests/fixtureapps/sleepy.py (deflated 42%)
  adding: waitress/tests/fixtureapps/toolarge.pyc (deflated 43%)
  adding: waitress/tests/fixtureapps/badcl.py (deflated 45%)
  adding: waitress/tests/support.py (deflated 52%)
  adding: waitress/tests/test_task.pyc (deflated 78%)
  adding: waitress/tests/test_channel.pyc (deflated 78%)
  adding: waitress/tests/test_regression.pyc (deflated 68%)
  adding: waitress/tests/test_parser.py (deflated 80%)
  adding: waitress/tests/test_server.py (deflated 78%)
  adding: waitress/tests/test_receiver.py (deflated 87%)
  adding: waitress/tests/test_compat.py (deflated 51%)
  adding: waitress/tests/test_runner.pyc (deflated 72%)
  adding: waitress/__init__.pyc (deflated 50%)
  adding: waitress/channel.pyc (deflated 58%)
  adding: waitress/runner.pyc (deflated 54%)
  adding: waitress/buffers.py (deflated 74%)
  adding: waitress/__init__.py (deflated 61%)
  adding: waitress/runner.py (deflated 58%)
  adding: waitress/parser.py (deflated 69%)
  adding: waitress/compat.py (deflated 69%)
  adding: waitress/buffers.pyc (deflated 69%)
  adding: waitress/utilities.pyc (deflated 60%)
  adding: waitress/parser.pyc (deflated 53%)
  adding: waitress/task.py (deflated 72%)
  adding: waitress-0.8.5-py2.7.egg-info/ (stored 0%)
  adding: waitress-0.8.5-py2.7.egg-info/dependency_links.txt (stored 0%)
  adding: waitress-0.8.5-py2.7.egg-info/installed-files.txt (deflated 83%)
  adding: waitress-0.8.5-py2.7.egg-info/top_level.txt (stored 0%)
  adding: waitress-0.8.5-py2.7.egg-info/PKG-INFO (deflated 65%)
  adding: waitress-0.8.5-py2.7.egg-info/not-zip-safe (stored 0%)
  adding: waitress-0.8.5-py2.7.egg-info/SOURCES.txt (deflated 71%)
  adding: waitress-0.8.5-py2.7.egg-info/entry_points.txt (deflated 33%)
  adding: waitress-0.8.5-py2.7.egg-info/requires.txt (deflated 5%)
[zart@feena unpacked]$ cd ..

Note that those files should be at top of zip, you can’t just zip -r9 library.zip unpacked

Checking the result:

[zart@feena ziplib-demo]$ PYTHONPATH=library.zip python
Python 2.7.1 (r271:86832, Apr 12 2011, 16:15:16)
[GCC 4.6.0 20110331 (Red Hat 4.6.0-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import waitress
>>> waitress
<module 'waitress' from '/home/zart/ziplib-demo/library.zip/waitress/__init__.pyc'>
>>>
>>> from wsgiref.simple_server import demo_app
>>> waitress.serve(demo_app)
serving on http://0.0.0.0:8080
^C>>>

Update: since python 3.5 there is also zipapp module which can help with bundling the whole package into .pyz file. For more complex needs pyinstaller, py2exe or py2app might better fit the bill.

Answered By: Zart

Python will execute zip files as if they were single scripts if they contain an __main__.py[c] file inside at the top level. Package imports will then also check inside the zip that __main__ is executing from within.

So create your setup.py (py_modules = ['__main__'] is important here along with specifying all your packages and other modules).

Then run python setup.py bdist --format zip to create the zip file. Now if you want it to be executable you can do the following. At this point you can execute the resulting zip file like any other python script.

One more step for Linux/Mac users reading this to improve convenience (although probably not your scenario as you mention py2exe)

echo '#!/usr/bin/env python' > my_executable_zip
cat output_of_setup_py_bdist.zip >> my_executable_zip
chmod +x my_executable_zip

This just prepends a #! line to the zip file so that when run from the shell you do not need to specify the interpreter. At this point you can execute it like any other binary on the system although secretly it is a zip file full of python. I typically create a makefile to run setup.py and then do this conversion.

Answered By: Matt

You could use the zipapp module from the standard library to create executable Python zip archives. It is available from Python 3.5 onwards.

One way to create a bundle is to add a top-level file named __main__.py, which will be the script that Python runs when the zip executable archive is executed.

Suppose your directory structure is now like this:

└── myapp
    ├── __main__.py
    ├── myprog1.py
    └── myprog2.py

If your code has external dependencies (e.g. listed in a file named requirements.txt), install them into the directory using:

pip3 install -r requirements.txt --target myapp/

note 1: This will fill the myapp/ directory with the external dependencies.

note 2: Debian/Ubuntu users may need to use the --system option for pip3, because the Debian/Ubuntu version of pip seems to use --user by default.

Then, create the zip executable archive using:

python3 -m zipapp myapp/

This will create a zip executable archive named myapp.pyz, which you can execute by running:

python3 myapp.pyz

When the zip executable archive is executed, it is __main__.py that is run.

If, in addition to Python scripts, you need to include other data files (e.g. text files, PNG images, etc.) used by the Python scripts, see: python: can executable zip files include data files?

Answered By: Flux
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.