How to roll my own pypi?
Question:
I would like to run my own internal pypi server, for egg distribution within my organization.
I have found a few projects, such as:
As I understand it, pypi.python.org uses software called Cheese Shop.
My questions:
- Why can’t I use cheeseshop itself? (I can’t find it, not sure it exists)
- How do other people solve this problem? (Currently we use blush svn to distribute eggs)
*edit: This seems canonical http://wiki.python.org/moin/PyPiImplementations. Still, I’m interested in feedback.
Answers:
Update: PyPi is now powered by Warehouse, which is the replacement for Cheese Shop.
The source to Cheese Shop can be downloaded from https://bitbucket.org/pypa/pypi/src. There is also an example, from the page you linked to, of using Apache as a “dumb” Python package repository:
# Mount pypi repositories into URI space
Alias /pypi /var/pypi
# /pypi/dev: Redirect for unknown packages (fallback to pypi)
RewriteCond /var/pypi/dev/$1 !-d
RewriteCond /var/pypi/dev/$1 !-f
RewriteRule ^/pypi/dev/([^/]+)/?$ http://pypi.python.org/pypi/$1/ [R,L]
RewriteCond /var/pypi/dev/$1/$2 !-f
RewriteRule ^/pypi/dev/([^/]+)/([^/]+)$ http://pypi.python.org/pypi/$1/$2 [R,L]
# /pypi/stable: Redirect for unknown packages (fallback to pypi)
RewriteCond /var/pypi/stable/$1 !-d
RewriteCond /var/pypi/stable/$1 !-f
RewriteRule ^/pypi/stable/([^/]+)/?$ http://pypi.python.org/pypi/$1/ [R,L]
RewriteCond /var/pypi/stable/$1/$2 !-f
RewriteRule ^/pypi/stable/([^/]+)/([^/]+)$ http://pypi.python.org/pypi/$1/$2 [R,L]
If you would like a lighter solution then deploying an entire pypi server, you could try using a server index generated by basketweaver.
Updated: crate.io has shut down and the domain is now something else entirely.
One project that hasn’t been mentioned is https://crate.io/, which seems very active. It claims to be a “Next Generation Python Packaging Index”, but they have their repositories split nicely into pieces that seem to welcome customization and remixing to your purposes.
For light-weight solution, use pypiserver.
Another missing from this (oldish) list:
djangopypi
Django based, which might be a slight overkill, but I love django and it makes it extremely simple to modify it to your need should it not be satisfying.
And crate source code is available, though documentation is, least that can be said, not-existent:
It’s a Django Application providing a Python Package Index. Uses a couple other packages from https://github.com/crateio so you might be able to roll out your own version without django.
I’m specifically thinking about a static one, I always thought there should be a very easy way to go explore directly some [pre-configured] repositories and shop cheese directly from my github/bitbucket public and private repos, with just a simple (gunicorn) process running.
There is a fork of djangopypi named djangopypi2 you can get it from https://github.com/popen2/djangopypi2/, I installed it and works for me, this option is what I had choose from a list of about 24 alternatives that I have found in a recently search, you can see the list here: http://uyeya.blogspot.com/2013/10/list-of-local-alternatives-of-pypi.html
I ran into the same problem, and ClueReleaseManager solved the problem of hosting an internal PyPI server/custom Python repository for me.
That answers question two, and I guess you already have the answer to question one.
Warehouse
Warehouse would be your best bet in 2017. From the project’s README:
Warehouse is a next generation Python Package Repository designed to replace
the legacy code base that currently powers PyPI
…
You can run Warehouse locally using docker and docker-compose. See
Getting started
in the documentation for instructions on how to set it up.
It is maintained by The Python Packaging Authority (PyPA) who work in cooperation with members of the Python core development team, and there is a live version running at https://pypi.org/ which mirrors everything in the legacy PyPI (https://pypi.python.org/).
devpi
We are using it in a corporate environment and are pretty satisfied. It supports replication, private indexes and index inheritance.
I would like to run my own internal pypi server, for egg distribution within my organization.
I have found a few projects, such as:
As I understand it, pypi.python.org uses software called Cheese Shop.
My questions:
- Why can’t I use cheeseshop itself? (I can’t find it, not sure it exists)
- How do other people solve this problem? (Currently we use blush svn to distribute eggs)
*edit: This seems canonical http://wiki.python.org/moin/PyPiImplementations. Still, I’m interested in feedback.
Update: PyPi is now powered by Warehouse, which is the replacement for Cheese Shop.
The source to Cheese Shop can be downloaded from https://bitbucket.org/pypa/pypi/src. There is also an example, from the page you linked to, of using Apache as a “dumb” Python package repository:
# Mount pypi repositories into URI space
Alias /pypi /var/pypi
# /pypi/dev: Redirect for unknown packages (fallback to pypi)
RewriteCond /var/pypi/dev/$1 !-d
RewriteCond /var/pypi/dev/$1 !-f
RewriteRule ^/pypi/dev/([^/]+)/?$ http://pypi.python.org/pypi/$1/ [R,L]
RewriteCond /var/pypi/dev/$1/$2 !-f
RewriteRule ^/pypi/dev/([^/]+)/([^/]+)$ http://pypi.python.org/pypi/$1/$2 [R,L]
# /pypi/stable: Redirect for unknown packages (fallback to pypi)
RewriteCond /var/pypi/stable/$1 !-d
RewriteCond /var/pypi/stable/$1 !-f
RewriteRule ^/pypi/stable/([^/]+)/?$ http://pypi.python.org/pypi/$1/ [R,L]
RewriteCond /var/pypi/stable/$1/$2 !-f
RewriteRule ^/pypi/stable/([^/]+)/([^/]+)$ http://pypi.python.org/pypi/$1/$2 [R,L]
If you would like a lighter solution then deploying an entire pypi server, you could try using a server index generated by basketweaver.
Updated: crate.io has shut down and the domain is now something else entirely.
One project that hasn’t been mentioned is https://crate.io/, which seems very active. It claims to be a “Next Generation Python Packaging Index”, but they have their repositories split nicely into pieces that seem to welcome customization and remixing to your purposes.
For light-weight solution, use pypiserver.
Another missing from this (oldish) list:
djangopypi
Django based, which might be a slight overkill, but I love django and it makes it extremely simple to modify it to your need should it not be satisfying.
And crate source code is available, though documentation is, least that can be said, not-existent:
It’s a Django Application providing a Python Package Index. Uses a couple other packages from https://github.com/crateio so you might be able to roll out your own version without django.
I’m specifically thinking about a static one, I always thought there should be a very easy way to go explore directly some [pre-configured] repositories and shop cheese directly from my github/bitbucket public and private repos, with just a simple (gunicorn) process running.
There is a fork of djangopypi named djangopypi2 you can get it from https://github.com/popen2/djangopypi2/, I installed it and works for me, this option is what I had choose from a list of about 24 alternatives that I have found in a recently search, you can see the list here: http://uyeya.blogspot.com/2013/10/list-of-local-alternatives-of-pypi.html
I ran into the same problem, and ClueReleaseManager solved the problem of hosting an internal PyPI server/custom Python repository for me.
That answers question two, and I guess you already have the answer to question one.
Warehouse
Warehouse would be your best bet in 2017. From the project’s README:
Warehouse is a next generation Python Package Repository designed to replace
the legacy code base that currently powers PyPI
…
You can run Warehouse locally using
dockeranddocker-compose. See
Getting started
in the documentation for instructions on how to set it up.
It is maintained by The Python Packaging Authority (PyPA) who work in cooperation with members of the Python core development team, and there is a live version running at https://pypi.org/ which mirrors everything in the legacy PyPI (https://pypi.python.org/).
devpi
We are using it in a corporate environment and are pretty satisfied. It supports replication, private indexes and index inheritance.