how to create local own pypi repository index without mirror?

Question

We have several own python packages and want to create local pypi repository for them using simple interface like https://pypi.python.org/simple/

This repository I want to create for local only without any mirrors due to security reason, and it will be put under apache’s control

The command pypimirror looks has to be initialized once, which needs to mirror.

How can I generate PyPi Simple Index based on local python packages.

Any other simple scripts for this ?

Asked By: Larry Cai

||

Source

Answer 1

If you are talking about running simplepypi then you will have your server for adding packages and serve them out.
To quote the documentation:

- Running this on the setup.py of your favorite package:

    python setup.py sdist upload -r local

If you were to use either os.walk or glob.glob on your local site-packages directory you could quickly filter for setup.py in each of the packages/directories and invoke the above on them.

If you just need to create a directory of tar.gz files complete with a .html list of them then you can use glob.glob on the top level of your site-packages directory – tar.gz each directory in turn and add the resulting filename to a list – you can then generate your index.html from that list.

You can use any of a large number of template engines for this or generate it yourself:

import glob
filelist = glob.glob("*.tar.gz")
tags = ['<A href="file:Where/%s">%s</A>' % (s,s) for s in tags]
head = """<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
<TITLE></TITLE>
<META NAME="Generator" CONTENT="Python Script">
<META NAME="Keywords" CONTENT="Cheeseshop">
<META NAME="Description" CONTENT="List of local python packages">
</HEAD>

<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#FF0000" VLINK="#800000" ALINK="#FF00FF" BACKGROUND="?">
"""
tail = """</BODY></HTML>"""
tags.insert(0,head)
tags.append(tail)
page = "n".join(tags)

Then save or serve you page.

Answered By: Steve Barnes

Answer 2

There is nothing special about the mirror, and you can use mod_rewrite to set it up yourself.

Dump your packages in a directory that is mapped to a URL. Here I am using /url/to/my/pypi/ an an example. The folder hierarchy should be /foo/bar/simple/[name of package]/[name of tarball]

Add the following to .htaccess or the global configuration for that directory where you packages are. The last block of lines is a fall back to the global pypi index:

Options +Indexes

RewriteEngine On

RewriteRule ^/robots.txt - [L]
RewriteRule ^/icons/.* - [L]
RewriteRule ^/index..* - [L]

RewriteCond /foo/bar/simple/ !-f
RewriteCond /foo/bar/simple/ !-d
RewriteRule ^/(.*)/?$ http://pypi.python.org/ [R,L]

Update your ~/.pip/pip.conf to point to the new repository:
```
[global]
index-url = http://localhost/url/to/my/pypi/
```
Or use the -i http://localhost/url/to/my/pypi/ option at the command line.

Answered By: Burhan Khalid

Answer 3

Since you asked to answer here:

Take a look at pip2pi, it seems to be exactly what you are looking for.

Answered By: K Z

Answer 4

We had a similar need at my company. Basically how can we upload “closed source” packages to an index while being able to install them as if they were on PyPI?

We have sponsored a project called devpi which acts as a PyPI cache (packages you access from PyPI will be cached on your server) as well as a powerful and fast index server. The documentation is available at http://doc.devpi.net/latest/.

Next on the roadmap is mirroring for multi geos deployment. To kick the tires on your machine takes about 5 minutes (look at the quick start guides). Finally devpi is compatible with both pip and easy_install (i.e. you do not need the devpi client installed on your machine).

Hope this help.

Answered By: Laurent Brack

Answer 5

The simplest way is to organize the package distfiles into package-named dirs and run a simple HTTP server. No extra packages needed, Python’s stdlib is enough. Directory structure example:

└── repodir
    ├── setuptools
    │   ├── setuptools-38.1.0-py2.py3-none-any.whl 
    │   ├── setuptools-38.1.0.zip
    │   ├── setuptools-39.2.0-py2.py3-none-any.whl 
    │   └── setuptools-39.2.0.zip
    ├── wheel
    │   └── wheel-0.31.1-py2.py3-none-any.whl 
    ...

Start the server:

$ cd repodir/
$ python3 -m http.server 9000
$ # or for Python 2:
$ python2 -m SimpleHTTPServer 9000

The local repo is up and running. Now you can pass the repo to pip:

$ pip install wheel --extra-index-url=http://127.0.0.1:9000

or even persist the repo URL in the pip.conf to not to enter it each time:

# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000

Reference: Python Packaging user Guide, Hosting your own simple repository

Answered By: hoefling

how to create local own pypi repository index without mirror?

Question:

Answers: