Python setuptools: How can I list a private repository under install_requires?
Question:
I am creating a setup.py
file for a project which depends on private GitHub repositories. The relevant parts of the file look like this:
from setuptools import setup
setup(name='my_project',
...,
install_requires=[
'public_package',
'other_public_package',
'private_repo_1',
'private_repo_2',
],
dependency_links=[
'https://github.com/my_account/private_repo_1/master/tarball/',
'https://github.com/my_account/private_repo_2/master/tarball/',
],
...,
)
I am using setuptools
instead of distutils
because the latter does not support the install_requires
and dependency_links
arguments per this answer.
The above setup file fails to access the private repos with a 404 error – which is to be expected since GitHub returns a 404 to unauthorized requests for a private repository. However, I can’t figure out how to make setuptools
authenticate.
Here are some things I’ve tried:
-
Use git+ssh://
instead of https://
in dependency_links
as I would if installing the repo with pip
. This fails because setuptools doesn’t recognize this protocol (“unknown url type: git+ssh”), though the distribute documentation says it should. Ditto git+https
and git+http
.
-
https://<username>:<password>@github.com/...
– still get a 404. (This method doesn’t work with curl
or wget
from the command line either – though curl -u <username> <repo_url> -O <output_file_name>
does work.)
-
Upgrading setuptools (0.9.7) and virtualenv (1.10) to the latest versions. Also tried installing distribute though this overview says it was merged back into setuptools. Either way, no dice.
Currently I just have setup.py
print out a warning that the private repos must be downloaded separately. This is obviously less than ideal. I feel like there’s something obvious that I’m missing, but can’t think what it might be. 🙂
Duplicate-ish question with no answers here.
Answers:
Edit: This appears to only work with public github repositories, see comments.
dependency_links=[
'https://github.com/my_account/private_repo_1/tarball/master#egg=private_repo_1',
'https://github.com/my_account/private_repo_2/tarball/master#egg=private_repo_2',
],
Above syntax seems to work for me with setuptools 1.0. At the moment at least the syntax of adding “#egg=project_name-version” to VCS dependencies is documented in the link you gave to distribute documentation.
Here’s what worked for me:
install_requires=[
'private_package_name==1.1',
],
dependency_links=[
'git+ssh://[email protected]/username/private_repo.git#egg=private_package_name-1.1',
]
Note that you have to have the version number in the egg name, otherwise it will say it can’t find the package.
Using archive URL from github works for me, for public repositories. E.g.
dependency_links = [
'https://github.com/username/reponame/archive/master.zip#egg=eggname-version',
]
I found a (hacky) workaround:
#!/usr/bin/env python
from setuptools import setup
import os
os.system('pip install git+https://github-private.corp.com/user/repo.git@master')
setup( name='original-name'
, ...
, install_requires=['repo'] )
I understand that there are ethical issues with having a system call in a setup script, but I can’t think of another way to do this.
I couldn’t find any good documentation on this, but came across the solution mainly through trial & error. Further, installing from pip & setuptools have some subtle differences; but this way should work for both.
GitHub don’t (currently, as of August 2016) offer an easy way to get the zip / tarball of private repos. So you need to point setuptools to tell setuptools that you’re pointing to a git repo:
from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']
setup(
# ...
install_requires='package',
dependency_links = [
'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
.format(github_token=github_token, package=package, version=master)
]
A couple of notes here:
- For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
- You need to include some version number (here is
0
) at the end of the link, even if there’s no package on PyPI. This has to be a actual number, not a word.
- You need to preface with
git+
to tell setuptools it’s to clone the repo, rather than pointing at a zip / tarball
version
can be a branch, a tag, or a commit hash
- You need to supply
--process-dependency-links
if installing from pip
This work for our scenario:
- package is on github in a private repo
- we want to install it into site-packages (not into ./src with -e)
- being able to use pip install -r requirements.txt
- being able to use pip install -e reposdir (or from github), where the dependencies are only specified in requirements.txt
https://github.com/pypa/pip/issues/3610#issuecomment-356687173
I was trying to get this to work for installing with pip, but the above was not working for me. From [1] I understood the PEP508
standard should be used, from [2] I retrieved an example which actually does work (at least for my case).
Please note; this is with pip 20.0.2
on Python 3.7.4
setup(
name='<package>',
...
install_requires=[
'<normal_dependency>',
# Private repository
'<dependency_name> @ git+ssh://[email protected]/<user>/<repo_name>@<branch>',
# Public repository
'<dependency_name> @ git+https://github.com/<user>/<repo_name>@<branch>',
],
)
After specifying my package this way installation works fine (also with -e
settings and without the need to specify --process-dependency-links
).
References
[1] https://github.com/pypa/pip/issues/4187
[2] https://github.com/pypa/pip/issues/5566
Via Tom Hemmes’ answer I found this is the only thing that worked for me:
install_requires=[
'<package> @ https://github.com/<username>/<package>/archive/<branch_name>.zip']
With pip 20.1.1, this works for me
install_requires=[ "packson3@https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/138/packson3/dist/packson3-1.0.0.tar.gz"],
in setup.py
I am creating a setup.py
file for a project which depends on private GitHub repositories. The relevant parts of the file look like this:
from setuptools import setup
setup(name='my_project',
...,
install_requires=[
'public_package',
'other_public_package',
'private_repo_1',
'private_repo_2',
],
dependency_links=[
'https://github.com/my_account/private_repo_1/master/tarball/',
'https://github.com/my_account/private_repo_2/master/tarball/',
],
...,
)
I am using setuptools
instead of distutils
because the latter does not support the install_requires
and dependency_links
arguments per this answer.
The above setup file fails to access the private repos with a 404 error – which is to be expected since GitHub returns a 404 to unauthorized requests for a private repository. However, I can’t figure out how to make setuptools
authenticate.
Here are some things I’ve tried:
-
Use
git+ssh://
instead ofhttps://
independency_links
as I would if installing the repo withpip
. This fails because setuptools doesn’t recognize this protocol (“unknown url type: git+ssh”), though the distribute documentation says it should. Dittogit+https
andgit+http
. -
https://<username>:<password>@github.com/...
– still get a 404. (This method doesn’t work withcurl
orwget
from the command line either – thoughcurl -u <username> <repo_url> -O <output_file_name>
does work.) -
Upgrading setuptools (0.9.7) and virtualenv (1.10) to the latest versions. Also tried installing distribute though this overview says it was merged back into setuptools. Either way, no dice.
Currently I just have setup.py
print out a warning that the private repos must be downloaded separately. This is obviously less than ideal. I feel like there’s something obvious that I’m missing, but can’t think what it might be. 🙂
Duplicate-ish question with no answers here.
Edit: This appears to only work with public github repositories, see comments.
dependency_links=[
'https://github.com/my_account/private_repo_1/tarball/master#egg=private_repo_1',
'https://github.com/my_account/private_repo_2/tarball/master#egg=private_repo_2',
],
Above syntax seems to work for me with setuptools 1.0. At the moment at least the syntax of adding “#egg=project_name-version” to VCS dependencies is documented in the link you gave to distribute documentation.
Here’s what worked for me:
install_requires=[
'private_package_name==1.1',
],
dependency_links=[
'git+ssh://[email protected]/username/private_repo.git#egg=private_package_name-1.1',
]
Note that you have to have the version number in the egg name, otherwise it will say it can’t find the package.
Using archive URL from github works for me, for public repositories. E.g.
dependency_links = [
'https://github.com/username/reponame/archive/master.zip#egg=eggname-version',
]
I found a (hacky) workaround:
#!/usr/bin/env python
from setuptools import setup
import os
os.system('pip install git+https://github-private.corp.com/user/repo.git@master')
setup( name='original-name'
, ...
, install_requires=['repo'] )
I understand that there are ethical issues with having a system call in a setup script, but I can’t think of another way to do this.
I couldn’t find any good documentation on this, but came across the solution mainly through trial & error. Further, installing from pip & setuptools have some subtle differences; but this way should work for both.
GitHub don’t (currently, as of August 2016) offer an easy way to get the zip / tarball of private repos. So you need to point setuptools to tell setuptools that you’re pointing to a git repo:
from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']
setup(
# ...
install_requires='package',
dependency_links = [
'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
.format(github_token=github_token, package=package, version=master)
]
A couple of notes here:
- For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
- You need to include some version number (here is
0
) at the end of the link, even if there’s no package on PyPI. This has to be a actual number, not a word. - You need to preface with
git+
to tell setuptools it’s to clone the repo, rather than pointing at a zip / tarball version
can be a branch, a tag, or a commit hash- You need to supply
--process-dependency-links
if installing from pip
This work for our scenario:
- package is on github in a private repo
- we want to install it into site-packages (not into ./src with -e)
- being able to use pip install -r requirements.txt
- being able to use pip install -e reposdir (or from github), where the dependencies are only specified in requirements.txt
https://github.com/pypa/pip/issues/3610#issuecomment-356687173
I was trying to get this to work for installing with pip, but the above was not working for me. From [1] I understood the PEP508
standard should be used, from [2] I retrieved an example which actually does work (at least for my case).
Please note; this is with pip 20.0.2
on Python 3.7.4
setup(
name='<package>',
...
install_requires=[
'<normal_dependency>',
# Private repository
'<dependency_name> @ git+ssh://[email protected]/<user>/<repo_name>@<branch>',
# Public repository
'<dependency_name> @ git+https://github.com/<user>/<repo_name>@<branch>',
],
)
After specifying my package this way installation works fine (also with -e
settings and without the need to specify --process-dependency-links
).
References
[1] https://github.com/pypa/pip/issues/4187
[2] https://github.com/pypa/pip/issues/5566
Via Tom Hemmes’ answer I found this is the only thing that worked for me:
install_requires=[
'<package> @ https://github.com/<username>/<package>/archive/<branch_name>.zip']
With pip 20.1.1, this works for me
install_requires=[ "packson3@https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/138/packson3/dist/packson3-1.0.0.tar.gz"],
in setup.py