Credentials in pip.conf for private PyPI
Question:
I have a private PyPI repository. Is there any way to store credentials in pip.conf similar to .pypirc?
What I mean. Currently in .pypirc you can have such configuration:
[distutils]
index-servers = custom
[custom]
repository: https://pypi.example.com
username: johndoe
password: changeme
From what I’ve found that you can put in pip.conf:
[global]
index = https://username:[email protected]/pypi
index-url = https://username:[email protected]/simple
cert = /etc/ssl/certs/ca-certificates.crt
But here I see two problems:
- For each url you’ll need each time to specify the same username and password.
- Username and password become visible in the logs, cause they are part of the url.
Is there any way to store username and password outside of url?
Answers:
How about storing the Username/Password as environment variables,
export username=username
export password=password
and referring to them in the pip.conf like so:
[global]
index = https://$username:[email protected]/pypi
index-url = https://$username:[email protected]/simple
cert = /etc/ssl/certs/ca-certificates.crt
I use Gitlab CI’s secret variables for storing credentials. Check for an equivalent in your CI tool.
You could store credentials for Pip to use in ~/.netrc like this:
machine pypi.example.com
login johndoe
password changeme
Pip will use these credentials when accessing https://pypi.example.com but won’t log them. You must specify the index server separately (such as in pip.conf as in the question).
Note that ~/.netrc must be owned by the user pip executes as. It must not be readable by any other user, either. An invalid file is silently ignored. You can ensure the permissions are correct like this:
chown $USER ~/.netrc
chmod 0600 ~/.netrc
This permissions check doesn’t apply before Python 3.4, but it’s a good idea in any case.
Internally Pip uses requests when making HTTP requests. requests uses the standard library netrc module to read the file, so the character set is limited to an ASCII subset.
Given issue 4789 is still open, you can utilize the following approach in your requirements.txt:
--extra-index-url=https://${PYPI_USERNAME}:${PYPI_PASSWORD}@my.privatepypi.com
private-package==1.2.3
...
If you try to run pip install -r requirements.txt with those environment variables set, you will find that pip still asks for credentials. This is because pip does not interpolate the expression ${PYPI_USERNAME} as one would expect, but rather url encodes it. Your username and password in this example is expressed as https://%24%7BPYPI_USERNAME%7D:%24%7BPYPI_PASSWORD%[email protected]
The work around here, and I admit there should be a better approach, but we can run pip as a script:
python3 -m pip install -r requirements.txt
From man:
-m module-name
Searches sys.path for the named module and runs the corresponding .py file as a script.
I know it is not answering the question the way it was formulated, but one could also use keyring python module for pip authentication:
https://pypi.org/project/keyring/
https://pip.pypa.io/en/stable/topics/authentication/#keyring-support%3E
I have a private PyPI repository. Is there any way to store credentials in pip.conf similar to .pypirc?
What I mean. Currently in .pypirc you can have such configuration:
[distutils]
index-servers = custom
[custom]
repository: https://pypi.example.com
username: johndoe
password: changeme
From what I’ve found that you can put in pip.conf:
[global]
index = https://username:[email protected]/pypi
index-url = https://username:[email protected]/simple
cert = /etc/ssl/certs/ca-certificates.crt
But here I see two problems:
- For each url you’ll need each time to specify the same username and password.
- Username and password become visible in the logs, cause they are part of the url.
Is there any way to store username and password outside of url?
How about storing the Username/Password as environment variables,
export username=username
export password=password
and referring to them in the pip.conf like so:
[global]
index = https://$username:[email protected]/pypi
index-url = https://$username:[email protected]/simple
cert = /etc/ssl/certs/ca-certificates.crt
I use Gitlab CI’s secret variables for storing credentials. Check for an equivalent in your CI tool.
You could store credentials for Pip to use in ~/.netrc like this:
machine pypi.example.com
login johndoe
password changeme
Pip will use these credentials when accessing https://pypi.example.com but won’t log them. You must specify the index server separately (such as in pip.conf as in the question).
Note that ~/.netrc must be owned by the user pip executes as. It must not be readable by any other user, either. An invalid file is silently ignored. You can ensure the permissions are correct like this:
chown $USER ~/.netrc
chmod 0600 ~/.netrc
This permissions check doesn’t apply before Python 3.4, but it’s a good idea in any case.
Internally Pip uses requests when making HTTP requests. requests uses the standard library netrc module to read the file, so the character set is limited to an ASCII subset.
Given issue 4789 is still open, you can utilize the following approach in your requirements.txt:
--extra-index-url=https://${PYPI_USERNAME}:${PYPI_PASSWORD}@my.privatepypi.com
private-package==1.2.3
...
If you try to run pip install -r requirements.txt with those environment variables set, you will find that pip still asks for credentials. This is because pip does not interpolate the expression ${PYPI_USERNAME} as one would expect, but rather url encodes it. Your username and password in this example is expressed as https://%24%7BPYPI_USERNAME%7D:%24%7BPYPI_PASSWORD%[email protected]
The work around here, and I admit there should be a better approach, but we can run pip as a script:
python3 -m pip install -r requirements.txt
From man:
-m module-name
Searches sys.path for the named module and runs the corresponding .py file as a script.
I know it is not answering the question the way it was formulated, but one could also use keyring python module for pip authentication:
https://pypi.org/project/keyring/
https://pip.pypa.io/en/stable/topics/authentication/#keyring-support%3E