Publishing package to PyPi from Azure Devops hangs at twine upload
Question:
I want to publish a built .whl package to PyPi.org from our Azure Devops release pipeline, but the script twine upload keeps hanging without error, completion or time out. So the actual upload of our package (which is extremely small) does not work.
This is how it is set up:
The yaml build:
trigger:
- master
pr: none
pool:
vmImage: 'ubuntu-latest'
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.7'
addToPath: true
architecture: 'x64'
- script: python -m pip install --upgrade pip setuptools wheel
displayName: 'Install tools'
- script: pip install -r src/requirements.txt
displayName: 'Install requirements'
- script: |
python src/setup.py bdist_wheel
displayName: 'Artifact creation'
- script: python -m pip install --upgrade twine
displayName: 'Install Twine'
- task: TwineAuthenticate@1
inputs:
pythonUploadServiceConnection: 'AzureML PyPi feed'
- script: |
python -m twine upload --config-file $(PYPIRC_PATH) dist/*.whl
displayName: 'Publish to PyPi through Twine'
The service connection:
- The authentication method is specified by
Authentication Token.
- Python repository url for upload:
https://upload.pypi.org/legacy
- EndpointName:
azure-pypi
- Service Connection name:
AzureML PyPi feed
- [X] Grant access permission to all pipelines
The release logs (most relevant parts):
TwineAuthenticate step:
Starting: TwineAuthenticate
==============================================================================
Task : Python twine upload authenticate
Description : Authenticate for uploading Python distributions using twine. Add '-r FeedName/EndpointName --config-file $(PYPIRC_PATH)' to your twine upload command. For feeds present in this organization, use the feed name as the repository (-r). Otherwise, use the endpoint name defined in the service connection.
Version : 1.165.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/package/twine-authenticate
==============================================================================
75c64e89-xxxxxredactedxxxxx exists true
Adding authentication to configuration for registry azure-pypi
Successfully added auth for 0 internal feed and 1 external endpoint.
Finishing: TwineAuthenticate
Publishing step:
Starting: Publish to PyPi through Twine
==============================================================================
Task : Command line
Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version : 2.164.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
python -m twine upload --config-file /home/vsts/work/_temp/twineAuthenticate/Wv8nMR/.pypirc dist/*.whl
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/0d772e31-148f-4e3c-999b-b2a43a02b287.sh
Uploading distributions to https://upload.pypi.org/legacy/
This keeps unchanged for more than 30 minutes, so I just cancel my release without the package being deployed. Any idea on this?
Answers:
I have a temporary workaround for now:
I skipped the usage of a config file and executed the following code instead:
- script: |
python -m twine upload --skip-existing --verbose -p $(pypi-api-token) -u __token__ --repository $(pypi-project-name) --repository-url https://upload.pypi.org/legacy/ dist/*.whl
displayName: 'Publish to PyPi through Twine'
And there, I noticed a bit more (and better) exception logging, that pointed me out two things:
- The token I was using for my config file was set to a project scope with a different name than the project I specified in my Service Connection
- The repository url really has to end with the trailing forward slash, because otherwise you get a RedirectDetected exception.
Based on the above findings, I updated my yaml snippet to the following:
- script: |
python -m twine upload --skip-existing --verbose --repository $(pypi-project-name) --config-file $(PYPIRC_PATH) dist/*.whl
displayName: 'Publish to PyPi through Twine'
But when executing this, I now get the following exception message in the build:
Generating script.
Script contents:
python -m twine upload --skip-existing --verbose --repository arcus-azureml --config-file /home/vsts/work/_temp/twineAuthenticate/2QGKVH/.pypirc dist/*.whl
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/7605dac9-5fa9-4856-94af-e938018278a5.sh
Uploading distributions to https://upload.pypi.org/legacy/
Uploading arcus_azureml-0.0.2-py3-none-any.whl
0%| | 0.00/5.80k [00:00<?, ?B/s]
100%|██████████| 5.80k/5.80k [00:00<00:00, 52.4kB/s]HTTPError: 403 Client Error: Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details for url: https://upload.pypi.org/legacy/
Content received from server:
<html>
<head>
<title>403 Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details</title>
</head>
<body>
<h1>403 Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details</h1>
Access was denied to this resource.<br/><br/>
Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details
I also outputted the contents of the pypirc file, using the CAT command and the contents of the file are like this:
[distutils]
index-servers=arcus-azureml
[arcus-azureml]
repository=https://upload.pypi.org/legacy/
username=build
password=***
Updated solution
So, the fix to have the 403 arranged was to change my Service Connection to use ‘Username and Password’ as authentication method , instead of Authentication Token, with the following settings:
- Username:
__token__
- Password: The actual api token as password
After performing this, everything was working the way I wanted it.
In my case, I needed to publish to testpypi. My AzureDevops twine update kept hanging on the "Querying keyring for username" regardless of a service connection with Authorization Token or Username and Password. Solution was to set the Python repository url to https://test.pypi.org/legacy/ rather than https://test.pypi.org. Once that was solved, I too required the service connection to be Username and Password, with username set to __token__
I want to publish a built .whl package to PyPi.org from our Azure Devops release pipeline, but the script twine upload keeps hanging without error, completion or time out. So the actual upload of our package (which is extremely small) does not work.
This is how it is set up:
The yaml build:
trigger:
- master
pr: none
pool:
vmImage: 'ubuntu-latest'
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.7'
addToPath: true
architecture: 'x64'
- script: python -m pip install --upgrade pip setuptools wheel
displayName: 'Install tools'
- script: pip install -r src/requirements.txt
displayName: 'Install requirements'
- script: |
python src/setup.py bdist_wheel
displayName: 'Artifact creation'
- script: python -m pip install --upgrade twine
displayName: 'Install Twine'
- task: TwineAuthenticate@1
inputs:
pythonUploadServiceConnection: 'AzureML PyPi feed'
- script: |
python -m twine upload --config-file $(PYPIRC_PATH) dist/*.whl
displayName: 'Publish to PyPi through Twine'
The service connection:
- The authentication method is specified by
Authentication Token. - Python repository url for upload:
https://upload.pypi.org/legacy - EndpointName:
azure-pypi - Service Connection name:
AzureML PyPi feed - [X] Grant access permission to all pipelines
The release logs (most relevant parts):
TwineAuthenticate step:
Starting: TwineAuthenticate
==============================================================================
Task : Python twine upload authenticate
Description : Authenticate for uploading Python distributions using twine. Add '-r FeedName/EndpointName --config-file $(PYPIRC_PATH)' to your twine upload command. For feeds present in this organization, use the feed name as the repository (-r). Otherwise, use the endpoint name defined in the service connection.
Version : 1.165.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/package/twine-authenticate
==============================================================================
75c64e89-xxxxxredactedxxxxx exists true
Adding authentication to configuration for registry azure-pypi
Successfully added auth for 0 internal feed and 1 external endpoint.
Finishing: TwineAuthenticate
Publishing step:
Starting: Publish to PyPi through Twine
==============================================================================
Task : Command line
Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version : 2.164.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
python -m twine upload --config-file /home/vsts/work/_temp/twineAuthenticate/Wv8nMR/.pypirc dist/*.whl
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/0d772e31-148f-4e3c-999b-b2a43a02b287.sh
Uploading distributions to https://upload.pypi.org/legacy/
This keeps unchanged for more than 30 minutes, so I just cancel my release without the package being deployed. Any idea on this?
I have a temporary workaround for now:
I skipped the usage of a config file and executed the following code instead:
- script: |
python -m twine upload --skip-existing --verbose -p $(pypi-api-token) -u __token__ --repository $(pypi-project-name) --repository-url https://upload.pypi.org/legacy/ dist/*.whl
displayName: 'Publish to PyPi through Twine'
And there, I noticed a bit more (and better) exception logging, that pointed me out two things:
- The token I was using for my config file was set to a project scope with a different name than the project I specified in my Service Connection
- The repository url really has to end with the trailing forward slash, because otherwise you get a RedirectDetected exception.
Based on the above findings, I updated my yaml snippet to the following:
- script: |
python -m twine upload --skip-existing --verbose --repository $(pypi-project-name) --config-file $(PYPIRC_PATH) dist/*.whl
displayName: 'Publish to PyPi through Twine'
But when executing this, I now get the following exception message in the build:
Generating script.
Script contents:
python -m twine upload --skip-existing --verbose --repository arcus-azureml --config-file /home/vsts/work/_temp/twineAuthenticate/2QGKVH/.pypirc dist/*.whl
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/7605dac9-5fa9-4856-94af-e938018278a5.sh
Uploading distributions to https://upload.pypi.org/legacy/
Uploading arcus_azureml-0.0.2-py3-none-any.whl
0%| | 0.00/5.80k [00:00<?, ?B/s]
100%|██████████| 5.80k/5.80k [00:00<00:00, 52.4kB/s]HTTPError: 403 Client Error: Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details for url: https://upload.pypi.org/legacy/
Content received from server:
<html>
<head>
<title>403 Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details</title>
</head>
<body>
<h1>403 Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details</h1>
Access was denied to this resource.<br/><br/>
Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details
I also outputted the contents of the pypirc file, using the CAT command and the contents of the file are like this:
[distutils]
index-servers=arcus-azureml
[arcus-azureml]
repository=https://upload.pypi.org/legacy/
username=build
password=***
Updated solution
So, the fix to have the 403 arranged was to change my Service Connection to use ‘Username and Password’ as authentication method , instead of Authentication Token, with the following settings:
- Username:
__token__ - Password: The actual api token as password
After performing this, everything was working the way I wanted it.
In my case, I needed to publish to testpypi. My AzureDevops twine update kept hanging on the "Querying keyring for username" regardless of a service connection with Authorization Token or Username and Password. Solution was to set the Python repository url to https://test.pypi.org/legacy/ rather than https://test.pypi.org. Once that was solved, I too required the service connection to be Username and Password, with username set to __token__