Can I turn off parallel jobs in Azure DevOps pipeline?
Question:
I am new to Azure pipelines. I am trying to create pipeline for deploying simple python application.
But I get error
No hosted parallelism has been purchased or granted
As I understand microsoft disabled the free grant of parallel jobs for public projects and for certain private projects in new organizations. But what if I don’t need parallel jobs? I need jobs just to run one after the other. Can I turn off using of parallel jobs?
I chose template "Python package" and set environment variables "python.version" only one version "3.7". But it doesn’t help. I still have the same error
No hosted parallelism has been purchased or granted
Answers:
Free tier supports 1 Parallel job which is 1 job only.
See the microsoft’s defination of parallel job below-
What is a parallel job?
When you define a pipeline, you can define it as a collection of jobs. When a pipeline runs, you can run multiple jobs as part of that pipeline. Each running job consumes a parallel job that runs on an agent. When there aren’t enough parallel jobs available for your organization, the jobs are queued up and run one after the other.
As you rightly mentioned it’s temporarily disabled by msft for Private Projects. However, you can ask for granting access to the free job. This can take upto take 2-3 days for you to get access.
To request the free grant for public or private projects, submit a request here
Free grant discontinuation
Microsoft disabled the free grant of parallel jobs for public projects and for certain private projects in new organizations. Because users abused (e.g. crypto mining) the free computational resources. 1
You can solve the problem in two ways:
- Request the free grant by submitting the https://aka.ms/azpipelines-parallelism-request form.
- Create a Self-hosted agent.
Requesting the free grant usually takes two or three days. If you do not want to wait for the request approval, you can create your own self-hosted agent.
Create a Self-hosted Agent
In order to create a self-hosted agent, go to Project Settings and select the Agent pools option under the Pipelines section. Press the Add pool button and configure the agent:
- Select the Pool type as Self-hosted
- Add a descriptive pool Name (in this example
my-personal-computer)
- Check the Pipeline permissions box so you do not need to grant permission manually
- Click on the Create button
Now, navigate to the freshly created agent. On the top-right corner, press the New Agent button. You can create Windows, macOS, and Linux agents. Based on your computer select the appropriate OS then follow the instructions:
-
Extract the archive: Download the agent then extract it to a folder
-
Configure the agent: Run the config scripts and select the default settings:
- Server URL: Copy and paste the organization URL which looks like the following
https://dev.azure.com/<my-organization-name>
- Personal Access Token (PAT): Go to the Personal Access Tokens option under the User Settings icon. Make sure that you generate a PAT which has Read & manage access to the Agent pools
- Agent pool name: The newly created pool which is the
my-personal-computer in our case
- Agent Name: Give a meaningful name (e.g
Windows-Agent) or stay with the default
- Work folder: Press enter for the default
- Agent as Service: Press enter for the default
-
Run the agent: Execute the run script
Once it is done, you can see that the Agent is up and running under the Agents panel.
The self-hosted agent is connected to Azure DevOps and listens for new jobs.
Note: If you create a service during the agent configuration then you do not need to run the agent manually.
Configure the Pipeline
The last step is to update the Pipeline to use the recently created agent. Go to the Pipelines and select your pipeline. Click on Edit and add the pool: <my-selfhosted-agent> section to the config file. Here is an azure-pipeline.yml file example:
trigger:
- main
pool: my-personal-computer
strategy:
matrix:
Python36:
python.version: '3.6'
Python37:
python.version: '3.7'
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'
- script: |
python -m pip install --upgrade pip
pip install -r requirements.txt
displayName: 'Install dependencies'
- script: |
pip install pytest pytest-azurepipelines
pytest
displayName: 'pytest'
It will go to the pool and select an available agent. We have only one agent so it selects it and run the various jobs on it. You can go inside your self-hosted agent folder and get the logs from the _work directory. You can also view the output of the jobs on Azure DevOps.
I am new to Azure pipelines. I am trying to create pipeline for deploying simple python application.
But I get error
No hosted parallelism has been purchased or granted
As I understand microsoft disabled the free grant of parallel jobs for public projects and for certain private projects in new organizations. But what if I don’t need parallel jobs? I need jobs just to run one after the other. Can I turn off using of parallel jobs?
I chose template "Python package" and set environment variables "python.version" only one version "3.7". But it doesn’t help. I still have the same error
No hosted parallelism has been purchased or granted
Free tier supports 1 Parallel job which is 1 job only.
See the microsoft’s defination of parallel job below-
What is a parallel job?
When you define a pipeline, you can define it as a collection of jobs. When a pipeline runs, you can run multiple jobs as part of that pipeline. Each running job consumes a parallel job that runs on an agent. When there aren’t enough parallel jobs available for your organization, the jobs are queued up and run one after the other.
As you rightly mentioned it’s temporarily disabled by msft for Private Projects. However, you can ask for granting access to the free job. This can take upto take 2-3 days for you to get access.
To request the free grant for public or private projects, submit a request here
Free grant discontinuation
Microsoft disabled the free grant of parallel jobs for public projects and for certain private projects in new organizations. Because users abused (e.g. crypto mining) the free computational resources. 1
You can solve the problem in two ways:
- Request the free grant by submitting the https://aka.ms/azpipelines-parallelism-request form.
- Create a Self-hosted agent.
Requesting the free grant usually takes two or three days. If you do not want to wait for the request approval, you can create your own self-hosted agent.
Create a Self-hosted Agent
In order to create a self-hosted agent, go to Project Settings and select the Agent pools option under the Pipelines section. Press the Add pool button and configure the agent:
- Select the Pool type as Self-hosted
- Add a descriptive pool Name (in this example
my-personal-computer) - Check the Pipeline permissions box so you do not need to grant permission manually
- Click on the Create button
Now, navigate to the freshly created agent. On the top-right corner, press the New Agent button. You can create Windows, macOS, and Linux agents. Based on your computer select the appropriate OS then follow the instructions:
-
Extract the archive: Download the agent then extract it to a folder
-
Configure the agent: Run the config scripts and select the default settings:
- Server URL: Copy and paste the organization URL which looks like the following
https://dev.azure.com/<my-organization-name> - Personal Access Token (PAT): Go to the Personal Access Tokens option under the User Settings icon. Make sure that you generate a PAT which has Read & manage access to the Agent pools
- Agent pool name: The newly created pool which is the
my-personal-computerin our case - Agent Name: Give a meaningful name (e.g
Windows-Agent) or stay with the default - Work folder: Press enter for the default
- Agent as Service: Press enter for the default
- Server URL: Copy and paste the organization URL which looks like the following
-
Run the agent: Execute the run script
Once it is done, you can see that the Agent is up and running under the Agents panel.
The self-hosted agent is connected to Azure DevOps and listens for new jobs.
Note: If you create a service during the agent configuration then you do not need to run the agent manually.
Configure the Pipeline
The last step is to update the Pipeline to use the recently created agent. Go to the Pipelines and select your pipeline. Click on Edit and add the pool: <my-selfhosted-agent> section to the config file. Here is an azure-pipeline.yml file example:
trigger:
- main
pool: my-personal-computer
strategy:
matrix:
Python36:
python.version: '3.6'
Python37:
python.version: '3.7'
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'
- script: |
python -m pip install --upgrade pip
pip install -r requirements.txt
displayName: 'Install dependencies'
- script: |
pip install pytest pytest-azurepipelines
pytest
displayName: 'pytest'
It will go to the pool and select an available agent. We have only one agent so it selects it and run the various jobs on it. You can go inside your self-hosted agent folder and get the logs from the _work directory. You can also view the output of the jobs on Azure DevOps.