Change python version to 3.x
Question:
According to poetry’s docs, the proper way to setup a new project is with poetry new poetry-demo
, however this creates a project based on the now deprecated python2.7 by creating the following toml file:
[tool.poetry]
name = "poetry-demo"
version = "0.1.0"
description = ""
authors = ["Harsha Goli <[email protected]>"]
[tool.poetry.dependencies]
python = "^2.7"
[tool.poetry.dev-dependencies]
pytest = "^4.6"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
How can I update this to 3.7? Simply changing python = "^2.7"
to python = "^3.7"
results in the following error when poetry install
is run:
[SolverProblemError]
The current project's Python requirement (2.7.17) is not compatible with some of the required packages Python requirement:
- zipp requires Python >=3.6
Because no versions of pytest match >=4.6,<4.6.9 || >4.6.9,<5.0
and pytest (4.6.9) depends on importlib-metadata (>=0.12), pytest (>=4.6,<5.0) requires importlib-metadata (>=0.12).
And because no versions of importlib-metadata match >=0.12,<1.5.0 || >1.5.0
and importlib-metadata (1.5.0) depends on zipp (>=0.5), pytest (>=4.6,<5.0) requires zipp (>=0.5).
Because zipp (3.1.0) requires Python >=3.6
and no versions of zipp match >=0.5,<3.1.0 || >3.1.0, zipp is forbidden.
Thus, pytest is forbidden.
So, because poetry-demo depends on pytest (^4.6), version solving failed.
Answers:
Interestingly, poetry is silently failing due to a missing package the tool itself relies on and continues to install a broken venv. Here’s how you fix it.
sudo apt install python3-venv
poetry env remove python3
poetry install
I had to remove pytest, and then reinstall with poetry add pytest
.
EDIT: I ran into this issue again when upgrading a project from python3.7 to python3.8 – for this instead of installing python3-venv
, you’d want to install python3.8-venv
instead
Whenever you change dependencies by hand in your pyproject.toml
you have to take care of these points:
-
Run poetry lock --no-update
afterwards. The reasons for this is, that poetry install
takes the poetry.lock
as input if can find one and not the pyproject.toml
.
-
If you change the python version and uses in-project virtualenv, remove the .venv
before running poetry install
. poetry doesn’t change the python version of a venv once it is created, because it uses the python version itself to create the virtualenv.
You can change in pyproject.toml and execute de this command "poetry env use 3.x" that works for me.
I had the same problem. I solve it by fixing the first line in the file /home/nordman/.poetry/bin/poetry
(nordman
is my local name).
Just change #!/usr/bin/env python
to #!/usr/bin/env python3
Poetry makes it super easy to work with different Python versions or virtual environments. The recommended way to specify your Python version according to Poetry docs is
poetry env use /path/to/preferred/python/version
You can get the path to your Python version by running
which python3.7
on Linux or
py -0p
on Windows.
I’ve experienced the exact same behavior on my MacBook! So to make this practical, first, let’s see, what is the default Python for the MacBook. I’m running macOS Big Sur 11.2.1 (20D74). In terminal:
python --version
Python 2.7.16
With that, let’s install poetry
. I’ve used poetry
installation script from the GitHub. Using this script is a recommended method to install poetry
(though I’d argue this is NOT the best idea to pipe scripts from the Internet into the python
interpreter)
The installation script is pretty smart: it attempts to detect which python
is available on the system:
def _which_python(self):
"""Decides which python executable we'll embed in the launcher script."""
allowed_executables = ["python", "python3"]
if WINDOWS:
allowed_executables += ["py.exe -3", "py.exe -2"]
...
So as the default Python is 2.7.16, and that is what returned by calling python
, the entry in ~/.poetry/bin
will look like:
#!/usr/bin/env python
And here you go! The default python
will end up in the pyproject.toml
file and you’ll need to do an extra dance to a) make sure python3
is a dependency for your project; and b) that the virtual environment will use python3
as python
interpreter.
As @mfalade mentioned, you can set environment with poetry env
:
poetry env use /path/to/python3
And also modifying the pyproject.toml
with:
...
[tool.poetry.dependencies]
python = "^3.9"
...
From this point you are good to go, poetry
will use python3
to create virtual environment for you project, dependency is specified in the file and … the grass is green again.
But can we use python3
by default? There is an insightful thread on GitHub and another one. And if you look back at the code snippet from the installation script above, you might wonder why not to check for python3
first and then for python, especially taking into account that this is the exact order for Windows installation. Well, you are not alone, I also wonder 😉
I would not suggest editing the ~/.poetry/bin/poetry
file directly as this file is generated by the installation script (so what will happen if you run installation script again? And again?).
Really, this is a minor annoyance, and knowing the tool it is easy to work around it. I’d expect that to be mentioned in installation guide though…
Anyway, hope it helps!
You can simply use pyenv
for that. Create .python-version
file inside your project and poetry
will match the exact python version.
# check current python version (set up globally)
❯ pyenv version
3.9.0 (set by /Users/[email protected]/.python-version)
# create .python-version file for project
❯ pyenv local 3.9.0
# check python version again (now it's set up locally)
❯ pyenv version
3.9.0 (set by /Users/[email protected]/Documents/myproject/.python-version)
❯ poetry lock
(...)
❯ poetry run python --version
Python 3.9.0
Use:
poetry env use 3.9
or
poetry env use $(which python3.9)
doesn’t work for some reason for python 3.9. Though having explicitly stated version requirement in pyproject.toml
file, it’s sad that one needs to manually enforce this.
The way I do it, and it’s super effective:
Use pyenv to manage the pyenv versions, and it’s plugin pyenv-virtualenv to manage the versions as virtual environments, then poetry to pick the version from them automatically as it manages the dependencies:
Use pyenv to install whatever version of python you want to use:
pyenv install -v 3.7
Create a virtual environment based on the installed version 3.7.
You will need the plugin pyenv-virtualenv.
pyenv virtualenv 3.7 arshbot_3.7_virtualenv
Assume a project called arshbot_proj. Working in $HOME directory:
mkdir ~/arshbot_proj
cd ~/arshbot_proj
Use the virtual env we just created. Attach this project to it.
Below creates .python-version file indicating arshbot_3.7_virtualenv.
pyenv local arshbot_3.7_virtualenv
Use poetry init to create the pyproject.toml inside the dir using
arshbot_3.7_virtualenv.
You could also use poetry new instead of poetry init to create project
structure alongside pyproject.toml instead of making your project first in
above step.
With pyproject.toml and .python-version both inside this dir, poetry
will automatically pick the 3.7 courtesy of arshbot_3.7_virtualenv.
Poetry will also use this virtual env to install packages at
~/.pyenv/versions/3.7/envs/arshbot_3.7_virtualenv/lib/python3.7/site-packages
poetry init
That is it. Poetry will pick the 3.7 automatically every time you run it from inside that directory attached to that virtual environment, with it activated. For different python version, just repeat the above steps, replace 3.7 with the new version.
The virtual environment will appear twice: In the envs directory as a virtual environment, and also in the versions directory as a version, with contents replicating when poetry installs packages.
BONUS:
We try to make it even more clear using a different approach, same concept, same results.
Since by default, if you didn’t change PYENV_ROOT, pyenv installs every python version in ~/.pyenv/versions:
If you want the system version(shipped with your distro) to be part of the versions you can select, use venv to mimick a pyenv installation of a python version, now call it system_ver in place of the 3.7. Since it’s already in the system, we don’t need pyenv to download it, we copy it over to our versions directory, so that it’s available to create a virtual environment
cd ~/.pyenv/versions
python3 -m venv --copies system system_ver
pyenv virtualenv system_ver system_ver_virtualenv
To use it in your project in place of previous version 3.7:
cd ~/arshbot_proj
pyenv local system_ver_virtualenv
Poetry will now use whatever version came originally with your distro. The –copies will ensure venv copies the files instead of use links, so you may omit it. Usually useful if you need to later make a multi stage dockerfile for the project using the files from the virtual environment.
I tried every solution I could find, but could not get poetry to pick up the upgraded version of Python. Finally found something that worked.
tldr;
- Run
poetry config virtualenvs.in-project true
to use local venvs
- Run
deactivate
to exit any existing venv
- Run
poetry shell
to activate a new, local venv
So here’s what I was seeing:
(my-project) ~/I/my-project ❯❯❯ poetry env info
Virtualenv
Python: 3.9.6
Implementation: CPython
Path: /Users/my-user/.local/share/virtualenvs/my-project-some-hash
Valid: True
System
Platform: darwin
OS: posix
Python: /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.9
I tried running poetry env remove python
and got an error:
(my-project) ~/I/my-project ❯❯❯ poetry env remove python
ValueError
Environment "my-project-some-hash-py3.9" does not exist.
Meanwhile, I saw somewhere that it’s recommend to use local virtual environments by setting this property, so I did it:
poetry config virtualenvs.in-project true
This did not solve my problem, but then I realized that changing this setting would not take affect automatically because I was already in another non-local virtual environment (see poetry env info
output above).
So based on the docs, I ran deactivate
to deactivate the current virtual environment.
Now, I saw this:
~/I/my-project ❯❯❯ poetry env info
Virtualenv
Python: 3.10.1
Implementation: CPython
Path: NA
System
Platform: darwin
OS: posix
Python: /Users/my-user/.pyenv/versions/3.10.1
Now I ran poetry shell
to create a new virtual environment and it picked up the new local venvs setting:
~/I/my-project ❯❯❯ poetry shell
Creating virtualenv my-project in /Users/my-user/projects/my-project/.venv
Spawning shell within /Users/my-user/projects/my-project/.venv
And finally, I saw the upgraded Python version I was expecting!
(.venv) ~/I/my-project ❯❯❯ poetry run python -V
Python 3.10.1
Update: After I did all of the above, I noticed that poetry env use
works fine!
(.venv) ~/I/my-project ❯❯❯ python -V
Python 3.10.1
(.venv) ~/I/my-project ❯❯❯ poetry env use 3.9.6
Recreating virtualenv my-project in /Users/my-user/projects/my-project/.venv
Using virtualenv: /Users/my-user/projects/my-project/.venv
(.venv) ~/I/my-project ❯❯❯ python -V
Python 3.9.6
I am running ubuntu 22.04 and only have python 3.10 installed. I’ve destroyed my system python enough times that I refuse to dance with older versions, or running any installs from pip. What I wanted is an older version of python, that was installed by not installed over my current python install. What I did was follow this guide.
sudo apt install libgdbm-dev build-essential libnss3-dev libreadline-dev libffi-dev libsqlite3-dev libbz2-dev libncurses5-dev libssl-dev zlib1g-dev
cd /tmp
wget https://www.python.org/ftp/python/3.9.14/Python-3.9.14.tgz
tar -xf Python-3.9.14.tgz
cd Python-3.9.14
./configure --enable-optimizations
## if you want to run in parallel set -j to how many cpus you have
## I subtracted 2 so my machine wouldn't have a stroke
# lscpu | egrep 'CPU(s)'
# make -j <cpus you are comfortable with>
make
## Super important to run altinstall to not overwrite
sudo make altinstall
## I updated the pyproject.toml to ^3.9
poetry env use /usr/local/bin/python3.9
Testing this out now, if I don’t post anything after this, it either worked or I am not longer living.
Just edit pyproject.toml
file and change the version of python to 3.7 as in :
python = "^3.7"
According to poetry’s docs, the proper way to setup a new project is with poetry new poetry-demo
, however this creates a project based on the now deprecated python2.7 by creating the following toml file:
[tool.poetry]
name = "poetry-demo"
version = "0.1.0"
description = ""
authors = ["Harsha Goli <[email protected]>"]
[tool.poetry.dependencies]
python = "^2.7"
[tool.poetry.dev-dependencies]
pytest = "^4.6"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
How can I update this to 3.7? Simply changing python = "^2.7"
to python = "^3.7"
results in the following error when poetry install
is run:
[SolverProblemError]
The current project's Python requirement (2.7.17) is not compatible with some of the required packages Python requirement:
- zipp requires Python >=3.6
Because no versions of pytest match >=4.6,<4.6.9 || >4.6.9,<5.0
and pytest (4.6.9) depends on importlib-metadata (>=0.12), pytest (>=4.6,<5.0) requires importlib-metadata (>=0.12).
And because no versions of importlib-metadata match >=0.12,<1.5.0 || >1.5.0
and importlib-metadata (1.5.0) depends on zipp (>=0.5), pytest (>=4.6,<5.0) requires zipp (>=0.5).
Because zipp (3.1.0) requires Python >=3.6
and no versions of zipp match >=0.5,<3.1.0 || >3.1.0, zipp is forbidden.
Thus, pytest is forbidden.
So, because poetry-demo depends on pytest (^4.6), version solving failed.
Interestingly, poetry is silently failing due to a missing package the tool itself relies on and continues to install a broken venv. Here’s how you fix it.
sudo apt install python3-venv
poetry env remove python3
poetry install
I had to remove pytest, and then reinstall with poetry add pytest
.
EDIT: I ran into this issue again when upgrading a project from python3.7 to python3.8 – for this instead of installing python3-venv
, you’d want to install python3.8-venv
instead
Whenever you change dependencies by hand in your pyproject.toml
you have to take care of these points:
-
Run
poetry lock --no-update
afterwards. The reasons for this is, thatpoetry install
takes thepoetry.lock
as input if can find one and not thepyproject.toml
. -
If you change the python version and uses in-project virtualenv, remove the
.venv
before runningpoetry install
. poetry doesn’t change the python version of a venv once it is created, because it uses the python version itself to create the virtualenv.
You can change in pyproject.toml and execute de this command "poetry env use 3.x" that works for me.
I had the same problem. I solve it by fixing the first line in the file /home/nordman/.poetry/bin/poetry
(nordman
is my local name).
Just change #!/usr/bin/env python
to #!/usr/bin/env python3
Poetry makes it super easy to work with different Python versions or virtual environments. The recommended way to specify your Python version according to Poetry docs is
poetry env use /path/to/preferred/python/version
You can get the path to your Python version by running
which python3.7
on Linux or
py -0p
on Windows.
I’ve experienced the exact same behavior on my MacBook! So to make this practical, first, let’s see, what is the default Python for the MacBook. I’m running macOS Big Sur 11.2.1 (20D74). In terminal:
python --version
Python 2.7.16
With that, let’s install poetry
. I’ve used poetry
installation script from the GitHub. Using this script is a recommended method to install poetry
(though I’d argue this is NOT the best idea to pipe scripts from the Internet into the python
interpreter)
The installation script is pretty smart: it attempts to detect which python
is available on the system:
def _which_python(self):
"""Decides which python executable we'll embed in the launcher script."""
allowed_executables = ["python", "python3"]
if WINDOWS:
allowed_executables += ["py.exe -3", "py.exe -2"]
...
So as the default Python is 2.7.16, and that is what returned by calling python
, the entry in ~/.poetry/bin
will look like:
#!/usr/bin/env python
And here you go! The default python
will end up in the pyproject.toml
file and you’ll need to do an extra dance to a) make sure python3
is a dependency for your project; and b) that the virtual environment will use python3
as python
interpreter.
As @mfalade mentioned, you can set environment with poetry env
:
poetry env use /path/to/python3
And also modifying the pyproject.toml
with:
...
[tool.poetry.dependencies]
python = "^3.9"
...
From this point you are good to go, poetry
will use python3
to create virtual environment for you project, dependency is specified in the file and … the grass is green again.
But can we use python3
by default? There is an insightful thread on GitHub and another one. And if you look back at the code snippet from the installation script above, you might wonder why not to check for python3
first and then for python, especially taking into account that this is the exact order for Windows installation. Well, you are not alone, I also wonder 😉
I would not suggest editing the ~/.poetry/bin/poetry
file directly as this file is generated by the installation script (so what will happen if you run installation script again? And again?).
Really, this is a minor annoyance, and knowing the tool it is easy to work around it. I’d expect that to be mentioned in installation guide though…
Anyway, hope it helps!
You can simply use pyenv
for that. Create .python-version
file inside your project and poetry
will match the exact python version.
# check current python version (set up globally)
❯ pyenv version
3.9.0 (set by /Users/[email protected]/.python-version)
# create .python-version file for project
❯ pyenv local 3.9.0
# check python version again (now it's set up locally)
❯ pyenv version
3.9.0 (set by /Users/[email protected]/Documents/myproject/.python-version)
❯ poetry lock
(...)
❯ poetry run python --version
Python 3.9.0
Use:
poetry env use 3.9
or
poetry env use $(which python3.9)
doesn’t work for some reason for python 3.9. Though having explicitly stated version requirement in pyproject.toml
file, it’s sad that one needs to manually enforce this.
The way I do it, and it’s super effective:
Use pyenv to manage the pyenv versions, and it’s plugin pyenv-virtualenv to manage the versions as virtual environments, then poetry to pick the version from them automatically as it manages the dependencies:
Use pyenv to install whatever version of python you want to use:
pyenv install -v 3.7
Create a virtual environment based on the installed version 3.7.
You will need the plugin pyenv-virtualenv.
pyenv virtualenv 3.7 arshbot_3.7_virtualenv
Assume a project called arshbot_proj. Working in $HOME directory:
mkdir ~/arshbot_proj
cd ~/arshbot_proj
Use the virtual env we just created. Attach this project to it.
Below creates .python-version file indicating arshbot_3.7_virtualenv.
pyenv local arshbot_3.7_virtualenv
Use poetry init to create the pyproject.toml inside the dir using
arshbot_3.7_virtualenv.
You could also use poetry new instead of poetry init to create project
structure alongside pyproject.toml instead of making your project first in
above step.
With pyproject.toml and .python-version both inside this dir, poetry
will automatically pick the 3.7 courtesy of arshbot_3.7_virtualenv.
Poetry will also use this virtual env to install packages at
~/.pyenv/versions/3.7/envs/arshbot_3.7_virtualenv/lib/python3.7/site-packages
poetry init
That is it. Poetry will pick the 3.7 automatically every time you run it from inside that directory attached to that virtual environment, with it activated. For different python version, just repeat the above steps, replace 3.7 with the new version.
The virtual environment will appear twice: In the envs directory as a virtual environment, and also in the versions directory as a version, with contents replicating when poetry installs packages.
BONUS:
We try to make it even more clear using a different approach, same concept, same results.
Since by default, if you didn’t change PYENV_ROOT, pyenv installs every python version in ~/.pyenv/versions:
If you want the system version(shipped with your distro) to be part of the versions you can select, use venv to mimick a pyenv installation of a python version, now call it system_ver in place of the 3.7. Since it’s already in the system, we don’t need pyenv to download it, we copy it over to our versions directory, so that it’s available to create a virtual environment
cd ~/.pyenv/versions
python3 -m venv --copies system system_ver
pyenv virtualenv system_ver system_ver_virtualenv
To use it in your project in place of previous version 3.7:
cd ~/arshbot_proj
pyenv local system_ver_virtualenv
Poetry will now use whatever version came originally with your distro. The –copies will ensure venv copies the files instead of use links, so you may omit it. Usually useful if you need to later make a multi stage dockerfile for the project using the files from the virtual environment.
I tried every solution I could find, but could not get poetry to pick up the upgraded version of Python. Finally found something that worked.
tldr;
- Run
poetry config virtualenvs.in-project true
to use local venvs - Run
deactivate
to exit any existing venv - Run
poetry shell
to activate a new, local venv
So here’s what I was seeing:
(my-project) ~/I/my-project ❯❯❯ poetry env info
Virtualenv
Python: 3.9.6
Implementation: CPython
Path: /Users/my-user/.local/share/virtualenvs/my-project-some-hash
Valid: True
System
Platform: darwin
OS: posix
Python: /usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.9
I tried running poetry env remove python
and got an error:
(my-project) ~/I/my-project ❯❯❯ poetry env remove python
ValueError
Environment "my-project-some-hash-py3.9" does not exist.
Meanwhile, I saw somewhere that it’s recommend to use local virtual environments by setting this property, so I did it:
poetry config virtualenvs.in-project true
This did not solve my problem, but then I realized that changing this setting would not take affect automatically because I was already in another non-local virtual environment (see poetry env info
output above).
So based on the docs, I ran deactivate
to deactivate the current virtual environment.
Now, I saw this:
~/I/my-project ❯❯❯ poetry env info
Virtualenv
Python: 3.10.1
Implementation: CPython
Path: NA
System
Platform: darwin
OS: posix
Python: /Users/my-user/.pyenv/versions/3.10.1
Now I ran poetry shell
to create a new virtual environment and it picked up the new local venvs setting:
~/I/my-project ❯❯❯ poetry shell
Creating virtualenv my-project in /Users/my-user/projects/my-project/.venv
Spawning shell within /Users/my-user/projects/my-project/.venv
And finally, I saw the upgraded Python version I was expecting!
(.venv) ~/I/my-project ❯❯❯ poetry run python -V
Python 3.10.1
Update: After I did all of the above, I noticed that poetry env use
works fine!
(.venv) ~/I/my-project ❯❯❯ python -V
Python 3.10.1
(.venv) ~/I/my-project ❯❯❯ poetry env use 3.9.6
Recreating virtualenv my-project in /Users/my-user/projects/my-project/.venv
Using virtualenv: /Users/my-user/projects/my-project/.venv
(.venv) ~/I/my-project ❯❯❯ python -V
Python 3.9.6
I am running ubuntu 22.04 and only have python 3.10 installed. I’ve destroyed my system python enough times that I refuse to dance with older versions, or running any installs from pip. What I wanted is an older version of python, that was installed by not installed over my current python install. What I did was follow this guide.
sudo apt install libgdbm-dev build-essential libnss3-dev libreadline-dev libffi-dev libsqlite3-dev libbz2-dev libncurses5-dev libssl-dev zlib1g-dev
cd /tmp
wget https://www.python.org/ftp/python/3.9.14/Python-3.9.14.tgz
tar -xf Python-3.9.14.tgz
cd Python-3.9.14
./configure --enable-optimizations
## if you want to run in parallel set -j to how many cpus you have
## I subtracted 2 so my machine wouldn't have a stroke
# lscpu | egrep 'CPU(s)'
# make -j <cpus you are comfortable with>
make
## Super important to run altinstall to not overwrite
sudo make altinstall
## I updated the pyproject.toml to ^3.9
poetry env use /usr/local/bin/python3.9
Testing this out now, if I don’t post anything after this, it either worked or I am not longer living.
Just edit pyproject.toml
file and change the version of python to 3.7 as in :
python = "^3.7"