Github Actions: poetry installs black but CI workflow does not find it
Question:
I am setting up a python code quality workflow locally (pre-commit) and on Github Actions (GHA). Environment is managed with poetry.
While the local precommit works fine, the remote GHA workflow fails, saying it does not find black, while looking at the workflow logs it seems it was installed just fine. Workflow was largely copied from this great writeup: https://jacobian.org/til/github-actions-poetry/
Where am I making a mistake? Here are the relevant files:
codequal.yml
name: Python QA
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
permissions:
contents: read
jobs:
pylint:
runs-on: ubuntu-latest
name: Python QA
steps:
- name: Check out
uses: actions/checkout@v3
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: "3.10"
# Cache the installation of Poetry itself, e.g. the next step. This prevents the workflow
# from installing Poetry every time, which can be slow. Note the use of the Poetry version
# number in the cache key, and the "-0" suffix: this allows you to invalidate the cache
# manually if/when you want to upgrade Poetry, or if something goes wrong. This could be
# mildly cleaner by using an environment variable, but I don't really care.
- name: cache poetry install
uses: actions/cache@v3
with:
path: ~/.local
key: poetry-1.3.2
# Install Poetry. You could do this manually, or there are several actions that do this.
# `snok/install-poetry` seems to be minimal yet complete, and really just calls out to
# Poetry's default install script, which feels correct. I pin the Poetry version here
# because Poetry does occasionally change APIs between versions and I don't want my
# actions to break if it does.
#
# The key configuration value here is `virtualenvs-in-project: true`: this creates the
# venv as a `.venv` in your testing directory, which allows the next step to easily
# cache it.
- uses: snok/[email protected]
with:
version: 1.3.2
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true
# Cache your dependencies (i.e. all the stuff in your `pyproject.toml`). Note the cache
# key: if you're using multiple Python versions, or multiple OSes, you'd need to include
# them in the cache key. I'm not, so it can be simple and just depend on the poetry.lock.
- name: cache deps
id: cache-deps
uses: actions/cache@v3
with:
path: .venv
key: pydeps-${{ hashFiles('**/poetry.lock') }}
# Install dependencies. `--no-root` means "install all dependencies but not the project
# itself", which is what you want to avoid caching _your_ code. The `if` statement
# ensures this only runs on a cache miss.
- run: poetry install --no-interaction --no-root
if: steps.cache-deps.outputs.cache-hit != 'true'
# Now install _your_ project. This isn't necessary for many types of projects -- particularly
# things like Django apps don't need this. But it's a good idea since it fully-exercises the
# pyproject.toml and makes that if you add things like console-scripts at some point that
# they'll be installed and working.
- run: poetry install --no-interaction
################################################################
# Now finally run your code quality tools
################################################################
- name: Format with black
run: |
black 'src'
- name: Lint with pylint
run: |
pylint --fail-under=7.0 --recursive=y --enable=W 'src'
Relevant section of GitHub Action logging of codequal.yml
Run poetry install --no-interaction --no-root
Creating virtualenv rssita in /home/runner/work/rssita/rssita/.venv
Installing dependencies from lock file
Package operations: 49 installs, 1 update, 0 removals
• Updating setuptools (65.6.3 -> 67.0.0)
• Installing attrs (22.2.0)
• Installing certifi (2022.12.7)
• Installing charset-normalizer (3.0.1)
• Installing distlib (0.3.6)
• Installing exceptiongroup (1.1.0)
• Installing filelock (3.9.0)
• Installing idna (3.4)
• Installing iniconfig (2.0.0)
• Installing nvidia-cublas-cu11 (11.10.3.66)
• Installing nvidia-cuda-nvrtc-cu11 (11.7.99)
• Installing nvidia-cuda-runtime-cu11 (11.7.99)
• Installing nvidia-cudnn-cu11 (8.5.0.96)
• Installing packaging (23.0)
• Installing platformdirs (2.6.2)
• Installing pluggy (1.0.0)
• Installing tomli (2.0.1)
• Installing typing-extensions (4.4.0)
• Installing urllib3 (1.26.14)
• Installing cfgv (3.3.1)
• Installing click (8.1.3)
• Installing coverage (7.1.0)
• Installing identify (2.5.17)
• Installing emoji (2.2.0)
• Installing mccabe (0.7.0)
• Installing mypy-extensions (0.4.3)
• Installing nodeenv (1.7.0)
• Installing numpy (1.24.1)
• Installing pathspec (0.11.0)
• Installing protobuf (4.21.12)
• Installing pycodestyle (2.10.0)
• Installing pyflakes (3.0.1)
• Installing pytest (7.2.1)
• Installing pyyaml (6.0)
• Installing requests (2.28.2)
• Installing sgmllib3k (1.0.0)
• Installing six (1.16.0)
• Installing torch (1.13.1)
• Installing tqdm (4.64.1)
• Installing types-urllib3 (1.26.25.4)
• Installing virtualenv (20.17.1)
• Installing black (22.12.0)
• Installing feedparser (6.0.10)
• Installing flake8 (6.0.0)
• Installing isort (5.12.0)
• Installing mypy (0.991)
• Installing pre-commit (3.0.2)
• Installing pytest-cov (4.0.0)
• Installing stanza (1.4.2)
• Installing types-requests (2.28.11.8)
1s
Run poetry install --no-interaction
Installing dependencies from lock file
No dependencies to install or update
Installing the current project: rssita (0.1.0)
0s
Run black 'src'
/home/runner/work/_temp/0fc25aa8-4903-45ae-9d8e-9c11f60dca11.sh: line 1: black: command not found
Error: Process completed with exit code 127.
Project tree on local:
(base) bob@Roberts-Mac-mini rssita % tree
.
├── README.md
├── __pycache__
│ └── test_feeds.cpython-310-pytest-7.2.1.pyc
├── poetry.lock
├── pyproject.toml
├── setup.cfg
├── src
│ └── rssita
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-310-pytest-7.2.1.pyc
│ │ ├── __init__.cpython-310.pyc
│ │ ├── feeds.cpython-310-pytest-7.2.1.pyc
│ │ ├── feeds.cpython-310.pyc
│ │ ├── rssita.cpython-310-pytest-7.2.1.pyc
│ │ └── termcolors.cpython-310-pytest-7.2.1.pyc
│ ├── feeds.py
│ ├── rssita.py
│ └── termcolors.py
└── tests
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-310.pyc
│ └── test_feeds.cpython-310-pytest-7.2.1.pyc
└── test_feeds.py
Answers:
Since the dependencies are installed in a virtual environment managed by Poetry, you need to use Poetry to run Black:
- name: Format with black
run: poetry run black 'src'
I am setting up a python code quality workflow locally (pre-commit) and on Github Actions (GHA). Environment is managed with poetry.
While the local precommit works fine, the remote GHA workflow fails, saying it does not find black, while looking at the workflow logs it seems it was installed just fine. Workflow was largely copied from this great writeup: https://jacobian.org/til/github-actions-poetry/
Where am I making a mistake? Here are the relevant files:
codequal.yml
name: Python QA
on:
push:
branches: [ "main" ]
pull_request:
branches: [ "main" ]
permissions:
contents: read
jobs:
pylint:
runs-on: ubuntu-latest
name: Python QA
steps:
- name: Check out
uses: actions/checkout@v3
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: "3.10"
# Cache the installation of Poetry itself, e.g. the next step. This prevents the workflow
# from installing Poetry every time, which can be slow. Note the use of the Poetry version
# number in the cache key, and the "-0" suffix: this allows you to invalidate the cache
# manually if/when you want to upgrade Poetry, or if something goes wrong. This could be
# mildly cleaner by using an environment variable, but I don't really care.
- name: cache poetry install
uses: actions/cache@v3
with:
path: ~/.local
key: poetry-1.3.2
# Install Poetry. You could do this manually, or there are several actions that do this.
# `snok/install-poetry` seems to be minimal yet complete, and really just calls out to
# Poetry's default install script, which feels correct. I pin the Poetry version here
# because Poetry does occasionally change APIs between versions and I don't want my
# actions to break if it does.
#
# The key configuration value here is `virtualenvs-in-project: true`: this creates the
# venv as a `.venv` in your testing directory, which allows the next step to easily
# cache it.
- uses: snok/[email protected]
with:
version: 1.3.2
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true
# Cache your dependencies (i.e. all the stuff in your `pyproject.toml`). Note the cache
# key: if you're using multiple Python versions, or multiple OSes, you'd need to include
# them in the cache key. I'm not, so it can be simple and just depend on the poetry.lock.
- name: cache deps
id: cache-deps
uses: actions/cache@v3
with:
path: .venv
key: pydeps-${{ hashFiles('**/poetry.lock') }}
# Install dependencies. `--no-root` means "install all dependencies but not the project
# itself", which is what you want to avoid caching _your_ code. The `if` statement
# ensures this only runs on a cache miss.
- run: poetry install --no-interaction --no-root
if: steps.cache-deps.outputs.cache-hit != 'true'
# Now install _your_ project. This isn't necessary for many types of projects -- particularly
# things like Django apps don't need this. But it's a good idea since it fully-exercises the
# pyproject.toml and makes that if you add things like console-scripts at some point that
# they'll be installed and working.
- run: poetry install --no-interaction
################################################################
# Now finally run your code quality tools
################################################################
- name: Format with black
run: |
black 'src'
- name: Lint with pylint
run: |
pylint --fail-under=7.0 --recursive=y --enable=W 'src'
Relevant section of GitHub Action logging of codequal.yml
Run poetry install --no-interaction --no-root
Creating virtualenv rssita in /home/runner/work/rssita/rssita/.venv
Installing dependencies from lock file
Package operations: 49 installs, 1 update, 0 removals
• Updating setuptools (65.6.3 -> 67.0.0)
• Installing attrs (22.2.0)
• Installing certifi (2022.12.7)
• Installing charset-normalizer (3.0.1)
• Installing distlib (0.3.6)
• Installing exceptiongroup (1.1.0)
• Installing filelock (3.9.0)
• Installing idna (3.4)
• Installing iniconfig (2.0.0)
• Installing nvidia-cublas-cu11 (11.10.3.66)
• Installing nvidia-cuda-nvrtc-cu11 (11.7.99)
• Installing nvidia-cuda-runtime-cu11 (11.7.99)
• Installing nvidia-cudnn-cu11 (8.5.0.96)
• Installing packaging (23.0)
• Installing platformdirs (2.6.2)
• Installing pluggy (1.0.0)
• Installing tomli (2.0.1)
• Installing typing-extensions (4.4.0)
• Installing urllib3 (1.26.14)
• Installing cfgv (3.3.1)
• Installing click (8.1.3)
• Installing coverage (7.1.0)
• Installing identify (2.5.17)
• Installing emoji (2.2.0)
• Installing mccabe (0.7.0)
• Installing mypy-extensions (0.4.3)
• Installing nodeenv (1.7.0)
• Installing numpy (1.24.1)
• Installing pathspec (0.11.0)
• Installing protobuf (4.21.12)
• Installing pycodestyle (2.10.0)
• Installing pyflakes (3.0.1)
• Installing pytest (7.2.1)
• Installing pyyaml (6.0)
• Installing requests (2.28.2)
• Installing sgmllib3k (1.0.0)
• Installing six (1.16.0)
• Installing torch (1.13.1)
• Installing tqdm (4.64.1)
• Installing types-urllib3 (1.26.25.4)
• Installing virtualenv (20.17.1)
• Installing black (22.12.0)
• Installing feedparser (6.0.10)
• Installing flake8 (6.0.0)
• Installing isort (5.12.0)
• Installing mypy (0.991)
• Installing pre-commit (3.0.2)
• Installing pytest-cov (4.0.0)
• Installing stanza (1.4.2)
• Installing types-requests (2.28.11.8)
1s
Run poetry install --no-interaction
Installing dependencies from lock file
No dependencies to install or update
Installing the current project: rssita (0.1.0)
0s
Run black 'src'
/home/runner/work/_temp/0fc25aa8-4903-45ae-9d8e-9c11f60dca11.sh: line 1: black: command not found
Error: Process completed with exit code 127.
Project tree on local:
(base) bob@Roberts-Mac-mini rssita % tree
.
├── README.md
├── __pycache__
│ └── test_feeds.cpython-310-pytest-7.2.1.pyc
├── poetry.lock
├── pyproject.toml
├── setup.cfg
├── src
│ └── rssita
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-310-pytest-7.2.1.pyc
│ │ ├── __init__.cpython-310.pyc
│ │ ├── feeds.cpython-310-pytest-7.2.1.pyc
│ │ ├── feeds.cpython-310.pyc
│ │ ├── rssita.cpython-310-pytest-7.2.1.pyc
│ │ └── termcolors.cpython-310-pytest-7.2.1.pyc
│ ├── feeds.py
│ ├── rssita.py
│ └── termcolors.py
└── tests
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-310.pyc
│ └── test_feeds.cpython-310-pytest-7.2.1.pyc
└── test_feeds.py
Since the dependencies are installed in a virtual environment managed by Poetry, you need to use Poetry to run Black:
- name: Format with black
run: poetry run black 'src'