Error in pip install transformers: Building wheel for tokenizers (pyproject.toml): finished with status 'error'

Question:

I’m building a docker image on cloud server via the following docker file:

# base image
FROM python:3

# add python file to working directory
ADD ./ /

# install and cache dependencies
RUN pip install --upgrade pip
RUN pip install RUST
RUN pip install transformers
RUN pip install torch
RUN pip install slack_sdk
RUN pip install slack_bolt
RUN pip install pandas
RUN pip install gensim
RUN pip install nltk
RUN pip install psycopg2
RUN pip install openpyxl
......

When installing the transformers package, the following error occurs:

STEP 5: RUN pip install transformers
Collecting transformers
  Downloading transformers-4.15.0-py3-none-any.whl (3.4 MB)
Collecting filelock
  ......
  Downloading click-8.0.3-py3-none-any.whl (97 kB)
Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (pyproject.toml): started
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp_3y7hw5q
       cwd: /tmp/pip-install-bsy5f4da/tokenizers_e09b9f903acd40f0af4a997fe1d8fdb4
  Complete output (50 lines):
  running bdist_wheel
  ......
  copying py_src/tokenizers/trainers/__init__.pyi -> build/lib.linux-x86_64-3.10/tokenizers/trainers
  copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.linux-x86_64-3.10/tokenizers/tools
  running build_ext
  error: can't find Rust compiler
  
  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
  
  To update pip, run:
  
      pip install --upgrade pip
  
  and then retry package installation.
  
  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
  Building wheel for tokenizers (pyproject.toml): finished with status 'error'
Failed to build tokenizers
subprocess exited with status 1
subprocess exited with status 1
error building at STEP "RUN pip install transformers": exit status 1
time="2022-01-18T07:24:56Z" level=error msg="exit status 1"
Dockerfile build failed - exit status 1exit status 1

I’m not very sure about what’s happening here. Can anyone help me? Thanks in advance.

Asked By: Rachel Dong

||

Answers:

The logs say

error: can't find Rust compiler

You need to install a rust compiler. See https://www.rust-lang.org/tools/install. You can modify the installation instructions for a docker image like this (from https://stackoverflow.com/a/58169817/5666087):

RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
Answered By: jkr

Just use a clean python=3.8 environment and try again.

Answered By: namespace-Pt
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.