Error building "tokenizers" wheel when installing "bntransformer" via pip in Colab environment

[basirul-billah]

I encountered an error while trying to install the "bntransformer" library in a Colab environment using both commands pip install -U bntransformer and pip install -U bntransformer. The installation process failed with the following error message:

`
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting bntransformer
Using cached bntransformer-2.1.0-py3-none-any.whl (6.2 kB)
Collecting transformers==4.6.1 (from bntransformer)
Using cached transformers-4.6.1-py3-none-any.whl (2.2 MB)
Collecting sentencepiece (from bntransformer)
Using cached sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.6.1->bntransformer) (3.12.0)
Collecting huggingface-hub==0.0.8 (from transformers==4.6.1->bntransformer)
Using cached huggingface_hub-0.0.8-py3-none-any.whl (34 kB)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.6.1->bntransformer) (1.22.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from transformers==4.6.1->bntransformer) (23.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.6.1->bntransformer) (2022.10.31)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.6.1->bntransformer) (2.27.1)
Collecting sacremoses (from transformers==4.6.1->bntransformer)
Using cached sacremoses-0.0.53-py3-none-any.whl
Collecting tokenizers<0.11,>=0.10.1 (from transformers==4.6.1->bntransformer)
Using cached tokenizers-0.10.3.tar.gz (212 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.6.1->bntransformer) (4.65.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.6.1->bntransformer) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.6.1->bntransformer) (2022.12.7)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.6.1->bntransformer) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.6.1->bntransformer) (3.4)
Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from sacremoses->transformers==4.6.1->bntransformer) (1.16.0)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from sacremoses->transformers==4.6.1->bntransformer) (8.1.3)
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from sacremoses->transformers==4.6.1->bntransformer) (1.2.0)
Building wheels for collected packages: tokenizers
error: subprocess-exited-with-error

× Building wheel for tokenizers (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Building wheel for tokenizers (pyproject.toml) ... error
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
`

Steps to reproduce

• Open a new Colab Notebook.
• Write pip install -U bntransformer or pip install -U bntransformer

Additional information

• Python version: 3.10.12 (main, Jun 7 2023, 12:45:35) [GCC 9.4.0]

[sagorbrur]

Hi @basirul-billah ,
I am guessing the problem is transformers version.
I will debug it and get back to you. It will take a longer time though.
Thanks

[Reinhardt-i]

Encountered the same issue!

[sagorbrur]

Hi @Reinhardt-i ,
Problem with transformers version.
I tried with transformers-4.38.0 and it's working fine.
I will update the wheel file shortly.
meanwhile, you can clone the repository, update setup.py with transformers version 4.38.0, and install the package by pip install -e .
I am hoping it will work.