Files
HuggingFace_transformer/setup.py
Nicolas Patry 08cb5718ec Enabling tokenizers upgrade. (#14941)
* Enabling `tokenizers` upgrade.

* Moved ugly comment.

* Tokenizers==0.11.1 needs an update to keep borrow checker

happy in highly contiguous calls.

* Support both 0.11.1 and 0.11.0
2021-12-30 17:30:58 +01:00

13 KiB