SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[tokenizers] update tokenizers version pin (#27494)

Browse Source 
* update `tokenizers` version pin

* force tokenizers>=0.15

* use  0.14

Co-authored-by: Lysandre <lysandre@huggingface.co>

---------

Co-authored-by: Lysandre <lysandre@huggingface.co>
This commit is contained in:
Arthur
2023-11-15 10:46:02 +01:00
committed by ArthurZucker
parent 10f3e7b31b
commit 7983bca630
2 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
setup.py
View File

@@ -175,7 +175,7 @@ _deps = [
"tf2onnx",
"timeout-decorator",
"timm",
"tokenizers>=0.14,<0.15",
"tokenizers>=0.14,<0.19",
"torch>=1.10,!=1.12.0",
"torchaudio",
"torchvision",

2
src/transformers/dependency_versions_table.py
View File

@@ -80,7 +80,7 @@ deps = {
"tf2onnx": "tf2onnx",
"timeout-decorator": "timeout-decorator",
"timm": "timm",
"tokenizers": "tokenizers>=0.14,<0.15",
"tokenizers": "tokenizers>=0.14,<0.19",
"torch": "torch>=1.10,!=1.12.0",
"torchaudio": "torchaudio",
"torchvision": "torchvision",