SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Upgrading to tokenizers 0.19.0 (#30289)

Browse Source 
* [DO NOT MERGE] Testing tokenizers 0.19.0rc0

* Accounting for the breaking change.

* Ruff.

* Upgrading to tokenizers `0.19` (new release with preprend_scheme fixed
and new surface for BPE tiktoken bug).
This commit is contained in:
Nicolas Patry
2024-04-17 17:17:50 +02:00
committed by GitHub
parent c15aad0939
commit 8e5f76f511
4 changed files with 26 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
setup.py
View File

@@ -174,7 +174,7 @@ _deps = [
"tf2onnx",
"timeout-decorator",
"timm",
"tokenizers>=0.14,<0.19",
"tokenizers>=0.19,<0.20",
"torch",
"torchaudio",
"torchvision",