bump tokenizers, fix added tokens fast (#32535)

* update based on tokenizers release * update * nits * update * revert re addition * don't break that yet * fmt * revert unwanted * update tokenizers version * update dep table * update * update in conversion script as well * some fix * revert * fully revert * fix training * remove set trace * fixup * update * update
2024-09-25 13:47:20 +02:00
parent 5e2916bc14
commit c6379858f3
4 changed files with 24 additions and 46 deletions
--- a/setup.py
+++ b/setup.py
@@ -181,7 +181,7 @@ _deps = [
    "timeout-decorator",
    "tiktoken",
    "timm<=0.9.16",
-    "tokenizers>=0.19,<0.20",
+    "tokenizers>=0.20,<0.21",
    "torch",
    "torchaudio",
    "torchvision",