SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Fix token pass (#24862)

Browse Source 
* Fix how token is passed along in from_pretrained for tokenizers

* It's actually not necessary
This commit is contained in:
Sylvain Gugger
2023-07-17 13:27:11 -04:00
committed by GitHub
parent f42a35e611
commit 49eb357564
1 changed files with 0 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
src/transformers/tokenization_utils_base.py
View File

@@ -1720,10 +1720,6 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
) )
token = use_auth_token token = use_auth_token
if token is not None:
# change to `token` in a follow-up PR
kwargs["use_auth_token"] = token
user_agent = {"file_type": "tokenizer", "from_auto_class": from_auto_class, "is_fast": "Fast" in cls.__name__} user_agent = {"file_type": "tokenizer", "from_auto_class": from_auto_class, "is_fast": "Fast" in cls.__name__}
if from_pipeline is not None: if from_pipeline is not None:
user_agent["using_pipeline"] = from_pipeline user_agent["using_pipeline"] = from_pipeline