SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Some model_types cannot be in the mapping (#13259)

Browse Source 
* Some tokenizers cannot be in the mapping

* Style
This commit is contained in:
Lysandre Debut
2021-08-25 18:56:16 +02:00
committed by GitHub
parent 68b6907290
commit 5af8df5afb
1 changed files with 3 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
src/transformers/tokenization_utils_base.py
View File

@@ -1799,7 +1799,9 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
break break
if model_type is not None: if model_type is not None:
config_tokenizer_class, config_tokenizer_class_fast = TOKENIZER_MAPPING_NAMES[model_type] config_tokenizer_class, config_tokenizer_class_fast = TOKENIZER_MAPPING_NAMES.get(
model_type, (None, None)
)
if config_tokenizer_class is None: if config_tokenizer_class is None:
config_tokenizer_class = config_tokenizer_class_fast config_tokenizer_class = config_tokenizer_class_fast