SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Fix incorrect model<->tokenizer mapping in tokenization testing (#19872)

Browse Source 
* Fix model-tokenizer mapping

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
This commit is contained in:
Yih-Dar
2022-10-25 16:02:13 +02:00
committed by GitHub
parent eedaba682f
commit f9257843b5
1 changed files with 5 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
tests/test_tokenization_common.py
View File

@@ -120,9 +120,12 @@ def merge_model_tokenizer_mappings(
tokenizer = tokenizer_mapping[configuration][0]
tokenizer_fast = tokenizer_mapping[configuration][1]
model_tokenizer_mapping.update({tokenizer: (configuration, model)})
if tokenizer is not None:
if configuration.__name__.startswith(tokenizer.__name__.replace("Tokenizer", "")):
model_tokenizer_mapping.update({tokenizer: (configuration, model)})
if tokenizer_fast is not None:
model_tokenizer_mapping.update({tokenizer_fast: (configuration, model)})
if configuration.__name__.startswith(tokenizer_fast.__name__.replace("TokenizerFast", "")):
model_tokenizer_mapping.update({tokenizer_fast: (configuration, model)})
return model_tokenizer_mapping