SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[Tokenization] fix edge case for bert tokenization (#3517)

Browse Source 
* fix egde gase for bert tokenization

* add Lysandres comments for improvement

* use new is_pretokenized_flag
This commit is contained in:
Patrick von Platen
2020-04-07 22:26:31 +02:00
committed by GitHub
parent 80fa0f7812
commit b0ad069517
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/tokenization_utils.py
View File

@@ -1396,7 +1396,7 @@ class PreTrainedTokenizer(SpecialTokensMixin):
input_ids = []
for ids_or_pair_ids in batch_text_or_text_pairs:
if isinstance(ids_or_pair_ids, (list, tuple)) and len(ids_or_pair_ids) == 2:
if isinstance(ids_or_pair_ids, (list, tuple)) and len(ids_or_pair_ids) == 2 and not is_pretokenized:
ids, pair_ids = ids_or_pair_ids
else:
ids, pair_ids = ids_or_pair_ids, None