SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

More explicit error when failing to tensorize overflowing tokens (#5633)

Browse Source
This commit is contained in:
Lysandre Debut
2020-07-09 13:35:21 -04:00
committed by GitHub
parent b9d8af07e6
commit 3cc23eee06
1 changed files with 5 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
src/transformers/tokenization_utils_base.py
View File

@@ -511,6 +511,11 @@ class BatchEncoding(UserDict):
self[key] = tensor self[key] = tensor
except: # noqa E722 except: # noqa E722
if key == "overflowing_tokens":
raise ValueError(
"Unable to create tensor returning overflowing tokens of different lengths. "
"Please see if a fast version of this tokenizer is available to have this feature available."
)
raise ValueError( raise ValueError(
"Unable to create tensor, you should probably activate truncation and/or padding " "Unable to create tensor, you should probably activate truncation and/or padding "
"with 'padding=True' 'truncation=True' to have batched tensors with the same length." "with 'padding=True' 'truncation=True' to have batched tensors with the same length."