SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Warn for unexpected argument combinations (#13509)

Browse Source 
* Warn for unexpected argument combinations

* Updated the waning message for pad_to_max_length
This commit is contained in:
Yuta Hayashibe
2021-09-24 22:14:23 +09:00
committed by GitHub
parent e579f855fa
commit 66b01ce864
1 changed files with 5 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
src/transformers/tokenization_utils_base.py
View File

@@ -2222,6 +2222,11 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
padding_strategy = PaddingStrategy.MAX_LENGTH
elif padding is not False:
if padding is True:
if verbose:
if max_length is not None:
warnings.warn("`max_length` is ignored when `padding`=`True`.")
if old_pad_to_max_length is not False:
warnings.warn("Though `pad_to_max_length` = `True`, it is ignored because `padding`=`True`.")
padding_strategy = PaddingStrategy.LONGEST # Default to pad to the longest sequence in the batch
elif not isinstance(padding, PaddingStrategy):
padding_strategy = PaddingStrategy(padding)