SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Fix warning situation: UserWarning: max_length is ignored when padding=True" (#13829)

Browse Source 
* Removed wrong warning

* Raise a warning when `max_length` is given with wrong `truncation`

* Update the error message

* Update the warning message

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
This commit is contained in:
Yuta Hayashibe
2021-10-01 22:29:08 +09:00
committed by GitHub
parent 8bbb53e20b
commit 90f980ed35
1 changed files with 5 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
src/transformers/tokenization_utils_base.py
View File

@@ -2223,8 +2223,11 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
elif padding is not False: elif padding is not False:
if padding is True: if padding is True:
if verbose: if verbose:
if max_length is not None: if max_length is not None and (truncation is False or truncation == "do_not_truncate"):
warnings.warn("`max_length` is ignored when `padding`=`True`.") warnings.warn(
"`max_length` is ignored when `padding`=`True` and there is no truncation strategy. "
"To pad to max length, use `padding='max_length'`."
)
if old_pad_to_max_length is not False: if old_pad_to_max_length is not False:
warnings.warn("Though `pad_to_max_length` = `True`, it is ignored because `padding`=`True`.") warnings.warn("Though `pad_to_max_length` = `True`, it is ignored because `padding`=`True`.")
padding_strategy = PaddingStrategy.LONGEST # Default to pad to the longest sequence in the batch padding_strategy = PaddingStrategy.LONGEST # Default to pad to the longest sequence in the batch