[Tokenization] Fix #5181 - make #5155 more explicit - move back the default logging level in tests to WARNING (#5252)
* fix-5181 Padding to max sequence length while truncation to another length was wrong on slow tokenizers * clean up and fix #5155 * fix XLM test * Fix tests for Transfo-XL * logging only above WARNING in tests * switch slow tokenizers tests in @slow * fix Marian truncation tokenization test * style and quality * make the test a lot faster by limiting the sequence length used in tests
This commit is contained in:
@@ -1890,7 +1890,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin):
|
||||
if return_attention_mask is None:
|
||||
return_attention_mask = "attention_mask" in self.model_input_names
|
||||
|
||||
if padding_strategy == PaddingStrategy.LONGEST and max_length is None:
|
||||
if padding_strategy == PaddingStrategy.LONGEST:
|
||||
max_length = len(encoded_inputs["input_ids"])
|
||||
|
||||
needs_to_be_padded = (
|
||||
|
||||
Reference in New Issue
Block a user