SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Ensure fast tokenizer can construct tensor without pad token if only one sample is provided. (#4201)

Browse Source
This commit is contained in:
Funtowicz Morgan
2020-05-07 14:02:53 +00:00
committed by GitHub
parent 0a6cbea0a5
commit 026097b9ee
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/tokenization_utils.py
View File

@@ -2435,7 +2435,7 @@ class PreTrainedTokenizerFast(PreTrainedTokenizer):
) )
# Needed if we have to return a tensor # Needed if we have to return a tensor
pad_to_max_length = pad_to_max_length or (return_tensors is not None) pad_to_max_length = pad_to_max_length or (return_tensors is not None and len(batch_text_or_text_pairs) > 1)
# Throw an error if we can pad because there is no padding token # Throw an error if we can pad because there is no padding token
if pad_to_max_length and self.pad_token_id is None: if pad_to_max_length and self.pad_token_id is None: