SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Example of pad_to_multiple_of for padding and truncation guide & docstring update (#22278)

Browse Source 
* added an example of pad_to_multiple_of

* make style

* addressed feedback
This commit is contained in:
Maria Khalusova
2023-03-20 14:18:55 -04:00
committed by GitHub
parent fb0a38b4f2
commit 7bd8650512
2 changed files with 4 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
src/transformers/tokenization_utils_base.py
View File

@@ -1342,8 +1342,9 @@ ENCODE_KWARGS_DOCSTRING = r"""
tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)
which it will tokenize. This is useful for NER or token classification.
pad_to_multiple_of (`int`, *optional*):
If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta).
If set will pad the sequence to a multiple of the provided value. Requires `padding` to be activated.
This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
`>= 7.5` (Volta).
return_tensors (`str` or [`~utils.TensorType`], *optional*):
If set, will return tensors instead of list of python integers. Acceptable values are: