SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

add max_length to showcase the use of truncation (#8975)

Browse Source
This commit is contained in:
Navjot
2020-12-07 15:35:39 -08:00
committed by GitHub
parent 62d30e0583
commit c108d0b5a4
1 changed files with 2 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/quicktour.rst
View File

@@ -182,6 +182,7 @@ and get tensors back. You can specify all of that to the tokenizer:
... ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."], ... ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
... padding=True, ... padding=True,
... truncation=True, ... truncation=True,
... max_length=512,
... return_tensors="pt" ... return_tensors="pt"
... ) ... )
>>> ## TENSORFLOW CODE >>> ## TENSORFLOW CODE
@@ -189,6 +190,7 @@ and get tensors back. You can specify all of that to the tokenizer:
... ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."], ... ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
... padding=True, ... padding=True,
... truncation=True, ... truncation=True,
... max_length=512,
... return_tensors="tf" ... return_tensors="tf"
... ) ... )