SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Small docfile fixes (#6328)

Browse Source
This commit is contained in:
Sylvain Gugger
2020-08-10 05:37:12 -04:00
committed by GitHub
parent 1429b920d4
commit 6028ed92bd
6 changed files with 90 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/source/tokenizer_summary.rst
View File

@@ -73,7 +73,7 @@ subwords. This also enables the model to process words it has never seen before,
subwords it knows. For instance, the base :class:`~transformers.BertTokenizer` will tokenize "I have a new GPU!" like
this:
::
.. code-block::
>>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
@@ -87,7 +87,7 @@ predictions and reverse the tokenization).
Another example is when we use the base :class:`~transformers.XLNetTokenizer` to tokenize our previous text:
::
.. code-block::
>>> from transformers import XLNetTokenizer
>>> tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')