Doc styler examples (#14953)
* Fix bad examples * Add black formatting to style_doc * Use first nonempty line * Put it at the right place * Don't add spaces to empty lines * Better templates * Deal with triple quotes in docstrings * Result of style_doc * Enable mdx treatment and fix code examples in MDXs * Result of doc styler on doc source files * Last fixes * Break copy from
This commit is contained in:
@@ -104,6 +104,7 @@ seen before, by decomposing them into known subwords. For instance, the [`~trans
|
||||
|
||||
```py
|
||||
>>> from transformers import BertTokenizer
|
||||
|
||||
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
|
||||
>>> tokenizer.tokenize("I have a new GPU!")
|
||||
["i", "have", "a", "new", "gp", "##u", "!"]
|
||||
@@ -117,6 +118,7 @@ As another example, [`~transformers.XLNetTokenizer`] tokenizes our previously ex
|
||||
|
||||
```py
|
||||
>>> from transformers import XLNetTokenizer
|
||||
|
||||
>>> tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
|
||||
>>> tokenizer.tokenize("Don't you love 🤗 Transformers? We sure do.")
|
||||
["▁Don", "'", "t", "▁you", "▁love", "▁", "🤗", "▁", "Transform", "ers", "?", "▁We", "▁sure", "▁do", "."]
|
||||
|
||||
Reference in New Issue
Block a user