From 5a5af22ed540c14b241fdec8b1c2202745ec7d24 Mon Sep 17 00:00:00 2001 From: Romain Rigaux Date: Tue, 18 Aug 2020 07:22:43 -0700 Subject: [PATCH] [docs] Fix wrong newline in the middle of a paragraph (#6573) --- docs/source/preprocessing.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/preprocessing.rst b/docs/source/preprocessing.rst index 7f0295e341..f143549087 100644 --- a/docs/source/preprocessing.rst +++ b/docs/source/preprocessing.rst @@ -53,7 +53,6 @@ The tokenizer can decode a list of token ids in a proper sentence: As you can see, the tokenizer automatically added some special tokens that the model expect. Not all model need special tokens; for instance, if we had used` gtp2-medium` instead of `bert-base-cased` to create our tokenizer, we would have - seen the same sentence as the original one here. You can disable this behavior (which is only advised if you have added those special tokens yourself) by passing ``add_special_tokens=False``.