From 147f774671c72ab24d17547030bb1f2803925d3b Mon Sep 17 00:00:00 2001 From: Arthur <48595927+ArthurZucker@users.noreply.github.com> Date: Mon, 6 Nov 2023 15:11:00 +0100 Subject: [PATCH] [`PretrainedTokenizer`] add some of the most important functions to the doc (#27313) --- docs/source/en/main_classes/tokenizer.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/source/en/main_classes/tokenizer.md b/docs/source/en/main_classes/tokenizer.md index bf43014ee5..2ad7e45040 100644 --- a/docs/source/en/main_classes/tokenizer.md +++ b/docs/source/en/main_classes/tokenizer.md @@ -55,6 +55,8 @@ to a given token). [[autodoc]] PreTrainedTokenizer - __call__ + - add_tokens + - add_special_tokens - apply_chat_template - batch_decode - decode @@ -69,6 +71,8 @@ loaded very simply into 🤗 transformers. Take a look at the [Using tokenizers [[autodoc]] PreTrainedTokenizerFast - __call__ + - add_tokens + - add_special_tokens - apply_chat_template - batch_decode - decode