From ebbe8cc3fe7a2553e924353ab454bd026fd23135 Mon Sep 17 00:00:00 2001 From: Xing Han Lu Date: Sun, 28 Nov 2021 18:55:38 -0500 Subject: [PATCH] Tokenizers docs: Specify which class contains `__call__` method (#14379) * Update tokenizer.rst * Apply `make fixup` --- docs/source/main_classes/tokenizer.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/main_classes/tokenizer.rst b/docs/source/main_classes/tokenizer.rst index 8ef1ac56ba..18798e9b49 100644 --- a/docs/source/main_classes/tokenizer.rst +++ b/docs/source/main_classes/tokenizer.rst @@ -39,7 +39,8 @@ methods for using all the tokenizers: - Managing special tokens (like mask, beginning-of-sentence, etc.): adding them, assigning them to attributes in the tokenizer for easy access and making sure they are not split during tokenization. -:class:`~transformers.BatchEncoding` holds the output of the tokenizer's encoding methods (``__call__``, +:class:`~transformers.BatchEncoding` holds the output of the +:class:`~transformers.tokenization_utils_base.PreTrainedTokenizerBase`'s encoding methods (``__call__``, ``encode_plus`` and ``batch_encode_plus``) and is derived from a Python dictionary. When the tokenizer is a pure python tokenizer, this class behaves just like a standard python dictionary and holds the various model inputs computed by these methods (``input_ids``, ``attention_mask``...). When the tokenizer is a "Fast" tokenizer (i.e., backed by