Doc styling (#8067)
* Important files * Styling them all * Revert "Styling them all" This reverts commit 7d029395fdae8513b8281cbc2a6c239f8093503e. * Syling them for realsies * Fix syntax error * Fix benchmark_utils * More fixes * Fix modeling auto and script * Remove new line * Fixes * More fixes * Fix more files * Style * Add FSMT * More fixes * More fixes * More fixes * More fixes * Fixes * More fixes * More fixes * Last fixes * Make sphinx happy
This commit is contained in:
@@ -29,11 +29,12 @@ methods for using all the tokenizers:
|
||||
|
||||
:class:`~transformers.BatchEncoding` holds the output of the tokenizer's encoding methods (``__call__``,
|
||||
``encode_plus`` and ``batch_encode_plus``) and is derived from a Python dictionary. When the tokenizer is a pure python
|
||||
tokenizer, this class behaves just like a standard python dictionary and holds the various model inputs computed by these
|
||||
methods (``input_ids``, ``attention_mask``...). When the tokenizer is a "Fast" tokenizer (i.e., backed by HuggingFace
|
||||
`tokenizers library <https://github.com/huggingface/tokenizers>`__), this class provides in addition several advanced
|
||||
alignment methods which can be used to map between the original string (character and words) and the token space (e.g.,
|
||||
getting the index of the token comprising a given character or the span of characters corresponding to a given token).
|
||||
tokenizer, this class behaves just like a standard python dictionary and holds the various model inputs computed by
|
||||
these methods (``input_ids``, ``attention_mask``...). When the tokenizer is a "Fast" tokenizer (i.e., backed by
|
||||
HuggingFace `tokenizers library <https://github.com/huggingface/tokenizers>`__), this class provides in addition
|
||||
several advanced alignment methods which can be used to map between the original string (character and words) and the
|
||||
token space (e.g., getting the index of the token comprising a given character or the span of characters corresponding
|
||||
to a given token).
|
||||
|
||||
|
||||
PreTrainedTokenizer
|
||||
|
||||
Reference in New Issue
Block a user