Document tokenizer_class in configurations (#8152)

This commit is contained in:
Sylvain Gugger
2020-10-29 10:43:45 -04:00
committed by GitHub
parent 969859d5f6
commit b0f1c0ee30

View File

@@ -134,6 +134,8 @@ class PretrainedConfig(object):
Parameters linked to the tokenizer
- **tokenizer_class** (:obj:`str`, `optional`) -- The name of the associated tokenizer class to use (if none is
set, will use the tokenizer associated to the model by default).
- **prefix** (:obj:`str`, `optional`) -- A specific prompt that should be added at the beginning of each text
before calling the model.
- **bos_token_id** (:obj:`int`, `optional`)) -- The id of the `beginning-of-stream` token.