SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Document tokenizer_class in configurations (#8152)

Browse Source
This commit is contained in:
Sylvain Gugger
2020-10-29 10:43:45 -04:00
committed by GitHub
parent 969859d5f6
commit b0f1c0ee30
1 changed files with 2 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/configuration_utils.py
View File

@@ -134,6 +134,8 @@ class PretrainedConfig(object):
Parameters linked to the tokenizer Parameters linked to the tokenizer
- **tokenizer_class** (:obj:`str`, `optional`) -- The name of the associated tokenizer class to use (if none is
set, will use the tokenizer associated to the model by default).
- **prefix** (:obj:`str`, `optional`) -- A specific prompt that should be added at the beginning of each text - **prefix** (:obj:`str`, `optional`) -- A specific prompt that should be added at the beginning of each text
before calling the model. before calling the model.
- **bos_token_id** (:obj:`int`, `optional`)) -- The id of the `beginning-of-stream` token. - **bos_token_id** (:obj:`int`, `optional`)) -- The id of the `beginning-of-stream` token.