SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Update encode documentation (#4751)

Browse Source
This commit is contained in:
Lysandre Debut
2020-06-03 16:30:59 -04:00
committed by GitHub
parent ed4df85572
commit 2e4de76231
1 changed files with 3 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
src/transformers/tokenization_utils.py
View File

@@ -1377,9 +1377,10 @@ class PreTrainedTokenizer(SpecialTokensMixin):
**kwargs
):
"""
Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary.
Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Adds the model-specific
special tokens (such as beginning of sequence, end of sequence, sequence separator).
Same as doing ``self.convert_tokens_to_ids(self.tokenize(text))``.
If specifying ``add_special_tokens=False``, same as doing ``self.convert_tokens_to_ids(self.tokenize(text))``.
Args:
text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`):