SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Clarify use of unk_token in tokenizer docstrings (#9875)

Browse Source
This commit is contained in:
Ethan Chau
2021-01-29 02:11:53 -08:00
committed by GitHub
parent c2d0ffec8c
commit 99b9affa02
2 changed files with 1 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
src/transformers/tokenization_utils_base.py
View File

@@ -2043,14 +2043,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin):
def tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bool = False, **kwargs) -> List[str]:
"""
Converts a string in a sequence of tokens, using the backend Rust tokenizer.
Note that this method behave differently between fast and slow tokenizers:
- in fast tokenizers (instances of :class:`~transformers.PreTrainedTokenizerFast`), this method will
replace the unknown tokens with the :obj:`unk_token`,
- in slow tokenizers (instances of :class:`~transformers.PreTrainedTokenizer`), this method keep unknown
tokens unchanged.
Converts a string in a sequence of tokens, replacing unknown tokens with the :obj:`unk_token`.
Args:
text (:obj:`str`):