SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Clarify use of unk_token in tokenizer docstrings (#9875)

Browse Source
This commit is contained in:
Ethan Chau
2021-01-29 02:11:53 -08:00
committed by GitHub
parent c2d0ffec8c
commit 99b9affa02
2 changed files with 1 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
src/transformers/tokenization_utils.py
View File

@@ -230,9 +230,6 @@ class PreTrainedTokenizer(PreTrainedTokenizerBase):
""" """
Converts a string in a sequence of tokens, using the tokenizer. Converts a string in a sequence of tokens, using the tokenizer.
Note that, unlike Fast tokenizers (instances of PreTrainedTokenizerFast), this method won't replace the unknown
tokens with the `unk_token` yet (this is done in the `encode()` method)
Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies
(BPE/SentencePieces/WordPieces). Takes care of added tokens. (BPE/SentencePieces/WordPieces). Takes care of added tokens.

9
src/transformers/tokenization_utils_base.py
View File

@@ -2043,14 +2043,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin):
def tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bool = False, **kwargs) -> List[str]: def tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bool = False, **kwargs) -> List[str]:
""" """
Converts a string in a sequence of tokens, using the backend Rust tokenizer. Converts a string in a sequence of tokens, replacing unknown tokens with the :obj:`unk_token`.
Note that this method behave differently between fast and slow tokenizers:
- in fast tokenizers (instances of :class:`~transformers.PreTrainedTokenizerFast`), this method will
replace the unknown tokens with the :obj:`unk_token`,
- in slow tokenizers (instances of :class:`~transformers.PreTrainedTokenizer`), this method keep unknown
tokens unchanged.
Args: Args:
text (:obj:`str`): text (:obj:`str`):