SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Remove misleading documentation

Browse Source
This commit is contained in:
LysandreJik
2019-12-11 09:22:37 -05:00
parent 51ae203290
commit 4c12860f7a
1 changed files with 0 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
transformers/tokenization_utils.py
View File

@@ -628,7 +628,6 @@ class PreTrainedTokenizer(object):
Take care of added tokens. Take care of added tokens.
text: The sequence to be encoded. text: The sequence to be encoded.
return_tokens_mapped_to_origin: (optional) Set to True to return the index of each token in the initial whitespace tokenization. (default False).
**kwargs: passed to the child `self.tokenize()` method **kwargs: passed to the child `self.tokenize()` method
""" """
def lowercase_text(t): def lowercase_text(t):