SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[camembert] realign w/ recent changes

Browse Source
This commit is contained in:
Julien Chaumond
2019-11-15 23:40:20 -05:00
parent 26858f27cb
commit f9abf73e31
2 changed files with 6 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
transformers/tokenization_camembert.py
View File

@@ -87,7 +87,7 @@ class CamembertTokenizer(PreTrainedTokenizer):
special tokens for the model
Returns:
A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token.
A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
"""
if already_has_special_tokens:
if token_ids_1 is not None: