SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

XLM tokenizer should encode with bos token (#3791)

Browse Source 
* XLM tokenizer should encode with bos token

* Update tests
This commit is contained in:
Lysandre Debut
2020-04-17 11:28:55 -04:00
committed by GitHub
parent 1d4a35b396
commit 8b63a01d95
2 changed files with 7 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
tests/test_tokenization_xlm.py
View File

@@ -96,5 +96,5 @@ class XLMTokenizationTest(TokenizerTesterMixin, unittest.TestCase):
encoded_sentence = tokenizer.build_inputs_with_special_tokens(text)
encoded_pair = tokenizer.build_inputs_with_special_tokens(text, text_2)
assert encoded_sentence == [1] + text + [1]
assert encoded_pair == [1] + text + [1] + text_2 + [1]
assert encoded_sentence == [0] + text + [1]
assert encoded_pair == [0] + text + [1] + text_2 + [1]