SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
f03580fb0210240328fef73ae2f4ab48dcd91cea
HuggingFace_transformer/tests/test_tokenization_common.py
SaulLu 3aa37b945e Add test for a WordLevel tokenizer model (#12437) 
* add a test for a WordLevel tokenizer

* adapt common test to new tokenizer
2021-07-01 12:37:07 +02:00

167 KiB
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink