SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
b73dd1a0e441e2fc215c013dc4ea2fed65db6c5f
HuggingFace_transformer/tests/test_tokenization_bert.py
Mark Neumann 65a89a8976 Fix BasicTokenizer to respect never_split parameters (#2557) 
* add failing test

* fix call to _run_split_on_punc

* format with black
2020-01-17 14:57:56 -05:00

6.1 KiB
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink