SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
6ce238fe7a930fc03f39cdf72fc9ce9807c83e55
HuggingFace_transformer/tests/tokenization/test_tokenization_utils.py
Arthur eb4579cf43 tokenizer train from iterator without pre_tokenizers (#35396) 
* fix if else issues

* add a test

* fix the test

* style
2025-01-09 15:34:43 +01:00

20 KiB
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink