SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
d0b65bb4797dc11d1d9dc7b9f66e2b6bd5b47ca5
HuggingFace_transformer/tests/tokenization/test_tokenization_utils.py
Arthur eb4579cf43 tokenizer train from iterator without pre_tokenizers (#35396) 
* fix if else issues

* add a test

* fix the test

* style
2025-01-09 15:34:43 +01:00

20 KiB
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink