Arthur eb4579cf43 tokenizer train from iterator without pre_tokenizers (#35396) ...

* fix if else issues

* add a test

* fix the test

* style

2025-01-09 15:34:43 +01:00

..

__init__.py

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

test_tokenization_fast.py

Skip tests properly (#31308)

2024-06-26 21:59:08 +01:00

test_tokenization_utils.py

tokenizer train from iterator without pre_tokenizers (#35396)

2025-01-09 15:34:43 +01:00