SaulLu ae7bae8fe7 fix train_new_from_iterator in the case of byte-level tokenizers (#17549)

2022-06-08 15:30:41 +02:00

..

__init__.py

…

test_tokenization_fast.py

fix train_new_from_iterator in the case of byte-level tokenizers (#17549)

2022-06-08 15:30:41 +02:00

test_tokenization_utils.py

…