Ita Zaporozhets 3d7c2f9dea #32184 save total_vocab_size (#32240) ...

* save total_vocab_size = vocab_size + user added tokens to speed up operation

* updating length when added_tokens_decoder is set

* add test len(tokenizer)

2024-08-05 09:22:48 +02:00

..

__init__.py

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

test_tokenization_fast.py

Skip tests properly (#31308)

2024-06-26 21:59:08 +01:00

test_tokenization_utils.py

#32184 save total_vocab_size (#32240)

2024-08-05 09:22:48 +02:00