Ita Zaporozhets 3d7c2f9dea #32184 save total_vocab_size (#32240) ...

* save total_vocab_size = vocab_size + user added tokens to speed up operation

* updating length when added_tokens_decoder is set

* add test len(tokenizer)

2024-08-05 09:22:48 +02:00

13 KiB

Raw Blame History

View Raw