CL-ModelCloud 35c155052d Fix PretrainedTokenizerFast check => Fix PretrainedTokenizerFast Save (#35835) ...

* Fix the bug in tokenizer.save_pretrained when saving tokenizer_class to tokenizer_config.json

* Update tokenization_utils_base.py

* Update tokenization_utils_base.py

* Update tokenization_utils_base.py

* add tokenizer class type test

* code review

* code opt

* fix bug

* Update test_tokenization_fast.py

* ruff check

* make style

* code opt

* Update test_tokenization_fast.py

---------

Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai>
Co-authored-by: LRL-ModelCloud <165116337+LRL-ModelCloud@users.noreply.github.com>

2025-02-13 12:00:33 +01:00

..

__init__.py

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

test_tokenization_fast.py

Fix PretrainedTokenizerFast check => Fix PretrainedTokenizerFast Save (#35835)

2025-02-13 12:00:33 +01:00

test_tokenization_utils.py

tokenizer train from iterator without pre_tokenizers (#35396)

2025-01-09 15:34:43 +01:00