Files
HuggingFace_transformer/tests/models
Ita Zaporozhets deba7655e6 Add split special tokens (#30772)
* seems like `split_special_tokens` is used here

* split special token

* add new line at end of file

* moving split special token test to common tests

* added assertions

* test

* fixup

* add co-author

* passing rest of args to gptsan_japanese, fixing tests

* removing direct comparison of fast and slow models

* adding test support for UDOP and LayoutXLM

* ruff fix

* readd check if slow tokenizer

* modify test to handle bos tokens

* removing commented function

* trigger build

* applying review feedback - updated docstrings, var names, and simplified tests

* ruff fixes

* Update tests/test_tokenization_common.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* applying feedback, comments

* shutil temp directory fix

---------

Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com>
Co-authored-by: Ita Zaporozhets <itazaporozhets@Itas-MBP.localdomain>
Co-authored-by: itazap <itazap@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Co-authored-by: Ita Zaporozhets <itazaporozhets@Itas-MacBook-Pro.local>
2024-05-24 08:38:58 -07:00
..
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-04-05 17:49:46 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-04-01 07:51:00 +00:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2023-08-02 20:22:36 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-14 16:32:01 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-23 17:18:58 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-23 17:18:58 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-24 08:38:58 -07:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00
2024-05-22 06:40:15 +02:00