Arthur b52a03cd3b ⚠️⚠️[T5Tokenize] Fix T5 family tokenizers⚠️⚠️ (#24565) ...

* don't add space before single letter chars that don't have a merge

* fix the fix

* fixup

* add a test

* more testing

* fixup

* hack to make sure fast is also fixed

* update switch transformers test

* revert convert slow

* Update src/transformers/models/t5/tokenization_t5.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* add typechecking

* quality

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

2023-06-30 07:00:43 +02:00

..

__init__.py

Add Switch transformers (#19323)

2022-11-15 13:06:45 +01:00

test_modeling_switch_transformers.py

⚠️⚠️[T5Tokenize] Fix T5 family tokenizers⚠️⚠️ (#24565)

2023-06-30 07:00:43 +02:00