Files
HuggingFace_transformer/tests/models
Arthur 15cfe38942 [Core tokenization] add_dummy_prefix_space option to help with latest issues (#28010)
* add add_dummy_prefix_space option to slow

* checking kwargs might be better. Should be there for all spm tokenizer IMO

* nits

* fix copies

* more copied

* nits

* add prefix space

* nit

* nits

* Update src/transformers/convert_slow_tokenizer.py

* fix inti

* revert wrong styling

* fix

* nits

* style

* updates

* make sure we use slow tokenizer for conversion instead of looking for the decoder

* support llama ast well

* update llama tokenizer fast

* nits

* nits nits nits

* update the doc

* update

* update to fix tests

* skip unrelated tailing test

* Update src/transformers/convert_slow_tokenizer.py

* add proper testing

* test decode as well

* more testing

* format

* fix llama test

* Apply suggestions from code review
2024-02-20 12:50:31 +01:00
..
2023-11-28 17:40:01 +01:00
2023-10-24 16:49:26 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-06-16 15:40:49 +01:00
2023-08-02 20:22:36 +02:00
2024-01-11 16:18:27 +01:00
2023-08-02 20:22:36 +02:00
2024-01-31 14:19:02 +01:00
2023-09-18 20:28:36 +02:00
2023-11-16 17:43:19 +01:00
2023-08-02 20:22:36 +02:00
2023-09-26 07:06:04 +02:00
2023-08-02 20:22:36 +02:00
2024-01-11 16:18:27 +01:00
2023-08-02 20:22:36 +02:00
2023-12-21 15:14:46 +00:00
2023-08-02 20:22:36 +02:00
2023-09-05 10:12:25 +02:00
2023-09-05 10:12:25 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2024-02-14 07:15:18 +01:00
2023-08-02 20:22:36 +02:00
2024-01-11 16:18:27 +01:00
2023-08-16 17:45:02 +01:00
2024-01-11 16:18:27 +01:00
2024-01-11 16:18:27 +01:00
2023-08-02 20:22:36 +02:00
2022-05-03 14:42:02 +02:00