Arthur
e34da3ee3c
[LlamaTokenizerFast] Refactor default llama ( #28881 )
...
* push legacy to fast as well
* super strange
* Update src/transformers/convert_slow_tokenizer.py
* make sure we are BC
* fix Llama test
* nit
* revert
* more test
* style
* update
* small update w.r.t tokenizers
* nit
* don't split
* lol
* add a test for `add_prefix_space=False`
* fix gemma tokenizer as well
* update
* fix gemma
* nicer failures
* fixup
* update
* fix the example for legacy = False
* use `huggyllama/llama-7b` for the PR doctest
* nit
* use from_slow
* fix llama
2024-04-23 23:12:59 +02:00
..
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-15 09:36:06 +01:00
2024-04-05 17:49:46 +02:00
2024-03-12 17:46:19 +00:00
2024-04-01 07:51:00 +00:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-04-19 17:32:52 +01:00
2024-04-02 09:51:45 +02:00
2024-03-25 10:33:38 +01:00
2024-03-08 10:06:46 +00:00
2024-03-13 14:53:27 +01:00
2024-04-16 17:46:53 +01:00
2024-04-16 17:46:53 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-15 16:30:52 +01:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2024-04-17 12:19:18 +02:00
2024-04-23 10:39:38 +01:00
2024-03-12 17:46:19 +00:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2023-03-22 20:02:24 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-18 15:18:52 +02:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-04 14:18:09 +00:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-22 17:11:13 +01:00
2024-03-12 17:46:19 +00:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-01 07:51:00 +00:00
2023-06-29 10:17:36 +01:00
2024-04-18 09:30:40 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-15 09:36:06 +01:00
2024-02-20 16:17:51 +00:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-08 22:25:07 +08:00
2024-03-13 14:53:27 +01:00
2024-04-02 10:27:26 +02:00
2024-03-25 10:33:38 +01:00
2024-03-28 09:53:31 +00:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2023-11-23 21:00:39 +01:00
2024-04-23 23:12:59 +02:00
2024-04-15 09:36:06 +01:00
2024-03-25 10:33:38 +01:00
2024-03-28 09:31:24 +00:00
2023-08-02 20:22:36 +02:00
2024-03-25 10:33:38 +01:00
2024-03-28 11:25:50 +00:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-04-23 09:56:14 +01:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2023-10-05 10:27:05 +02:00
2024-03-25 10:33:38 +01:00
2024-04-04 10:11:09 +01:00
2024-04-15 17:03:03 +01:00
2024-03-25 10:33:38 +01:00
2024-04-02 10:27:26 +02:00
2024-04-16 17:46:53 +01:00
2024-04-22 17:02:55 +01:00
2024-03-28 09:53:31 +00:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-20 15:50:22 +00:00
2024-04-19 17:32:52 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-23 23:12:59 +02:00
2024-04-11 10:38:24 +02:00
2024-04-03 17:00:08 +01:00
2024-03-13 14:53:27 +01:00
2024-04-19 17:32:52 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-18 10:27:58 +02:00
2024-03-27 04:52:12 +01:00
2024-03-13 14:53:27 +01:00
2024-03-20 15:50:22 +00:00
2024-04-15 09:36:06 +01:00
2024-04-15 09:36:06 +01:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2023-08-02 20:22:36 +02:00
2022-05-03 14:42:02 +02:00
2024-03-13 14:53:27 +01:00
2024-04-17 14:38:48 +05:00
2024-04-17 14:38:48 +05:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-19 17:32:52 +01:00
2024-04-19 17:32:52 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2024-03-28 09:53:31 +00:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-04-17 17:59:07 +02:00
2024-03-04 11:04:49 +00:00
2024-03-25 10:33:38 +01:00
2023-12-14 15:17:30 +00:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-01-29 10:09:26 +00:00
2024-01-29 10:09:26 +00:00
2024-03-13 14:53:27 +01:00
2023-10-31 18:12:14 +01:00
2024-03-25 10:33:38 +01:00
2024-04-08 12:49:31 +01:00
2024-03-28 11:25:50 +00:00
2024-03-13 14:53:27 +01:00
2024-04-16 17:46:53 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-17 14:38:48 +05:00
2024-04-17 14:38:48 +05:00
2024-03-14 16:48:11 +01:00
2024-03-13 14:53:27 +01:00
2024-04-12 17:53:25 +02:00
2024-04-19 17:32:52 +01:00
2024-03-25 10:33:38 +01:00
2024-04-15 09:36:06 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-28 09:53:31 +00:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2023-09-05 10:12:25 +02:00
2023-09-05 10:12:25 +02:00
2024-03-25 10:33:38 +01:00
2024-02-20 16:17:51 +00:00
2024-04-19 17:32:52 +01:00
2024-03-13 14:53:27 +01:00
2024-04-15 09:36:06 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-08 23:51:58 +02:00
2024-04-01 07:51:00 +00:00
2024-04-11 14:59:30 +02:00
2024-04-19 18:31:43 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-28 09:53:31 +00:00
2024-03-25 10:33:38 +01:00
2024-03-12 17:46:19 +00:00
2024-03-25 10:33:38 +01:00
2024-03-12 17:46:19 +00:00
2024-03-25 10:33:38 +01:00
2023-09-18 23:47:54 +02:00
2023-08-02 20:22:36 +02:00
2024-04-15 09:36:06 +01:00
2024-01-23 11:01:50 +00:00
2024-04-19 09:06:01 +02:00
2024-03-28 09:53:31 +00:00
2023-10-03 10:52:34 +02:00
2024-01-11 16:18:27 +01:00
2024-03-12 17:46:19 +00:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-20 15:51:12 +00:00
2024-03-07 20:45:51 +00:00
2023-11-16 17:43:19 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2023-12-08 14:54:32 +01:00
2024-03-25 10:33:38 +01:00
2024-03-13 14:53:27 +01:00
2024-03-25 10:33:38 +01:00
2024-04-22 18:30:38 +01:00
2024-01-18 13:37:34 +00:00
2023-10-31 18:12:14 +01:00
2024-03-25 10:33:38 +01:00
2023-10-30 15:27:15 +01:00
2024-01-11 16:18:27 +01:00
2024-04-19 17:32:52 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2024-04-02 10:27:26 +02:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-02-20 12:06:46 +00:00
2024-04-19 17:32:52 +01:00
2024-02-16 08:16:58 +01:00
2024-03-25 10:33:38 +01:00
2024-03-25 10:33:38 +01:00
2022-05-03 14:42:02 +02:00