Junyang Lin
d6ffe74dfa
Add qwen2 (#28436)
* add config, modeling, and tokenization
* add auto and init
* update readme
* update readme
* update team name
* fixup
* fixup
* update config
* update code style
* update for fixup
* update for fixup
* update for fixup
* update for testing
* update for testing
* fix bug for config and tokenization
* fix bug for bos token
* not doctest
* debug tokenizer
* not doctest
* debug tokenization
* debug init for tokenizer
* fix style
* update init
* delete if in token auto
* add tokenizer doc
* add tokenizer in init
* Update dummy_tokenizers_objects.py
* update
* update
* debug
* Update tokenization_qwen2.py
* debug
* Update convert_slow_tokenizer.py
* add copies
* add copied from and make style
* update files map
* update test
* fix style
* fix merge reading and update tests
* fix tests
* fix tests
* fix style
* debug a variable in readme
* Update src/transformers/models/qwen2/configuration_qwen2.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* update test and copied from
* fix style
* update qwen2 tokenization and tests
* Update tokenization_qwen2.py
* delete the copied from after property
* fix style
* update tests
* update tests
* add copied from
* fix bugs
* update doc
* add warning for sliding window attention
* update qwen2 tokenization
* fix style
* Update src/transformers/models/qwen2/modeling_qwen2.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* fix tokenizer fast
---------
Co-authored-by: Ren Xuancheng <jklj077@users.noreply.github.com>
Co-authored-by: renxuancheng.rxc <renxuancheng.rxc@alibaba-inc.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2024-01-17 16:02:22 +01:00
..
2023-11-16 11:44:36 -08:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-15 10:13:52 -08:00
2023-11-28 08:40:44 -08:00
2023-11-03 10:57:03 -04:00
2023-12-09 05:38:14 +09:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-06 10:38:21 -08:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-10 13:49:10 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-11 18:03:42 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-23 17:44:08 +00:00
2023-12-09 05:38:14 +09:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-23 17:44:08 +00:00
2023-11-03 10:57:03 -04:00
2023-06-20 18:07:47 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-09-04 17:18:34 +01:00
2024-01-03 18:01:06 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-10-19 15:36:41 +02:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-09 05:38:14 +09:00
2023-12-09 05:38:14 +09:00
2023-11-03 10:57:03 -04:00
2023-12-09 05:38:14 +09:00
2024-01-15 09:09:22 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-10-30 21:42:19 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-20 14:25:07 +05:30
2023-12-07 07:05:00 +01:00
2023-12-11 10:22:26 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-28 13:19:50 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-09 05:38:14 +09:00
2024-01-10 18:09:06 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-24 12:34:24 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2023-11-13 14:20:54 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-09 05:38:14 +09:00
2023-12-11 18:03:42 +00:00
2023-12-11 18:03:42 +00:00
2023-12-05 15:31:35 +01:00
2023-11-29 13:36:38 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-01-17 14:22:44 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-07-24 15:34:19 +01:00
2023-11-03 10:57:03 -04:00
2024-01-17 16:02:22 +01:00
2023-11-03 10:57:03 -04:00
2023-06-20 18:07:47 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-07-13 11:46:54 -04:00
2023-11-06 19:45:03 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-01-08 16:40:36 +00:00
2023-12-14 15:14:13 +00:00
2023-12-14 15:14:13 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-01-08 18:17:16 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-08-25 09:20:37 +02:00
2023-08-03 14:12:07 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-06-20 18:07:47 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2023-11-25 00:59:17 +09:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-04 16:48:37 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-23 17:02:16 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-22 17:21:36 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-15 20:16:47 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-20 17:36:48 +01:00
2023-06-20 18:07:47 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00