Jingze Shi
d8590b4b0c
Add Doge model ( #35891 )
...
* Add Doge Model
* Fix code quality
* Rollback an error commit
* Fix config for open-source weights
* Revert "Fix config for open-source weights"
This reverts commit 229cdcac10a6a4274d1dd13b729bc14c98eb0c76.
* Add modular_doge
* Update Doge inherits from Llama
* Fix import bug
* [docs] Add usage of doge model
* Fix Doge import pretrainedconfig from modeling_utils to configuration_utils
* [docs] remove trust remote code from doge
* Fix dynamo bug in doge model
* Update docstrings
* Import apply_rotary_pos_emb and repeat_kv from Llama
* Fix all nits
* Fix code quality
* Fix some bugs
* Fix code quality
* Remove inherited `_update_causal_mask` from Llama
This leads to incorrect weight initialization.
* Fix the wrong tensor orderings in DogeCDMoE
* Fix attention mask bug
We have to provide attention_mask for dynamic mask computation
* Modify most implementations to inherit from Llama
But there are two problems:
1. `flex_attention_forward` is not updated properly
2. `Example` error in the forward method of DogeForCausalLM
* Modify CDMoE for batch efficient implementation
* Uniform MoE configuration names, just like QwenMoE
* Fix code quality
* Fix code quality
* Fix code quality
* Add tp plan of CDMoE Module
* Hybird DMA with sliding window
* Update valid tokens greater than window size
* Fix code quality
* Add `convert_doge_weights_to_hf`
* Fix STATE_DICT_MAPPING in convert_doge_weights_to_hf.py
* Fix nits in modular_doge
* Fix code quality
* Fix all nits
* Fix all nits
* Make sure the attention function is updated inside the class
* Fix code quality issues in the Doge model and add a test for it
* Fix `test_generate`
* Fix code quality
* Fix nits fllowing suggestions
* Fix code quality
* Fix code quality issues
* Fix nits
* Fix code quality nits
* Fix the missing parameters in the configuration.
* Fix the missing parameters in the configuration.
* Fix nits
* Add initialization of attention
* Fix last nits
* Simplify dynamic mask generation logic
* Rename router_logits to gate_logits for matching latest changes of MixtralModel
* Rename typings for matching latest changes of MixtralModel
* Fixes typo in comment
* Update src/transformers/models/doge/modular_doge.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com >
* Fix code quality issues to match other modular
* Fix code quality issues to match other modular
* Fix the static compilation errors
* Update model weights link
* Fix code quality issues to match other modular
* reapply modular and support for new outputs
* style
* simplify a lot
* fix import location
* reapply modular
* fix
* fix integration test
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com >
Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co >
Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com >
2025-07-08 11:44:29 +02:00
..
2025-06-16 14:21:18 -07:00
2025-05-28 09:19:09 -07:00
2025-06-13 11:07:09 +00:00
2025-06-24 15:05:29 +02:00
2025-06-05 14:36:54 -07:00
2025-06-13 11:07:09 +00:00
2025-06-26 11:04:23 +00:00
2025-06-13 11:07:09 +00:00
2025-06-16 10:46:30 -07:00
2025-06-18 16:01:25 -07:00
2025-03-03 10:33:46 -08:00
2025-05-27 11:51:41 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-04-30 12:15:43 +01:00
2025-05-27 11:51:22 -07:00
2025-06-13 11:07:09 +00:00
2025-06-30 10:42:56 -07:00
2025-05-23 13:03:47 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-20 13:46:19 -07:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-17 19:37:18 +01:00
2025-06-09 15:02:50 -07:00
2025-06-13 11:07:09 +00:00
2025-06-10 09:30:05 -07:00
2025-06-27 15:26:57 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-04-02 14:57:38 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-04-30 12:15:43 +01:00
2025-06-13 11:07:09 +00:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-02 12:58:01 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-05-27 17:03:55 +02:00
2025-06-13 11:07:09 +00:00
2025-06-17 10:30:03 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-27 10:35:30 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-04-04 11:36:05 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-17 19:37:18 +01:00
2025-07-03 10:02:58 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-01 08:54:22 -07:00
2025-04-30 12:15:43 +01:00
2025-06-12 10:26:50 -07:00
2025-07-08 11:44:29 +02:00
2025-04-14 16:24:01 +02:00
2025-06-25 11:38:25 +02:00
2025-06-13 11:07:09 +00:00
2025-06-18 17:33:29 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-04-30 12:15:43 +01:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-07-02 12:25:26 +01:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-05-21 10:43:11 +02:00
2025-04-30 12:15:43 +01:00
2025-04-30 12:15:43 +01:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-07 17:47:51 +02:00
2025-06-26 14:40:45 -07:00
2025-05-07 17:47:51 +02:00
2025-07-07 14:41:33 +02:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-07-08 08:22:04 +02:00
2025-07-08 08:22:04 +02:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-02 09:55:16 +02:00
2025-06-13 11:07:09 +00:00
2025-06-04 09:56:47 -07:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-05-06 06:47:43 +02:00
2025-06-13 11:07:09 +00:00
2025-02-03 20:06:03 +01:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-23 14:17:25 +00:00
2025-06-23 14:17:25 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-12 11:55:51 +02:00
2025-05-23 17:17:38 +02:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-25 16:09:00 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-07-07 15:56:57 -07:00
2025-06-13 11:07:09 +00:00
2025-06-17 18:10:23 +02:00
2025-06-13 11:07:09 +00:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-16 07:44:40 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 12:03:49 +00:00
2025-06-13 12:03:49 +00:00
2025-05-22 17:12:58 +02:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-15 10:44:19 +02:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-05-07 17:47:51 +02:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-04-15 11:33:09 +01:00
2025-05-07 17:47:51 +02:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-04-30 12:15:43 +01:00
2025-05-28 09:20:19 -07:00
2025-07-07 13:34:59 -04:00
2025-06-27 14:40:24 +00:00
2025-06-13 11:07:09 +00:00
2025-04-30 12:15:43 +01:00
2025-06-12 10:27:17 -07:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-27 14:39:43 +00:00
2025-06-13 11:07:09 +00:00
2025-05-27 16:24:36 -07:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-04-30 12:15:43 +01:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-07 17:47:51 +02:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-26 13:54:48 -07:00
2025-06-12 10:27:07 -07:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-26 14:40:45 -07:00
2025-04-17 23:08:24 +02:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-04-28 11:56:32 +01:00
2025-05-07 17:47:51 +02:00
2025-06-13 11:07:09 +00:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-03-31 09:50:49 +02:00
2025-03-31 09:50:49 +02:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 12:02:44 -07:00
2025-06-17 11:02:18 -07:00
2025-05-23 16:27:56 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-25 14:31:20 +00:00
2025-06-25 14:31:20 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-04-25 12:46:17 -07:00
2025-04-18 13:30:41 -07:00
2025-06-25 15:12:15 +00:00
2025-06-23 14:17:25 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-06-26 10:13:06 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-21 16:16:43 -07:00
2025-05-23 13:04:13 -07:00
2025-06-13 11:07:09 +00:00
2025-04-30 12:15:43 +01:00
2025-06-25 09:05:10 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-04-16 15:00:53 +02:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-18 09:38:58 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-17 19:37:18 +01:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-28 09:19:43 -07:00
2025-06-13 11:07:09 +00:00
2025-03-21 15:35:22 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-07-02 07:56:29 -07:00
2025-04-15 13:16:05 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 17:56:15 +01:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-22 09:16:38 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-09 13:00:38 -07:00
2025-06-09 12:26:31 -07:00
2025-06-13 11:07:09 +00:00
2025-06-09 12:26:23 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-06-04 22:59:17 +00:00