Add qwen2 (#28436)
* add config, modeling, and tokenization * add auto and init * update readme * update readme * update team name * fixup * fixup * update config * update code style * update for fixup * update for fixup * update for fixup * update for testing * update for testing * fix bug for config and tokenization * fix bug for bos token * not doctest * debug tokenizer * not doctest * debug tokenization * debug init for tokenizer * fix style * update init * delete if in token auto * add tokenizer doc * add tokenizer in init * Update dummy_tokenizers_objects.py * update * update * debug * Update tokenization_qwen2.py * debug * Update convert_slow_tokenizer.py * add copies * add copied from and make style * update files map * update test * fix style * fix merge reading and update tests * fix tests * fix tests * fix style * debug a variable in readme * Update src/transformers/models/qwen2/configuration_qwen2.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * update test and copied from * fix style * update qwen2 tokenization and tests * Update tokenization_qwen2.py * delete the copied from after property * fix style * update tests * update tests * add copied from * fix bugs * update doc * add warning for sliding window attention * update qwen2 tokenization * fix style * Update src/transformers/models/qwen2/modeling_qwen2.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * fix tokenizer fast --------- Co-authored-by: Ren Xuancheng <jklj077@users.noreply.github.com> Co-authored-by: renxuancheng.rxc <renxuancheng.rxc@alibaba-inc.com> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
This commit is contained in:
@@ -198,6 +198,7 @@ docs/source/en/model_doc/pop2piano.md
|
||||
docs/source/en/model_doc/prophetnet.md
|
||||
docs/source/en/model_doc/pvt.md
|
||||
docs/source/en/model_doc/qdqbert.md
|
||||
docs/source/en/model_doc/qwen2.md
|
||||
docs/source/en/model_doc/rag.md
|
||||
docs/source/en/model_doc/realm.md
|
||||
docs/source/en/model_doc/reformer.md
|
||||
@@ -745,6 +746,10 @@ src/transformers/models/pvt/image_processing_pvt.py
|
||||
src/transformers/models/pvt/modeling_pvt.py
|
||||
src/transformers/models/qdqbert/configuration_qdqbert.py
|
||||
src/transformers/models/qdqbert/modeling_qdqbert.py
|
||||
src/transformers/models/qwen2/configuration_qwen2.py
|
||||
src/transformers/models/qwen2/modeling_qwen2.py
|
||||
src/transformers/models/qwen2/tokenization_qwen2.py
|
||||
src/transformers/models/qwen2/tokenization_qwen2_fast.py
|
||||
src/transformers/models/rag/configuration_rag.py
|
||||
src/transformers/models/rag/modeling_rag.py
|
||||
src/transformers/models/rag/modeling_tf_rag.py
|
||||
|
||||
Reference in New Issue
Block a user