Junyang Lin d6ffe74dfa Add qwen2 (#28436) ...

* add config, modeling, and tokenization

* add auto and init

* update readme

* update readme

* update team name

* fixup

* fixup

* update config

* update code style

* update for fixup

* update for fixup

* update for fixup

* update for testing

* update for testing

* fix bug for config and tokenization

* fix bug for bos token

* not doctest

* debug tokenizer

* not doctest

* debug tokenization

* debug init for tokenizer

* fix style

* update init

* delete if in token auto

* add tokenizer doc

* add tokenizer in init

* Update dummy_tokenizers_objects.py

* update

* update

* debug

* Update tokenization_qwen2.py

* debug

* Update convert_slow_tokenizer.py

* add copies

* add copied from and make style

* update files map

* update test

* fix style

* fix merge reading and update tests

* fix tests

* fix tests

* fix style

* debug a variable in readme

* Update src/transformers/models/qwen2/configuration_qwen2.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* update test and copied from

* fix style

* update qwen2 tokenization  and tests

* Update tokenization_qwen2.py

* delete the copied from after property

* fix style

* update tests

* update tests

* add copied from

* fix bugs

* update doc

* add warning for sliding window attention

* update qwen2 tokenization

* fix style

* Update src/transformers/models/qwen2/modeling_qwen2.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* fix tokenizer fast

---------

Co-authored-by: Ren Xuancheng <jklj077@users.noreply.github.com>
Co-authored-by: renxuancheng.rxc <renxuancheng.rxc@alibaba-inc.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

2024-01-17 16:02:22 +01:00

..

benchmark

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

bettertransformer

Fixed malapropism error (#26660)

2023-10-09 11:04:57 +02:00

deepspeed

Fix initialization for missing parameters in from_pretrained under ZeRO-3 (#28245)

2024-01-09 14:58:21 +00:00

extended

Device agnostic trainer testing (#27131)

2023-10-30 18:16:40 +00:00

fixtures

[WIP] add SpeechT5 model (#18922)

2023-02-03 12:43:46 -05:00

fsdp

fix resuming from ckpt when using FSDP with FULL_STATE_DICT (#27891)

2023-12-16 19:41:43 +05:30

generation

Config: warning when saving generation kwargs in the model config (#28514)

2024-01-16 18:31:01 +00:00

models

Add qwen2 (#28436)

2024-01-17 16:02:22 +01:00

optimization

Make schedulers picklable by making lr_lambda fns global (#21768)

2023-03-02 12:08:43 -05:00

peft_integration

[Peft] modules_to_save support for peft integration (#27466)

2023-11-14 10:32:57 +01:00

pipelines

Tokenizer kwargs in textgeneration pipe (#28362)

2024-01-15 16:52:18 +01:00

quantization

[GPTQ] Fix test (#28018)

2024-01-15 11:22:54 -05:00

repo_utils

Allow # Ignore copy (#27328)

2023-12-07 10:00:08 +01:00

sagemaker

Broken links fixed related to datasets docs (#27569)

2023-11-17 13:44:09 -08:00

tokenization

[Styling] stylify using ruff (#27144)

2023-11-16 17:43:19 +01:00

tools

Add support for for loops in python interpreter (#24429)

2023-06-26 09:58:14 -04:00

trainer

Support DeepSpeed when using auto find batch size (#28088)

2024-01-10 06:03:13 -05:00

utils

improve dev setup comments and hints (#28495)

2024-01-15 18:36:40 +00:00

__init__.py

…

test_backbone_common.py

Align backbone stage selection with out_indices & out_features (#27606)

2023-12-20 18:33:17 +00:00

test_cache_utils.py

Generate: SinkCache can handle iterative prompts (#27907)

2023-12-08 20:02:20 +00:00

test_configuration_common.py

[ PretrainedConfig] Improve messaging (#27438)

2023-11-15 14:10:39 +01:00

test_configuration_utils.py

Config: warning when saving generation kwargs in the model config (#28514)

2024-01-16 18:31:01 +00:00

test_feature_extraction_common.py

Split common test from core tests (#24284)

2023-06-15 07:30:24 -04:00

test_feature_extraction_utils.py

Remove-auth-token (#27060)

2023-11-13 14:20:54 +01:00

test_image_processing_common.py

Fix a couple of typos and add an illustrative test (#26941)

2023-12-11 15:51:51 +00:00

test_image_processing_utils.py

Remove-auth-token (#27060)

2023-11-13 14:20:54 +01:00

test_image_transforms.py

Normalize floating point cast (#27249)

2023-11-10 15:35:27 +00:00

test_modeling_common.py

symbolic_trace: add past_key_values, llama, sdpa support (#28447)

2024-01-17 11:50:53 +01:00

test_modeling_flax_common.py

Split common test from core tests (#24284)

2023-06-15 07:30:24 -04:00

test_modeling_flax_utils.py

Default to msgpack for safetensors (#27460)

2023-11-13 15:17:01 +01:00

test_modeling_tf_common.py

Replace build() with build_in_name_scope() for some TF tests (#28046)

2023-12-14 17:42:25 +00:00

test_modeling_tf_utils.py

Replace build() with build_in_name_scope() for some TF tests (#28046)

2023-12-14 17:42:25 +00:00

test_modeling_utils.py

Config: warning when saving generation kwargs in the model config (#28514)

2024-01-16 18:31:01 +00:00

test_pipeline_mixin.py

Shorten the conversation tests for speed + fixing position overflows (#26960)

2023-10-31 14:20:04 +00:00

test_sequence_feature_extraction_common.py

Fix typo (#25966)

2023-09-05 10:12:25 +02:00

test_tokenization_common.py

[ TokenizationUtils] Fix add_special_tokens when the token is already there (#28520)

2024-01-16 16:36:29 +01:00

test_tokenization_utils.py

Remove-auth-token (#27060)

2023-11-13 14:20:54 +01:00