SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
13,142 Commits 5 Branches 239 Tags
f7d80cb3d25e26cfae7fa5c91bc6fe8552ed68ea
Commit Graph

13142 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ethan
f7d80cb3d2 Fix steps bugs in no trainer examples (#24197) 
Fix step bugs in no trainer + load checkpoint + grad acc
 2023-06-12 11:49:55 -04:00
Marc Sun
08ae37c820 Fix _load_pretrained_model (#24200) 
Fix test
 2023-06-12 11:31:06 -04:00
Zach Mueller
ebd94b0f6f 🚨🚨🚨 Replace DataLoader logic for Accelerate in Trainer, remove unneeded tests 🚨🚨🚨 (#24028) 
* Working integration

* Fix failing test

* Revert label host logic

* Bring it back!
 2023-06-12 11:23:37 -04:00
Kihoon Son
dc42a9d76f 🌐 [i18n-KO] Translated tasks_summary.mdx to Korean (#23977) 
* 🌐 [i18n-KO] Translated tasks_summary.mdx to Korean

Co-Authored-By: Hyeonseo Yun <0525yhs@gmail.com>
Co-Authored-By: Wonhyeong Seo <wonhseo@kakao.com>
Co-Authored-By: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-Authored-By: Gabriel Yang <gabrielwithhappy@gmail.com>
Co-Authored-By: Jungnerd <46880056+jungnerd@users.noreply.github.com>
Co-Authored-By: Nayeon Han <nayeon2.han@gmail.com>

* Apply suggestions from code review

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>

* Update _toctree.yml

* Delete generation_strategies.mdx

* Delete tasks_explained.mdx

---------

Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Gabriel Yang <gabrielwithhappy@gmail.com>
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>
Co-authored-by: Nayeon Han <nayeon2.han@gmail.com>
 2023-06-12 11:07:15 -04:00
Joao Gante
60b69f7de2 Generate: detect special architectures when loaded from PEFT (#24198) 2023-06-12 16:06:20 +01:00
Jacob
97527898da typo: fix typos in CONTRIBUTING.md and deepspeed.mdx (#24184) 
* typo: fix typos in CONTRIBUTING.md and deepspeed.mdx

* Update CONTRIBUTING.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-06-12 15:43:58 +01:00
Yih-Dar
dadc9fb427 Update GPTNeoXLanguageGenerationTest (#24193) 
* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-12 15:37:12 +02:00
Yih-Dar
a9cdb059a8 Fix device issue in OpenLlamaModelTest::test_model_parallelism (#24195) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-12 15:21:27 +02:00
Joao Gante
9f81f4f6dd Generate: force caching on the main model, in assisted generation (#24177) 2023-06-12 14:10:49 +01:00
Kihoon Son
535f92aea3 [i18n]Translated "attention.mdx" to korean (#23878) 
* [i18n]Translated "attention.mdx" to korean

Co-Authored-By: Hyeonseo Yun <0525yhs@gmail.com>
Co-Authored-By: Wonhyeong Seo <wonhseo@kakao.com>
Co-Authored-By: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-Authored-By: Gabriel Yang <gabrielwithhappy@gmail.com>
Co-Authored-By: Nayeon Han <nayeon2.han@gmail.com>
Co-Authored-By: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>

* Update _toctree.yml

---------

Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Gabriel Yang <gabrielwithhappy@gmail.com>
Co-authored-by: Nayeon Han <nayeon2.han@gmail.com>
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>
 2023-06-12 08:59:18 -04:00
AinL
ba64ec07bb Change ProgressCallback to use dynamic_ncols=True (#24101) 
* Change ProgressCallback to use dynamic_ncols=True

* style: make style

* Revert "style: make style"

This reverts commit dee484904cd30a072d80e3be0a3d74a03cff30c6.

* run make style only trainer_callback
 2023-06-12 08:56:48 -04:00
NielsRogge
93f73a3848 Fix push to hub (#24187) 
Add fix
 2023-06-12 08:51:09 -04:00
Yih-Dar
e26c6f03be Fix Wav2Vec2 CI OOM (#24190) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-12 11:39:04 +02:00
Yih-Dar
8f093fb799 Avoid OOM in doctest CI (#24139) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-10 09:47:38 +02:00
Stas Bekman
0d217f428f [tests] fix bitsandbytes import issue (#24151) 
fix bitsandbytes import issue
 2023-06-09 21:53:11 -07:00
Lysandre Debut
deff5979fe Tool types (#24032) 
* Tool types

* Tests + fixes

* Isolate types

* Oops

* Review comments + docs

* Tests + docs

* soundfile -> vision
 2023-06-09 13:34:07 -04:00
Freddie Vargus
061580c82c Fix typo in streamers.py (#24144) 2023-06-09 17:27:46 +01:00
LiamSwayne
12bb853ccd [documentation] grammatical fixes in image_classification.mdx (#24141) 
Update image_classification.mdx
 2023-06-09 16:59:44 +01:00
Yih-Dar
d0d1632958 Fix Pipeline CI OOM issue (#24124) 
* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-09 16:49:02 +02:00
Arthur
a7501f6fc6 [BlenderBotSmall] Update doc example (#24092) 
* small tokenizer uses `__start__` and `__end__`

* fix PR doctest
 2023-06-09 16:31:57 +02:00
Arthur
5af3a1aa48 [lamaTokenizerFast] Update documentation (#24132) 
* Update documentation

* nits
 2023-06-09 16:30:20 +02:00
Younes Belkada
62fe753325 [SAM] Fix sam slow test (#24140) 
* fix sam test

* update pipeline typehint
 2023-06-09 16:22:09 +02:00
Yih-Dar
847b47c0ee Fix XGLM OOM on CI (#24123) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-09 15:20:59 +02:00
Yih-Dar
b8fe259f16 Fix SAM OOM issue on CI (#24125) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-09 15:07:08 +02:00
Yih-Dar
707023d155 Fix TF Rag OOM issue (#24122) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-09 15:03:11 +02:00
Sourab Mangrulkar
f2b918356c fix bugs with trainer (#24134) 
* fix the deepspeed test failures

* apex fix

* FSDP save ckpt fix

* Update src/transformers/trainer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2023-06-09 17:54:53 +05:30
Joao Gante
be10092e63 Generate: PT's top_p enforces min_tokens_to_keep when it is 1 (#24111) 2023-06-09 13:20:05 +01:00
Matt
03585f3734 Correctly build models and import call_context for older TF versions (#24138) 2023-06-09 13:11:01 +01:00
Younes Belkada
a6d05d55f6 [bnb] Fix bnb config json serialization (#24137) 
* fix bnb config json serialization

* forward contrib credits from discussions

---------

Co-authored-by: Andrechang <Andrechang@users.noreply.github.com>
 2023-06-09 13:41:14 +02:00
Elliott Wang
e2972dffdd PLAM => PaLM (#24129) 2023-06-09 12:32:16 +01:00
Arthur
535542d38d [Lllama] Update tokenization code to ensure parsing of the special tokens [core] (#24042) 
* preventllama fast from returning token type ids

* remove type hints

* normalised False
 2023-06-09 09:36:19 +02:00
Yih-Dar
2e2088f24b Avoid GPT-2 daily CI job OOM (in TF tests) (#24106) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-08 18:21:09 +02:00
Serge Panev
9322c24476 Fix typo in Llama docstrings (#24020) 
* Fix typo in Llama docstrings

Signed-off-by: Serge Panev <spanev@nvidia.com>

* Update

Signed-off-by: Serge Panev <spanev@nvidia.com>

* make style

Signed-off-by: Serge Panev <spanev@nvidia.com>

---------

Signed-off-by: Serge Panev <spanev@nvidia.com>
 2023-06-08 17:19:07 +01:00
Radamés Ajna
a73883ae9e add trust_remote_code option to CLI download cmd (#24097) 
* add trust_remote_code option

* require_torch
 2023-06-08 11:13:57 -04:00
Younes Belkada
8b169142f8 [GPT2] Add correct keys on _keys_to_ignore_on_load_unexpected on all child classes of GPT2PreTrainedModel (#24113) 
* add correct keys on `_keys_to_ignore_on_load_unexpected`

* oops
 2023-06-08 10:21:42 -04:00
Marc Sun
71a114d3e0 fix get_keys_to_not_convert function (#24095) 
* fix get_keys_to_not_convert funct

* Fix style
 2023-06-08 10:14:27 -04:00
Sylvain Gugger
8c5f306719 Update the pin on Accelerate (#24110) 2023-06-08 10:11:01 -04:00
Younes Belkada
2200bf7a45 [Trainer] Correct behavior of _load_best_model for PEFT models (#24103) 
* v1

* some refactor

- add ST format as well

* fix

* add `ADAPTER_WEIGHTS_NAME` & `ADAPTER_SAFE_WEIGHTS_NAME`
 2023-06-08 15:38:30 +02:00
Sourab Mangrulkar
0f23605094 reset accelerate env variables after each test (#24107) 2023-06-08 09:19:07 -04:00
Sadra Barikbin
5fa0a1b23b Fix a tiny typo in WhisperForConditionalGeneration::generate docstring (#24045) 2023-06-08 13:54:56 +01:00
Sylvain Gugger
ba695c1efd v4.31.0.dev0 2023-06-07 16:49:00 -04:00
Sylvain Gugger
c3572e6bfb Add AzureOpenAiAgent (#24058) 
* Add AzureOpenAiAgent

* quality

* Update src/transformers/tools/agents.py

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

---------

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
 2023-06-07 16:34:53 -04:00
Zachary Mueller
5eb3d3c702 Up pinned accelerate version (#24089) 
* Min accelerate

* Also min version

* Min accelerate

* Also min version

* To different minor version

* Empty
 2023-06-07 16:21:51 -04:00
Sourab Mangrulkar
d1c039e398 fix accelerator prepare during eval only mode (#24014) 
* fix mixed precision prep during eval only mode

* update to address comments

* update to reflect the changes in accelerate
 2023-06-08 01:03:13 +05:30
Sylvain Gugger
2c887cf8e0 Do not prepare lr scheduler as it as the right number of steps (#24088) 
* Do not prepare lr scheduler as it as the right number of steps

* Trigger CI

* Trigger CI

* Trigger CI

* Add fake comment

* Remove fake comment

* Trigger CI please!
 2023-06-07 15:31:32 -04:00
Sourab Mangrulkar
12298cb65c fix executable batch size issue (#24067) 
* fix executable batch size issue

* fix

* undo
 2023-06-07 22:08:04 +05:30
Mishig
ef010071ee Update delete_doc_comment_trigger.yml (#24084) 
fix base workflow name
 2023-06-07 17:55:48 +02:00
Sylvain Gugger
89b00eef94 Fix expected value in tests of the test fetcher (#24077) 
* Fix expected value in tests of the test fetcher

* Fix trigger for repo util tests
 2023-06-07 11:38:56 -04:00
Mishig
5c9394b54c [doc build] Use secrets (#24079) 2023-06-07 17:33:39 +02:00
Matt
1fc832b454 Make the TF dummies even smaller (#24071) 
* Let's see if we can use the smallest possible dummies

* Make GPT-2's dummies a little longer

* Just use (1,2) as the default shape

* Update other dummies in sync

* Correct imports for Keras 2.13

* Shrink the Wav2Vec2 dummies
 2023-06-07 16:23:05 +01:00