SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
13,656 Commits 5 Branches 239 Tags
704bf595eb08aeedac61ab43a479f3f342dbefe9
Commit Graph

13656 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yoach Lacombe
704bf595eb Update Bark generation configs and tests (#25409) 
* update bark generation configs for more coherent parameter

* make style

* update bark hub repo
 2023-08-09 18:28:02 +02:00
Hyeonseo Yun
cf84738d2e 🌐 [i18n-KO] Translated model_summary.md to Korean (#24625) 
* docs: ko: model_summary.md

* feat: nmt and manual edit model_summary.mdx

* fix: resolve suggestions

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>

* fix: resolve suggestions2

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>

---------

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
 2023-08-09 18:27:27 +02:00
MinJae Kang
133aac09b0 🌐 [i18n-KO] Translated add_new_model.md to Korean (#24957) 
* docs: ko: add_new_model.md

* feat: chatgpt draft

* fix: manual edits

* fix: change document title

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: add anchor to header

* Update docs/source/ko/add_new_model.md

Co-authored-by: 이서정 <97655267+sjlee-wise@users.noreply.github.com>

* Update docs/source/ko/add_new_model.md

Co-authored-by: 이서정 <97655267+sjlee-wise@users.noreply.github.com>

* Update docs/source/ko/add_new_model.md

Co-authored-by: 이서정 <97655267+sjlee-wise@users.noreply.github.com>

* fix: edit with reviews

* feat: edit toctree

---------

Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>
Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-authored-by: 이서정 <97655267+sjlee-wise@users.noreply.github.com>
 2023-08-09 18:24:29 +02:00
Maria Khalusova
f2a43c7383 VQA task guide (#25244) 
* initial commit

* semi-finished task guide draft

* image link

* Apply suggestions from code review

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/tasks/visual_question_answering.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* feedback addressed

* Apply suggestions from code review

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* nits addressed

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-08-09 08:29:06 -04:00
Joao Gante
eb3ded16f7 Generate: lower severity of parameterization checks (#25407) 2023-08-09 13:15:06 +01:00
David Reguera
ef74da6582 16059 - Add extra type hints for AltCLIPModel (#25399) 2023-08-09 13:13:33 +01:00
Joao Gante
f456b4d10b Generate: generation config validation fixes in docs (#25405) 2023-08-09 13:07:11 +01:00
Alan Ji
00b93cda21 Improve training args (#25401) 
* enhanced tips for some training args

* make style
 2023-08-09 13:50:13 +02:00
Joao Gante
3deed1f97e Generate: length validation (#25384) 2023-08-09 11:48:32 +01:00
Joao Gante
d59b872c9e Docs: introduction to generation with LLMs (#25240) 
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-08-09 11:09:20 +01:00
amyeroberts
ea5dda2290 YOLOS - Revert default return_pixel_mask value (#25404) 
Revert default return_pixel_mask value
 2023-08-09 11:09:09 +01:00
Sylvain Gugger
599377161b Fix path for dynamic module creation (#25402) 2023-08-09 10:46:05 +02:00
jiqing-feng
85447bb22e rm useless condition since the previous condition contains it. (#25403) 2023-08-09 09:31:24 +02:00
David Reguera
1564a81ac5 16059 - Add missing type hints for ASTModel (#25364) 
* 16059 - Add missing type hints for ASTModel

* Add an additional type hint

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

---------

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
 2023-08-09 08:31:57 +02:00
SeongWooChoi
1367142afd 🌐 [i18n-KO] Translated perf_train_cpu_many.md to Korean (#24923) 
* docs: ko: perf_train_cpu_many.md

* feat: chatgpt draft

* fix: manual edits

* fix: resolve suggestions

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

---------

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>
 2023-08-09 08:15:31 +02:00
Abhipsha Das
41c5f45bfe [DOCS] Add example for TopPLogitsWarper (#25361) 
* [DOCS] Add example for `TopPLogitsWarper`

* fix typo

* address review feedback

* address review nits
 2023-08-08 19:18:33 +02:00
Marc Sun
3a05e010e0 change version (#25387) 2023-08-08 13:05:41 -04:00
amyeroberts
e3490104da Add copied from for image processor methods (#25121) 
* Add copied from statements for image processors

* Move out rescale and normalize to base image processor

* Remove rescale and normalize from vit (post rebase)

* Update docstrings and tidy up

* PR comments
 2023-08-08 17:02:49 +01:00
Yih-Dar
5b517e1764 Use small config for OneFormerModelTest.test_model_with_labels (#25383) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-08 17:15:34 +02:00
Yih-Dar
9c7b744795 Fix missing usage of token (#25382) 
* add missing tokens

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-08 16:27:24 +02:00
Joao Gante
5bd8c011bb Generate: add config-level validation (#25381) 2023-08-08 13:53:03 +01:00
Yih-Dar
9e57e0c063 Fix torch_job worker(s) crashing (#25374) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-08 14:12:56 +02:00
나건주
6247d1b2b6 🌐 [i18n-KO] Translated add_tensorflow_model.md to Korean (#25017) 
* docs: ko: add_tensorflow_model.md

* feat: chatgpt draft

* fix: manual edits

* fix: manual edits

* fix: resolve suggestions

* fix: manual edits
 2023-08-08 13:56:34 +02:00
Alan Ji
26ce4dd8b7 Enable tests to run on third-party devcies (#25327) 
* enable unit tests to run on third-party devcies other than CUDA and CPU.

* remove the modification that enabled ut on MPS

* control test on third-party device by env variable

* update

---------

Co-authored-by: statelesshz <jihuazhong1@huawei.com>
 2023-08-08 13:48:50 +02:00
Yih-Dar
5744482abc Fix token in example template (#25351) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-08 12:00:31 +02:00
Zach Mueller
01ab39b65f Load state in else (#25318) 
* Load else

* New approach

* Propagate
 2023-08-08 05:41:00 -04:00
amyeroberts
36d5b8b06c MaskFormer, Mask2Former - replace einsum for tracing (#25297) 
* Replace einsum with ops for tracing

* Fix comment
 2023-08-08 10:37:14 +01:00
Sanchit Gandhi
dedd11160d [ASR Pipeline] Clarify return timestamps (#25344) 
* [ASR Pipeline] Clarify return timestamps

* fix indentation

* fix ctc check

* fix ctc error message!

* fix test

* fix other test

* add new tests

* final comment
 2023-08-08 10:16:00 +01:00
JB (Don)
5ea2595ecd Add warning for missing attention mask when pad tokens are detected (#25345) 
* Add attention mask and pad token warning to many of the models

* Remove changes under examples/research_projects

These files are not maintained by HG.

* Skip the warning check during torch.fx or JIT tracing

* Switch ordering for the warning and input shape assignment

This ordering is a little cleaner for some of the cases.

* Add missing line break in one of the files
 2023-08-08 10:49:21 +02:00
Yih-Dar
6ea3ee3cd2 Fix test_model_parallelism (#25359) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-08 10:48:45 +02:00
Matthew Hoffman
d4bd33cc9f Register ModelOutput subclasses as supported torch.utils._pytree nodes (#25358) 
* Register ModelOutput subclasses as supported torch.utils._pytree nodes

Fixes #25357 where DDP with static_graph=True does not sync gradients when calling backward() over tensors contained in ModelOutput subclasses

* Add test for torch pytree ModelOutput serialization and deserialization
 2023-08-08 08:12:11 +02:00
David Reguera
a23ac36f8c [DOCS] Add descriptive docstring to MinNewTokensLength (#25196) 
* Add descriptive docstring to MinNewTokensLength

It addresses https://github.com/huggingface/transformers/issues/24783

* Refine the differences between `min_length` and `min_new_tokens`

* Remove extra line

* Remove extra arguments in generate

* Add a missing space

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Run the linter

* Add clarification comments

---------

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-08-08 08:09:17 +02:00
Pedro Lira
080a97119c Add mask2former fp16 support (#25093) 
* Add mask2former fp16 support

* Clear consistency/quality issues

* Fix consistency/quality (2)

* Add integration test for mask2former (fp16 case)

* Fix code quality

* Add integration test for maskformer (fp16 case)

* Add integration test for oneformer (fp16 case)

* Remove slow decorator from fp16 tests

* Fix lint

* Remove usage of full inference and value checks for fp16

* Temporarily comment slow for {mask, mask2, one}former

* Add fp16 support to oneformer

* Revert "Temporarily comment slow for {mask, mask2, one}former"

This reverts commit e5371edabd301cf56079def0421a0a87df307cb0.

* Remove dtype conversion noop
 2023-08-07 20:07:29 +01:00
Merve Noyan
5ee9693a1c Docs: Added benchmarks for torch.compile() for vision models (#24748) 
* added benchmarks for compile

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* added more models

* added more models fr

* added visualizations

* minor fix

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Added links to models and put charts side by side

* Added batch comparisons

* Added more comparisons

* Fix table

* Added link to wheel

* Update perf_torch_compile.md

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-08-07 17:18:43 +01:00
Rishab26
676247fd6b [DOCS] Add NoRepeatNGramLogitsProcessor Example for LogitsProcessor class (#25186) 
* Add Description And Example to Docstring

* make style corrections

* make style

* Doc Style Consistent With HF

* Apply make style

* Modify Docstring

* Edit Type in Docstring

* Feedback Incorporated

* Edit Docstring

* make style

* Post Review Changes

* Review Feedback Incorporated

* Styling

* Formatting

* make style

* pep8
 2023-08-07 17:02:14 +01:00
Phuc Van Phan
5fe36970e5 Adding more information in help parser on train_file and validation_file (#25324) 
chorse: adding new doc on train and val
 2023-08-07 17:56:13 +02:00
Sylvain Gugger
baf1daa58e Migrate Trainer from Repository to upload_folder (#25095) 
* First draft

* Deal with progress bars

* Update src/transformers/utils/hub.py

Co-authored-by: Lucain <lucainp@gmail.com>

* Address review comments

* Forgot one

* Pin hf_hub

* Add argument for push all and fix tests

* Fix tests

* Address review comments

---------

Co-authored-by: Lucain <lucainp@gmail.com>
 2023-08-07 17:47:22 +02:00
Yih-Dar
c177606fb4 Fix more offload edge cases (#25342) 
* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-07 17:45:41 +02:00
Joao Gante
7d65697da7 Generate: remove Marian hack (#25294) 
Remove Marian hack
 2023-08-07 15:38:24 +01:00
Jackmin801
145109382a Allow trust_remote_code in example scripts (#25248) 
* pytorch examples

* pytorch mim no trainer

* cookiecutter

* flax examples

* missed line in pytorch run_glue

* tensorflow examples

* tensorflow run_clip

* tensorflow run_mlm

* tensorflow run_ner

* tensorflow run_clm

* pytorch example from_configs

* pytorch no trainer examples

* Revert "tensorflow run_clip"

This reverts commit 261f86ac1f1c9e05dd3fd0291e1a1f8e573781d5.

* fix: duplicated argument
 2023-08-07 16:32:25 +02:00
calpt
65001cb1c8 Loosen output shape restrictions on GPT-style models (#25188) 
* Loosen output shape restrictions on GPT-style models

* Use more self-explanatory variables

* Revert "Use more self-explanatory variables"

This reverts commit 5fd9ab39119558b7e750f61aa4a19014dccc5ed5.
 2023-08-07 16:31:15 +02:00
oobabooga
d6bfba76be Generalize CFG to allow for positive prompts (#25339) 
* Generalize CFG to allow for positive prompts

* Add documentation, fix the correct class
 2023-08-07 16:25:15 +02:00
Yih-Dar
b0f23036f1 Update TF pin in docker image (#25343) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-07 12:32:34 +02:00
Injin Paek
b9da44bd3e 🌐 [i18n-KO] Translated perf_infer_gpu_one.md to Korean (#24978) 
* docs: ko: perf_infer_gpu_one

* feat: chatgpt draft

* fix: manual edits

* fix: manual edits

* fix: resolve suggestions

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: TaeYupNoh <107118671+TaeYupNoh@users.noreply.github.com>

* fix: resolve suggestions

* fix: resolve suggestions

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

---------

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: TaeYupNoh <107118671+TaeYupNoh@users.noreply.github.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
 2023-08-07 08:37:29 +02:00
Guillaume "Vermeille" Sanchez
d533465150 add CFG for .generate() (#24654) 2023-08-06 20:15:24 +01:00
mariecwhite
a6e6b1c622 Remove jnp.DeviceArray since it is deprecated. (#24875) 
* Remove jnp.DeviceArray since it is deprecated.

* Replace all instances of jnp.DeviceArray with jax.Array

* Update src/transformers/models/bert/modeling_flax_bert.py

---------

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
 2023-08-04 18:36:57 +01:00
Sanchit Gandhi
fdd81aea12 [Whisper] Better error message for outdated generation config (#25298) 2023-08-04 15:53:57 +01:00
Sylvain Gugger
fdaef3368b Document toc check and doctest check scripts (#25319) 
* Clean doc toc check and make doctest list better

* Add to Makefile
 2023-08-04 16:24:04 +02:00
Yih-Dar
ce6d153a53 Make bark could have tiny model (#25290) 
* temp

* update

* update

* update

* small dim

* small dim

* small dim

* fix

* update

* fix

* fix

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-04 15:13:14 +02:00
Sylvain Gugger
f0fd73a2de Document check copies (#25291) 
* Document check copies better and add tests

* Include header in check for copies

* Manual fixes

* Try autofix

* Fixes

* Clean tests

* Finalize doc

* Remove debug print

* More fixes
 2023-08-04 14:56:29 +02:00