SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
13,557 Commits 5 Branches 239 Tags
d53b8ad78077bbcebf6727fce0280fea26e29c7f
Commit Graph

2284 Commits

Author SHA1 Message Date
Yih-Dar
d53b8ad780 Update use_auth_token -> token in example scripts (#25167) 
* pytorch examples

* tensorflow examples

* flax examples

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-07-28 15:33:45 +02:00
Alan Ji
afa96fffdf make run_generation more generic for other devices (#25133) 
* make run_generation more generic for other devices

* use Accelerate to support any device type it supports.

* make style

* fix error usage of accelerator.prepare_model

* use `PartialState` to make sure everything is running on the right device

---------

Co-authored-by: statelesshz <jihuazhong1@huawei.com>
 2023-07-28 08:20:10 -04:00
Lucain
6232c380f2 Fix .push_to_hub and cleanup get_full_repo_name usage (#25120) 
* Fix .push_to_hub and cleanup get_full_repo_name usage

* Do not rely on Python bool conversion magic

* request changes
 2023-07-28 11:40:08 +02:00
dependabot[bot]
f1deb21fce Bump certifi from 2022.12.7 to 2023.7.22 in /examples/research_projects/visual_bert (#25097) 
Bump certifi in /examples/research_projects/visual_bert

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-25 17:25:14 -04:00
dependabot[bot]
45bde362d2 Bump certifi from 2022.12.7 to 2023.7.22 in /examples/research_projects/decision_transformer (#25098) 
Bump certifi in /examples/research_projects/decision_transformer

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-25 17:25:05 -04:00
dependabot[bot]
6b8dbc283c Bump certifi from 2022.12.7 to 2023.7.22 in /examples/research_projects/lxmert (#25096) 
Bump certifi in /examples/research_projects/lxmert

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-25 17:24:50 -04:00
Alan Ji
c879318cc5 replace per_gpu_eval_batch_size with per_device_eval_batch_size in readme of multiple-choice task (#25078) 
replace `per_gpu_eval_batch_size` with `per_device_eval_batch_size`
in readme of multiple-choice
 2023-07-25 08:11:56 -04:00
Zach Mueller
aa1b09c5d1 Change logic for logging in the examples (#24956) 
Change logic
 2023-07-20 12:30:10 -04:00
dependabot[bot]
9f912ef62a Bump aiohttp from 3.8.1 to 3.8.5 in /examples/research_projects/decision_transformer (#24954) 
Bump aiohttp in /examples/research_projects/decision_transformer

Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.8.1 to 3.8.5.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/v3.8.5/CHANGES.rst)
- [Commits](https://github.com/aio-libs/aiohttp/compare/v3.8.1...v3.8.5)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-20 12:17:38 -04:00
dependabot[bot]
9859806608 Bump pygments from 2.11.2 to 2.15.0 in /examples/research_projects/decision_transformer (#24949) 
Bump pygments in /examples/research_projects/decision_transformer

Bumps [pygments](https://github.com/pygments/pygments) from 2.11.2 to 2.15.0.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.11.2...2.15.0)

---
updated-dependencies:
- dependency-name: pygments
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-20 07:43:48 -04:00
statelesshz
37d8611ac9 replace no_cuda with use_cpu in test_pytorch_examples (#24944) 
* replace no_cuda with use_cpu in test_pytorch_examples

* remove codes that never be used

* fix style
 2023-07-20 07:09:04 -04:00
ranchlai
8fd8c8e49e Add multi-label text classification support to pytorch example (#24770) 
* Add text classification example

* set the problem type and finetuning task

* ruff reformated

* fix bug for unseting label_to_id for regression

* update README.md

* fixed finetuning task

* update comment

* check if label exists in feature before removing

* add useful logging
 2023-07-20 07:02:44 -04:00
Hwijeen Ahn
dd49404a89 check if eval dataset is dict (#24877) 
* check if eval dataset is dict

* formatting
 2023-07-18 13:33:41 -04:00
Sylvain Gugger
e9ad51306f 4.32.0.dev0 2023-07-17 13:30:44 -04:00
dependabot[bot]
38dfb86958 Bump cryptography from 41.0.0 to 41.0.2 in /examples/research_projects/decision_transformer (#24833) 
Bump cryptography in /examples/research_projects/decision_transformer

Bumps [cryptography](https://github.com/pyca/cryptography) from 41.0.0 to 41.0.2.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/41.0.0...41.0.2)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-07-17 07:17:17 -04:00
Bauke Brenninkmeijer
fc9e387dc0 Replacement of 20 asserts with exceptions (#24757) 
* initial replacements of asserts with errors/exceptions

* replace assert with exception in generation, align and bart

* reset formatting change

* reset another formatting issue

* Apply suggestion

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* don't touch this file

* change to 'is not False'

* fix type

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2023-07-12 07:45:09 -04:00
Gema Parreño
4b26a61631 Fix loading dataset docs link in run_translation.py example (#24594) 
* fix loading dataset link

* Update examples/tensorflow/translation/run_translation.py

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

* Update examples/tensorflow/translation/run_translation.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-07-03 15:21:21 +01:00
Jeroen Van Goey
fd8dcd0953 Udate link to RunHouse hardware setup documentation. (#24590) 
* Udate link to RunHouse hardware setup documentation.

* Fix link to hardware setup in other location as well
 2023-06-30 12:11:58 +01:00
Xiaoli Wang
239ace152b Fix TypeError: Object of type int64 is not JSON serializable (#24340) 
* Fix TypeError: Object of type int64 is not JSON serializable

* Convert numpy.float64 and numpy.int64 to float and int for json serialization

* Black reformatted examples/pytorch/token-classification/run_ner_no_trainer.py

* * make style
 2023-06-27 12:15:49 +01:00
Matt
8e164c5400 Improved keras imports (#24448) 
* An end to accursed version-specific imports

* No more K.is_keras_tensor() either

* Update dependency tables

* Use a cleaner call context function getter

* Add a cap to <2.14

* Add cap to examples requirements too
 2023-06-23 19:09:34 +01:00
Patrick von Platen
1609a436ec Add MMS CTC Fine-Tuning (#24281) 
* Add mms ctc fine tuning

* make style

* More fixes that are needed

* make fix-copies

* make draft for README

* add new file

* move to new file

* make style

* make style

* add quick test

* make style

* make style
 2023-06-15 01:10:27 +02:00
Ethan
f7d80cb3d2 Fix steps bugs in no trainer examples (#24197) 
Fix step bugs in no trainer + load checkpoint + grad acc
 2023-06-12 11:49:55 -04:00
Sylvain Gugger
ba695c1efd v4.31.0.dev0 2023-06-07 16:49:00 -04:00
Zachary Mueller
cbf6bc2350 Oops, missed one (#24054) 
Oops
 2023-06-06 13:30:19 -04:00
Zachary Mueller
072188d638 Act on deprecations in Accelerate no_trainer examples (#24053) 
Act on deprecation
 2023-06-06 13:04:38 -04:00
dependabot[bot]
539e2281cd Bump cryptography from 39.0.1 to 41.0.0 in /examples/research_projects/decision_transformer (#23964) 
Bump cryptography in /examples/research_projects/decision_transformer

Bumps [cryptography](https://github.com/pyca/cryptography) from 39.0.1 to 41.0.0.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/39.0.1...41.0.0)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-02 16:23:44 -04:00
Matt
167a0d8f87 Add an option to reduce compile() console spam (#23938) 
* Add an option to reduce compile() console spam

* Add annotations to the example scripts

* Add notes to the quicktour docs as well

* minor fix
 2023-06-02 15:28:52 +01:00
Sylvain Gugger
3ff443a6d9 Re-enable squad test (#23912) 
* Re-enable squad test

* [all-test]

* [all-test] Fix all test command

* Fix the all-test
 2023-05-31 13:44:26 -04:00
Sylvain Gugger
00f6ba0e7e Skip failing test for now 2023-05-31 06:31:33 -04:00
Ran Ran
e724246935 Fix no such file or directory error (#23783) 
* Fix no such file or directory error

* Address comment

* Fix formatting issue
 2023-05-26 14:24:57 -04:00
dependabot[bot]
d685e330b5 Bump tornado from 6.0.4 to 6.3.2 in /examples/research_projects/visual_bert (#23767) 
Bump tornado in /examples/research_projects/visual_bert

Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.0.4 to 6.3.2.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](https://github.com/tornadoweb/tornado/compare/v6.0.4...v6.3.2)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-25 16:16:12 -04:00
dependabot[bot]
4b0e7ded1c Bump tornado from 6.0.4 to 6.3.2 in /examples/research_projects/lxmert (#23766) 
Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.0.4 to 6.3.2.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](https://github.com/tornadoweb/tornado/compare/v6.0.4...v6.3.2)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-25 16:16:01 -04:00
Sylvain Gugger
6e4bc67099 Revamp test selection for the example tests (#23737) 
* Revamp test selection for the example tests

* Rename old XLA test and fake modif in run_glue

* Fixes

* Fake Trainer modif

* Remove fake modifs
 2023-05-25 09:38:21 -04:00
Wang, Yi
33687a3f61 add GPTJ/bloom/llama/opt into model list and enhance the jit support (#23291) 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
 2023-05-24 10:57:56 +01:00
zspo
003a0cf8cc Fix some docs what layerdrop does (#23691) 
* Fix some docs what layerdrop does

* Update src/transformers/models/data2vec/configuration_data2vec_audio.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Fix more docs

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2023-05-23 14:50:40 -04:00
dependabot[bot]
aa30cd4f3f Bump requests from 2.22.0 to 2.31.0 in /examples/research_projects/lxmert (#23668) 
Bump requests in /examples/research_projects/lxmert

Bumps [requests](https://github.com/psf/requests) from 2.22.0 to 2.31.0.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.22.0...v2.31.0)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-23 05:31:53 -04:00
dependabot[bot]
9bf72ae564 Bump requests from 2.22.0 to 2.31.0 in /examples/research_projects/visual_bert (#23670) 
Bump requests in /examples/research_projects/visual_bert

Bumps [requests](https://github.com/psf/requests) from 2.22.0 to 2.31.0.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.22.0...v2.31.0)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-23 05:31:30 -04:00
dependabot[bot]
ecc05f8c1e Bump requests from 2.27.1 to 2.31.0 in /examples/research_projects/decision_transformer (#23673) 
Bump requests in /examples/research_projects/decision_transformer

Bumps [requests](https://github.com/psf/requests) from 2.27.1 to 2.31.0.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.27.1...v2.31.0)

---
updated-dependencies:
- dependency-name: requests
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-05-23 05:28:09 -04:00
Zachary Mueller
b191d7db44 Update all no_trainer with skip_first_batches (#23664) 2023-05-22 14:49:31 -04:00
sshahrokhi
6f72e71f97 changing the requirements to a cpu torch version that works (#23483) 2023-05-22 12:58:55 -04:00
Boda Sadallah
a7920065f2 fix bug in group_texts function, that was inserting short batches (#23429) 
* fix bug in group_texts function, that was inserting short batches

* fully exclude short batches and return empty dict instead

* fix style
 2023-05-18 14:22:30 -04:00
Zachary Mueller
8a58809312 Fix translation no_trainer (#23407) 
* Fix translation
 2023-05-16 13:10:42 -04:00
Yih-Dar
d51296d9c2 skip test_run_squad_no_trainer for now (#23302) 
skip

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-11 19:26:48 +02:00
Hari
5d02e6bd20 Convert numpy arrays to lists before saving the evaluation metrics as json (#23268) 
* convert numpy array to list before writing to json

per_category_iou and per_category_accuracy  are ndarray in the eval_metrics

* code reformatted with make style
 2023-05-11 08:54:23 -04:00
Maria Khalusova
91f4c84a19 CTC example: updated trainer parameters to save tokenizer (#23243) 
trainer parameters changed to save tokenizer in addition to feature_extractor
 2023-05-10 07:45:10 -04:00
Sylvain Gugger
a0c0a78233 v4.30.0.dev0 2023-05-09 14:59:38 -04:00
Nicolas Patry
c34a525d2f Proposed fix for TF example now running on safetensors. (#23208) 
* Proposed fix for TF example now running on safetensors.

* Adding more warnings and returning keys.

* Trigger CI

* Trigger CI

---------

Co-authored-by: Sylvain Gugger <Sylvain.gugger@gmail.com>
 2023-05-09 13:04:27 -04:00
Sebastian
1a8f61110e fix: Update run_qa.py to work with deepset/germanquad (#23225) 
Call str on id to make sure any ints are converted into the expected format for squad datasets
 2023-05-09 09:20:10 -04:00
Sylvain Gugger
fd6970bc56 Skip failing test 2023-05-08 08:52:44 -04:00
Ashwin Mathur
fc6c8b0eaa Add no_trainer scripts to pre-train Vision Transformers (#23156) 
* Add run_mim_no_trainer.py draft from #20412

Add parse_args method and copy over other dependencies

Add Method call for sending telemetry

Initialize Accelerator

Make one log on every process

Set seed and Handle repository creation

Initialize dataset and Set validation split

Create Config

Adapt Config

Update Config

Create Feature Extractor

Create model

Set column names

Create transforms

Create mask generator

Create method to preprocess images

Shuffle datasets if needed and set transforms

Create Dataloaders

Add optimizer

Add learning rate scheduler

Prepare everything with our accelerator

Tie weights for TPU training

Recalculate training steps and training epochs

Set accelerator checkpointing steps

Initialize trackers and store configuration

Set total batch size

Fix typo: mlm -> mim

Log info at the start of training

Load in the weights and states from previous save

update the progress_bar if load from checkpoint

Define train loop

Add evaluation loop to training

Add to parse_args method

Push repo to hub

Save accelerator state

End training and save model and feature extractor

Remove unused imports

Fix trailing whitespace

* Update code based on comments, Rename feature_extractor to image_processor

* Fix linting

* Add argument for learning rate

* Add argument for setting number of training epochs

* Remove incorrect logger argument

* Convert max_train_steps to int for tqdm

---------

Co-authored-by: Saad Mahmud <shuvro.mahmud79@gmail.com>
 2023-05-05 13:22:49 -04:00