SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
14,893 Commits 5 Branches 239 Tags
64bdbd888c78dcef5aeaeabc842e12981c8aae7a
Commit Graph

424 Commits

Author SHA1 Message Date
Yih-Dar
64bdbd888c Don't set finetuned_from if it is a local path (#28482) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2024-01-15 11:38:20 +01:00
Joao Gante
4fb3d3a0f6 TF: purge TFTrainer (#28483) 2024-01-12 16:56:34 +00:00
Alex Hedges
95091e1582 Set cache_dir for evaluate.load() in example scripts (#28422) 
While using `run_clm.py`,[^1] I noticed that some files were being added
to my global cache, not the local cache. I set the `cache_dir` parameter
for the one call to `evaluate.load()`, which partially solved the
problem. I figured that while I was fixing the one script upstream, I
might as well fix the problem in all other example scripts that I could.

There are still some files being added to my global cache, but this
appears to be a bug in `evaluate` itself. This commit at least moves
some of the files into the local cache, which is better than before.

To create this PR, I made the following regex-based transformation:
`evaluate\.load\((.*?)\)` -> `evaluate\.load\($1,
cache_dir=model_args.cache_dir\)`. After using that, I manually fixed
all modified files with `ruff` serving as useful guidance. During the
process, I removed one existing usage of the `cache_dir` parameter in a
script that did not have a corresponding `--cache-dir` argument
declared.

[^1]: I specifically used `pytorch/language-modeling/run_clm.py` from
v4.34.1 of the library. For the original code, see the following URL:
acc394c4f5/examples/pytorch/language-modeling/run_clm.py.
 2024-01-11 15:38:44 +01:00
Lysandre
3ed3e3190c Dev version 2023-12-13 18:29:31 +01:00
Adam Louly
4850aaba6f fix no sequence length models error (#27522) 
* fix no sequence length models error

* block size check

---------

Co-authored-by: Adam Louly <adamlouly@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
 2023-12-11 18:01:26 +00:00
Dave Berenbaum
fe41647afc uses dvclive_test mode in examples/pytorch/test_accelerate_examples.py (#27763) 2023-11-30 14:52:03 +01:00
Peter Pan
ce31508134 docs: replace torch.distributed.run by torchrun (#27528) 
* docs: replace torch.distributed.run by torchrun

 `transformers` now officially support pytorch >= 1.10.
 The entrypoint `torchrun`` is present from 1.10 onwards.

Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

* Update src/transformers/trainer.py

with @ArthurZucker's suggestion

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

---------

Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
 2023-11-27 16:26:33 +00:00
Mathias Nielsen
f31af3927f [ examples] fix loading jsonl with load dataset in run translation example (#26924) 
* Renamed variable extension to builder_name

* If builder name is jsonl change to json to align with load_datasets

* Apply suggestions from code review

Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>

---------

Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
 2023-11-20 15:45:42 +01:00
V.Prasanna kumar
ffbcfc0166 Broken links fixed related to datasets docs (#27569) 
fixed the broken links belogs to dataset library of transformers
 2023-11-17 13:44:09 -08:00
Matt
2e72bbab2c Incorrect setting for num_beams in translation and summarization examples (#27519) 
* Remove the torch main_process_first context manager from TF examples

* Correctly set num_beams=1 in our examples, and add a guard in GenerationConfig.validate()

* Update src/transformers/generation/configuration_utils.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-11-15 18:18:54 +00:00
Adam Louly
e6522e49a7 Fixing the failure of models without max_position_embeddings attribute. (#27499) 
fix max pos issue

Co-authored-by: Adam Louly <adamlouly@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
 2023-11-15 18:16:42 +00:00
Zach Mueller
a85ea4b19a Fix wav2vec2 params (#27515) 
Fix test
 2023-11-15 09:24:03 -05:00
Yih-Dar
c8b6052ff6 Final fix of the accelerate installation issue (#27408) 
* fix

* [test-all] commit

* fix

* [test-all] commit

* [test-all] commit

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-09 18:52:29 +01:00
Dave Berenbaum
791ec370d1 Adds dvclive callback (#27352) 
* dvclive trainer callback

* style fixes

* dvclive link fixes
 2023-11-09 12:19:31 +00:00
Zach Mueller
e9adb0c9cf Change thresh in test (#27378) 
Change thresh
 2023-11-09 04:44:36 -05:00
Zach Mueller
845aa832b7 Remove unused param from example script tests (#27354) 
Unused param
 2023-11-08 09:07:32 -05:00
Zach Mueller
efa57cb234 Fix example tests from failing (#27353) 
* Fix example tests from failing

* CHange thresh
 2023-11-08 07:45:21 -05:00
Hz, Ji
b6dbfee0a2 moving example of benchmarking to legacy dir (#27337) 
move example of benchmarking to legacy
 2023-11-08 09:27:37 +01:00
Lysandre
bc78fd1274 Dev version 2023-11-02 18:15:36 +01:00
Dong-geon Lee
25e6e9418c Unify warning styles for better readability (#27184) 2023-10-31 18:12:14 +00:00
Hz, Ji
cd19b19378 make tests of pytorch_example device agnostic (#27081) 2023-10-30 14:56:41 +00:00
Gema Parreño
722e936491 [Typo fix] flag config in WANDB (#27130) 
typo fix flag config
 2023-10-29 18:22:26 +00:00
Lucain
66b088faf0 Provide alternative when warning on use_auth_token (#27105) 2023-10-27 14:32:54 +02:00
Michal Jamroz
e2d6d5ce57 Normalize only if needed (#26049) 
* Normalize only if needed

* Update examples/pytorch/image-classification/run_image_classification.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* if else in one line

* within block

* one more place, sorry for mess

* import order

* Update examples/pytorch/image-classification/run_image_classification.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update examples/pytorch/image-classification/run_image_classification_no_trainer.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-10-24 13:32:03 +01:00
YQ
f71c9ccf59 fix logit-to-multi-hot conversion in example (#26936) 
* fix logit to multi-hot converstion

* add comments

* typo
 2023-10-23 12:33:05 +02:00
Tom Aarsen
40ea9ab2a1 Add many missing spaces in adjacent strings (#26751) 
Add missing spaces in adjacent strings
 2023-10-12 10:28:40 +02:00
Zach Mueller
1d6a84749b Fix checkpoint path in no_trainer scripts (#26733) 
checkpoint path
 2023-10-11 16:16:27 +02:00
jheitmann
3eceaa3637 Fix source_prefix default value (#26654) 2023-10-10 20:49:10 +02:00
Phuc Van Phan
6015f91a5a refactor: change default block_size (#26229) 
* refactor: change default block_size

* fix: return tf to origin

* fix: change files to origin

* rebase

* rebase

* rebase

* rebase

* rebase

* rebase

* rebase

* rebase

* refactor: add min block_size to files

* reformat: add min block_size for run_clm tf
 2023-10-04 15:31:38 +01:00
Lysandre
bd6205919a v4.35.0.dev0 2023-10-03 16:54:37 +02:00
Phuc Van Phan
ba47efbfe4 docs: change assert to raise and some small docs (#26232) 
* docs: change assert to raise and some small docs

* docs: add rule and some document

* fix: fix bug

* fix: fix bug

* chorse: revert logging

* chorse: revert
 2023-09-28 10:14:17 +02:00
Phuc Van Phan
4fb64e285a chore: correct update_step and correct gradient_accumulation_steps (#26068) 2023-09-12 18:31:23 +01:00
Phuc Van Phan
5af2c62696 docs: add space to docs (#26067) 
* docs: add space to docs

* docs: remove reduntant space
 2023-09-11 22:03:26 +01:00
Phuc Van Phan
9cebae64ad docs: update link huggingface map (#26077) 2023-09-11 12:57:04 +01:00
Joao Gante
9a70d6e56f Trainer: delegate default generation values to generation_config (#25987) 2023-09-05 14:47:00 +01:00
Susnato Dhar
404ff8fc17 Fix typo (#25966) 
* Update feature_extraction_clap.py

* changed all lenght to length
 2023-09-05 10:12:25 +02:00
Lysandre
d8e13b3e04 v4.34.dev.0 2023-09-04 15:12:11 -04:00
Zach Mueller
be0e189bd3 Revert frozen training arguments (#25903) 
* Revert frozen training arguments

* TODO
 2023-09-01 11:24:12 -04:00
Phuc Van Phan
656e17f6f7 correct resume training steps number in progress bar (#25691) 
feat: correct update resume update with steps
 2023-08-23 20:09:14 +02:00
Sylvain Gugger
5c67682b16 v4.33.0.dev0 2023-08-21 07:07:04 -04:00
Zach Mueller
ca51499248 Make training args fully immutable (#25435) 
* Make training args fully immutable

* Working tests, PyTorch

* In test_trainer

* during testing

* Use proper dataclass way

* Fix test

* Another one

* Fix tf

* Lingering slow

* Exception

* Clean
 2023-08-15 11:47:47 -04:00
Gema Parreño
0acf56224b Update run_translation.py broken link example Pytoch (#25461) 
* Update run_translation.py

Fixed link

* Update run_translation.py
 2023-08-11 15:41:24 +02:00
Yih-Dar
9c7b744795 Fix missing usage of token (#25382) 
* add missing tokens

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-08 16:27:24 +02:00
Zach Mueller
01ab39b65f Load state in else (#25318) 
* Load else

* New approach

* Propagate
 2023-08-08 05:41:00 -04:00
Phuc Van Phan
5fe36970e5 Adding more information in help parser on train_file and validation_file (#25324) 
chorse: adding new doc on train and val
 2023-08-07 17:56:13 +02:00
Jackmin801
145109382a Allow trust_remote_code in example scripts (#25248) 
* pytorch examples

* pytorch mim no trainer

* cookiecutter

* flax examples

* missed line in pytorch run_glue

* tensorflow examples

* tensorflow run_clip

* tensorflow run_mlm

* tensorflow run_ner

* tensorflow run_clm

* pytorch example from_configs

* pytorch no trainer examples

* Revert "tensorflow run_clip"

This reverts commit 261f86ac1f1c9e05dd3fd0291e1a1f8e573781d5.

* fix: duplicated argument
 2023-08-07 16:32:25 +02:00
Yih-Dar
149cb0cce2 Add token arugment in example scripts (#25172) 
* fix

* fix

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-02 11:17:31 +02:00
Yih-Dar
d53b8ad780 Update use_auth_token -> token in example scripts (#25167) 
* pytorch examples

* tensorflow examples

* flax examples

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-07-28 15:33:45 +02:00
Alan Ji
afa96fffdf make run_generation more generic for other devices (#25133) 
* make run_generation more generic for other devices

* use Accelerate to support any device type it supports.

* make style

* fix error usage of accelerator.prepare_model

* use `PartialState` to make sure everything is running on the right device

---------

Co-authored-by: statelesshz <jihuazhong1@huawei.com>
 2023-07-28 08:20:10 -04:00
Lucain
6232c380f2 Fix .push_to_hub and cleanup get_full_repo_name usage (#25120) 
* Fix .push_to_hub and cleanup get_full_repo_name usage

* Do not rely on Python bool conversion magic

* request changes
 2023-07-28 11:40:08 +02:00