SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
16,701 Commits 5 Branches 239 Tags
74e19e81e2a23809af192532b9b0e7ea202be6f2
Commit Graph

102 Commits

Author SHA1 Message Date
Nanami
74e19e81e2 Fix spell mistakes (#33149) 2024-08-28 15:27:16 +02:00
Shaopeng Fu
7c31d05b59 fix: (issue #32124) Exception raised when running transformers/examples/flax/language-modeling/t5_tokenizer_model.py. (#32157) 
fix: Exception raised when running .
 2024-08-03 18:24:11 +02:00
Sai-Suraj-27
da79b18087 fix: Removed duplicate field definitions in some classes (#31888) 
Removed duplicate field definitions in classes.
 2024-07-10 13:46:31 +01:00
yukionfire
e9eeedaf3b remove duplicate words in msg (#31876) 2024-07-10 09:54:45 +01:00
Albert Villanova del Moral
a14b055b65 Pass datasets trust_remote_code (#31406) 
* Pass datasets trust_remote_code

* Pass trust_remote_code in more tests

* Add trust_remote_dataset_code arg to some tests

* Revert "Temporarily pin datasets upper version to fix CI"

This reverts commit b7672826ca.

* Pass trust_remote_code in librispeech_asr_dummy docstrings

* Revert "Pin datasets<2.20.0 for examples"

This reverts commit 833fc17a3e.

* Pass trust_remote_code to all examples

* Revert "Add trust_remote_dataset_code arg to some tests" to research_projects

* Pass trust_remote_code to tests

* Pass trust_remote_code to docstrings

* Fix flax examples tests requirements

* Pass trust_remote_dataset_code arg to tests

* Replace trust_remote_dataset_code with trust_remote_code in one example

* Fix duplicate trust_remote_code

* Replace args.trust_remote_dataset_code with args.trust_remote_code

* Replace trust_remote_dataset_code with trust_remote_code in parser

* Replace trust_remote_dataset_code with trust_remote_code in dataclasses

* Replace trust_remote_dataset_code with trust_remote_code arg
 2024-06-17 17:29:13 +01:00
Arthur
673440d073 update ruff version (#30932) 
* update ruff version

* fix research projects

* Empty

* Fix errors

---------

Co-authored-by: Lysandre <lysandre@huggingface.co>
 2024-05-22 06:40:15 +02:00
amyeroberts
57c965a8f1 Remove deprecated logic and warnings (#30743) 
* Remove deprecated logic and warnings

* Add back some code that seems to be important...

* Let's just add all he nllb stuff back; removing it is a bit more involved

* Remove kwargs

* Remove more kwargs
 2024-05-17 12:15:59 +01:00
amyeroberts
bbaa8ceff6 Fix canonical model --model_type in examples (#30480) 
Fix --model_type in examples
 2024-05-01 15:47:05 +01:00
Zach Mueller
60d5f8f9f0 🚨🚨🚨Deprecate evaluation_strategy to eval_strategy🚨🚨🚨 (#30190) 
* Alias

* Note alias

* Tests and src

* Rest

* Clean

* Change typing?

* Fix tests

* Deprecation versions
 2024-04-18 12:49:43 -04:00
Nicolas Patry
8e5f76f511 Upgrading to tokenizers 0.19.0 (#30289) 
* [DO NOT MERGE] Testing tokenizers 0.19.0rc0

* Accounting for the breaking change.

* Ruff.

* Upgrading to tokenizers `0.19` (new release with preprend_scheme fixed
and new surface for BPE tiktoken bug).
 2024-04-17 17:17:50 +02:00
Hilco van der Wilk
b6404866cd Update legacy Repository usage in various example files (#29085) 
* Update legacy Repository usage in `examples/pytorch/text-classification/run_glue_no_trainer.py`

Marked for deprecation here https://huggingface.co/docs/huggingface_hub/guides/upload#legacy-upload-files-with-git-lfs

* Fix import order

* Replace all example usage of deprecated Repository

* Fix remaining repo call and rename args variable

* Revert removing creation of gitignore files and don't change research examples
 2024-03-12 13:20:49 +00:00
Lysandre Debut
f497f564bb Update all references to canonical models (#29001) 
* Script & Manual edition

* Update
 2024-02-16 08:16:58 +01:00
Klaus Hipp
721ee783ca [Docs] Fix spelling and grammar mistakes (#28825) 
* Fix typos and grammar mistakes in docs and examples

* Fix typos in docstrings and comments

* Fix spelling of `tokenizer` in model tests

* Remove erroneous spaces in decorators

* Remove extra spaces in Markdown link texts
 2024-02-02 08:45:00 +01:00
zspo
d98591a12b [docs] fix some bugs about parameter description (#28806) 
Co-authored-by: p_spozzhang <p_spozzhang@tencent.com>
 2024-02-01 16:59:29 +00:00
Klaus Hipp
39fa400969 Fix input data file extension in examples (#28741) 2024-01-29 10:06:31 +00:00
V.Prasanna kumar
ffbcfc0166 Broken links fixed related to datasets docs (#27569) 
fixed the broken links belogs to dataset library of transformers
 2023-11-17 13:44:09 -08:00
Lucain
66b088faf0 Provide alternative when warning on use_auth_token (#27105) 2023-10-27 14:32:54 +02:00
Tom Aarsen
40ea9ab2a1 Add many missing spaces in adjacent strings (#26751) 
Add missing spaces in adjacent strings
 2023-10-12 10:28:40 +02:00
Roy Hvaara
fc63914399 [JAX] Replace uses of jnp.array in types with jnp.ndarray. (#26703) 
`jnp.array` is a function, not a type:
https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.array.html
so it never makes sense to use `jnp.array` in a type annotation. Presumably the intent was to write `jnp.ndarray` aka `jax.Array`.

Co-authored-by: Peter Hawkins <phawkins@google.com>
 2023-10-10 21:35:16 +02:00
Phuc Van Phan
910faa3e1f feat: adding num_proc to load_dataset (#26326) 
* feat: adding num_proc to load_dataset

* feat: add add_num_proc for run_mlm_flax

* feat: add num_proc for bart and t5

* chorse: remove
 2023-09-22 19:22:47 +02:00
Phuc Van Phan
8b5da9fc6e refactor: change default block_size in block size > max position embeddings (#26069) 
* refactor: change default block_size when not initialize

* reformat: add the min of block size
 2023-09-18 16:47:57 +01:00
Phuc Van Phan
5af2c62696 docs: add space to docs (#26067) 
* docs: add space to docs

* docs: remove reduntant space
 2023-09-11 22:03:26 +01:00
Phuc Van Phan
9cebae64ad docs: update link huggingface map (#26077) 2023-09-11 12:57:04 +01:00
Jackmin801
145109382a Allow trust_remote_code in example scripts (#25248) 
* pytorch examples

* pytorch mim no trainer

* cookiecutter

* flax examples

* missed line in pytorch run_glue

* tensorflow examples

* tensorflow run_clip

* tensorflow run_mlm

* tensorflow run_ner

* tensorflow run_clm

* pytorch example from_configs

* pytorch no trainer examples

* Revert "tensorflow run_clip"

This reverts commit 261f86ac1f1c9e05dd3fd0291e1a1f8e573781d5.

* fix: duplicated argument
 2023-08-07 16:32:25 +02:00
Yih-Dar
149cb0cce2 Add token arugment in example scripts (#25172) 
* fix

* fix

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-02 11:17:31 +02:00
Yih-Dar
d53b8ad780 Update use_auth_token -> token in example scripts (#25167) 
* pytorch examples

* tensorflow examples

* flax examples

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-07-28 15:33:45 +02:00
Lucain
6232c380f2 Fix .push_to_hub and cleanup get_full_repo_name usage (#25120) 
* Fix .push_to_hub and cleanup get_full_repo_name usage

* Do not rely on Python bool conversion magic

* request changes
 2023-07-28 11:40:08 +02:00
Bauke Brenninkmeijer
fc9e387dc0 Replacement of 20 asserts with exceptions (#24757) 
* initial replacements of asserts with errors/exceptions

* replace assert with exception in generation, align and bart

* reset formatting change

* reset another formatting issue

* Apply suggestion

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* don't touch this file

* change to 'is not False'

* fix type

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2023-07-12 07:45:09 -04:00
Alex Punnen
805db1fe13 num_noise_spans should be <= num_items #22246 (#22938) 2023-05-02 13:07:30 -04:00
Sylvain
ef28df0572 Fix quality due to ruff release 2023-03-22 20:45:08 -04:00
Aaron Gokaslan
5e8c8eb5ba Apply ruff flake8-comprehensions (#21694) 2023-02-22 09:14:54 +01:00
Sylvain Gugger
6f79d26442 Update quality tooling for formatting (#21480) 
* Result of black 23.1

* Update target to Python 3.7

* Switch flake8 to ruff

* Configure isort

* Configure isort

* Apply isort with line limit

* Put the right black version

* adapt black in check copies

* Fix copies
 2023-02-06 18:10:56 -05:00
Sylvain Gugger
05e72aa0c4 Adapt repository creation to latest hf_hub (#21158) 
* Adapt repository creation to latest hf_hub

* Update all examples

* Fix other tests, add Flax examples

* Address review comments
 2023-01-18 11:14:00 -05:00
fzyzcjy
ae3cbbcaf6 Fix tiny typo (#20841) 
* Fix typo

* Update README.md

* Update run_mlm_flax_stream.py

* Update README.md
 2022-12-20 03:17:59 -05:00
Duong A. Nguyen
4212bb0d60 [Re-submit] Compute true loss Flax examples (#19504) 
* Compute true loss

* fixup

* final

* final

* final

* Update examples/flax/language-modeling/run_bart_dlm_flax.py

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

* jax.tree_map => jax.tree_util.tree_map

* Compute true loss

* final

* fixup

* final

* final

* Update examples/flax/language-modeling/run_bart_dlm_flax.py

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

* jax.tree_map => jax.tree_util.tree_map

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
 2022-10-13 11:33:36 +01:00
Kaiyu Yang
e150c4e2fe Fix the error message in run_t5_mlm_flax.py (#19282) 2022-10-10 14:51:11 +01:00
Sanchit Gandhi
e6f221c8d4 [JAX] Replace all jax.tree_* calls with jax.tree_util.tree_* (#18361) 
* [JAX] Replace all jax.tree_* calls with jax.tree_util.tree_*

* fix double tree_util
 2022-09-09 15:18:56 +02:00
Karim Foda
d6eeb87170 Flax Remat for LongT5 (#17994) 
* [Flax] Add remat (gradient checkpointing)

* fix variable naming in test

* flip: checkpoint using a method

* fix naming

* fix class naming

* apply PVP's suggestions from code review

* add gradient_checkpointing to examples

* Add gradient_checkpointing to run_mlm_flax

* Add remat to longt5

* Add gradient checkpointing test longt5

* Fix args errors

* Fix remaining tests

* Make fixup & quality fixes

* replace kwargs

* remove unecessary kwargs

* Make fixup changes

* revert long_t5_flax changes

* Remove return_dict and copy to LongT5

* Remove test_gradient_checkpointing

Co-authored-by: sanchit-gandhi <sanchit@huggingface.co>
 2022-08-14 16:27:13 +01:00
Julien Chaumond
9129fd0377 transformers-cli login => huggingface-cli login (#18490) 
* zero chance anyone's using that constant no?

* `transformers-cli login` => `huggingface-cli login`

* `transformers-cli repo create` => `huggingface-cli repo create`

* `make style`
 2022-08-06 09:42:55 +02:00
Duong A. Nguyen
3909d7f139 Add Flax BART pretraining script (#18297) 
* add bart pretraining flax script

* fixup

* add bart pretraining flax script

* add BART to README

* add BART to README

* add BART to README

* add BART to README

* add BART to README

* add bos eos document

* Update README.md

* Update README.md

* Update examples/flax/language-modeling/run_bart_dlm_flax.py

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

* final

* final

* final

* remove use_auth_token ing from_config

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
 2022-08-01 12:06:30 -04:00
Sanchit Gandhi
7490a97cac [Flax] Fix incomplete batches in example scripts (#17863) 
* [Flax] Fix incomplete batches in example scripts

* fix dataloader batching

* convert jnp batch idxs to np array

* add missing `pad_shard_unpad` to final prediction generate step

* only `pad_shard_unpad` at inference time

* merge conflicts

* remove incomplete batch step from eval

* fix run_qa.py

* add `pad_shard_unpad` to run_flax_ner.py

* add `pad_shard_unpad` to run_flax_glue.py

* add `pad_shard_unpad` to run_image_classification.py

* make style

* fix mlm flax eval batches

* remove redundant imports
 2022-07-27 15:50:47 +01:00
Duong A. Nguyen
170fcaa604 Generalize decay_mask_fn to apply mask to all LayerNorm params (#18273) 
* generalize decay_mask_fn to find all layernorm params

* fixup

* generalising decay_mask_fn
 2022-07-27 12:23:57 +01:00
Duong A. Nguyen
4bea6584e3 Remove use_auth_token from the from_config method (#18192) 
* remove use_auth_token from from_config

* restore use_auth_token from_pretrained run_t5_mlm_flax
 2022-07-19 08:13:20 +02:00
Duong A. Nguyen
1e8140caad Fix RESOURCE_EXHAUSTED error when dealing with large datasets in Flax example scripts (#18069) 
* Fix RESOURCE_EXHAUSTED error for large datasets on Flax example scripts

* using np.permutation for creating batch_idx

* train_samples_idx -> training_samples_idx

* fix type hints
 2022-07-11 15:59:08 +02:00
Sylvain Gugger
3cab90279f Add examples telemetry (#17552) 
* Add examples telemetry

* Alternative approach

* Add to all other examples

* Add to templates as well

* Put framework separately

* Same for TensorFlow
 2022-06-07 11:57:52 -04:00
Sylvain Gugger
afe5d42d8d Black preview (#17217) 
* Black preview

* Fixup too!

* Fix check copies

* Use the same version as the CI

* Bump black
 2022-05-12 16:25:55 -04:00
Ahmed Elnaggar
5e68675755 Fix t5 shard on TPU Pods (#16527) 
* Fix t5 shard on TPU Pods

The current script doesn't work properly on a TPU pod because the global batch is not divided correctly per host.
This pull request fixes this issue by dividing the global batch to each host before it is shared on each host.

* fix style

Co-authored-by: ahmed-elnaggar <ahmed.elnaggar@allianz.com>
 2022-04-11 16:45:20 +02:00
Karim Foda
24a85cca61 Add use_auth to load_datasets for private datasets to PT and TF examples (#16521) 
* fix formatting and remove use_auth

* Add use_auth_token to Flax examples
 2022-04-04 10:27:45 -04:00
Stas Bekman
a73281e3e4 [examples] max samples can't be bigger than the len of dataset (#16501) 
* [examples] max samples can't be bigger than then len of dataset

* do tf and flax
 2022-03-30 12:33:16 -07:00
Yongrae Jo
8049dfa427 Update run_t5_mlm_flax.py (#16421) 
Fix typo in comment: proprocessed -> preprocessed
 2022-03-28 06:00:53 -04:00