SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
14,571 Commits 5 Branches 239 Tags
181f85da24f4cf5b0ce466e0be7fbe276a790edc
Commit Graph

330 Commits

Author SHA1 Message Date
fxmarty
f93c1e9ece Add RoCm scheduled CI & upgrade RoCm CI to PyTorch 2.1 (#26940) 
* add scheduled ci on amdgpu

* fix likely typo

* more tests, avoid parallelism

* precise comment

* fix report channel

* trigger docker build on this branch

* fix

* fix

* run rocm scheduled ci

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-21 14:55:13 +01:00
Yih-Dar
d903abfccc Fix AMD CI not showing GPU (#27555) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-17 10:44:37 +01:00
Yih-Dar
acb5b4aff5 Disable docker image build job latest-pytorch-amd for now (#27541) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-16 17:00:46 +01:00
Yih-Dar
64e21ca2a4 Make some jobs run on the GitHub Actions runners (#27512) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-15 10:43:16 +01:00
Yih-Dar
00dc856233 At most 2 GPUs for CI (#27435) 
At most 2 GPUs

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-10 16:19:06 +01:00
Mert Yanık
eb30a49b20 Translate index.md to Turkish (#27093) 
* Add index.md for tukish language

* Fix index.md (huggingface/transformers#27088)

* Add 'tr' to additional files

* Update docs/source/tr/_toctree.yml

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update index.md

---------

Co-authored-by: Mert Yanık <mert.yanik@lcwaikiki.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-11-08 08:35:20 -05:00
Yih-Dar
eef7ea98c3 Update doctest workflow file (#27306) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-06 11:27:48 +01:00
Yih-Dar
9dc4ce9ea7 Disable CI runner check (#27170) 
Disable runner check

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-31 11:59:21 +01:00
Aarya Balwadkar
a2f55a65cd Hindi translation of pipeline_tutorial.md (#26837) 
* hindi translation of pipeline_tutorial.md

* Update pipeline_tutorial.md

* Update build_documentation.yml

* Update build_pr_documentation.yml

* Updated build_documentation.yml

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-10-25 11:21:49 -07:00
Akhil
093848d3cc Added Telugu [te] translations (#26828) 
* Create index.md

* Create _toctree.yml

* Updated index.md in telugu

* Update _toctree.yml

* Create quicktour.md

* Update quicktour.md

* Create index.md

* Update quicktour.md

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Delete docs/source/hi/index.md

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update build_documentation.yml

Added telugu [te]

* Update build_pr_documentation.yml

Added Telugu [te]

* Update _toctree.yml

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-10-20 15:27:55 -07:00
Yih-Dar
12cc123359 Better way to run AMD CI with different flavors (#26634) 
* Enable testing against mi250

* Change BERT to trigger tests

* Revert BERT's change

* AMD CI

* AMD CI

---------

Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-16 16:24:30 +02:00
Rockerz
9b7668c03a add japanese documentation (#26138) 
* udpaet

* update

* Update docs/source/ja/autoclass_tutorial.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* add codes workflows/build_pr_documentation.yml

* Create preprocessing.md

* added traning.md

* Create Model_sharing.md

* add quicktour.md

* new

* ll

* Create benchmark.md

* Create Tensorflow_model

* add

* add community.md

* add create_a_model

* create custom_model.md

* create_custom_tools.md

* create fast_tokenizers.md

* create

* add

* Update docs/source/ja/_toctree.yml

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* md

* add

* commit

* add

* h

* Update docs/source/ja/peft.md

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* Update docs/source/ja/_toctree.yml

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* Update docs/source/ja/_toctree.yml

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* Suggested Update

* add perf_train_gpu_one.md

* added perf based MD files

* Modify toctree.yml and Add transmartion to md codes

* Add `serialization.md` and edit `_toctree.yml`

* add task summary and tasks explained

* Add and Modify files starting from T

* Add testing.md

* Create main_classes files

* delete main_classes folder

* Add toctree.yml

* Update llm_tutorail.md

* Update docs/source/ja/_toctree.yml

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update misspelled filenames

* Update docs/source/ja/_toctree.yml

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/_toctree.yml

* Update docs/source/ja/_toctree.yml

* missplled file names inmrpovements

* Update _toctree.yml

* close tip block

* close another tip block

* Update docs/source/ja/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/pipeline_tutorial.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/pipeline_tutorial.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/preprocessing.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/peft.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/add_new_model.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/testing.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/task_summary.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/tasks_explained.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update glossary.md

* Update docs/source/ja/transformers_agents.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/llm_tutorial.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/create_a_model.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/torchscript.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/benchmarks.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/troubleshooting.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/troubleshooting.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/troubleshooting.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/add_new_model.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update perf_torch_compile.md

* Update Year to default in en documentation

* Final Update

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
 2023-10-11 10:26:37 -07:00
Funtowicz Morgan
3632fb3c25 [AMD] Add initial version for run_tests_multi_gpu (#26346) 
* Add initial version for run_tests_multi_gpu

* Trigger change in BERT

* fix typo setup -> setup_gpu

* Add tag mi210

* Enable multi-gpu jobs

* One more

* Use dynamic device allocation

* Attempt to fix syntax for docker create

* fix script path

* fix

* temp machine type

* fix label

* Enable multi-gpu tests

* Rename multi-amd-gpu to multi-gpu

* Let's not be lazy dude

* Update rocm-smi output

* Add gpu_flavour in the matrix

* Fix typos

* merge single/multi dispatch into the matrix

* Format.

* Revert BERT's change

---------

Co-authored-by: Guillaume LEGENDRE <glegendre01@gmail.com>
 2023-10-03 11:13:45 +02:00
Yih-Dar
6ae71ec836 Update runs-on in workflow files (#26435) 
* update

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-09-27 19:25:52 +02:00
Funtowicz Morgan
2d71307dc0 Integrate AMD GPU in CI/CD environment (#26007) 
* Add a Dockerfile for PyTorch + ROCm based on official AMD released artifact

* Add a new artifact single-amdgpu testing on main

* Attempt to test the workflow without merging.

* Changed BERT to check if things are triggered

* Meet the dependencies graph on workflow

* Revert BERT changes

* Add check_runners_amdgpu to correctly mount and check availability

* Rename setup to setup_gpu for CUDA and add setup_amdgpu for AMD

* Fix all the needs.setup -> needs.setup_[gpu|amdgpu] dependencies

* Fix setup dependency graph to use check_runner_amdgpu

* Let's do the runner status check only on AMDGPU target

* Update the Dockerfile.amd to put ourselves in / rather than /var/lib

* Restore the whole setup for CUDA too.

* Let's redisable them

* Change BERT to trigger tests

* Restore BERT

* Add torchaudio with rocm 5.6 to AMD Dockerfile (#26050)

fix dockerfile

Co-authored-by: Felix Marty <felix@hf.co>

* Place AMD GPU tests in a separate workflow (correct branch) (#26105)

AMDGPU CI lives in an other workflow

* Fix invalid job name is dependencies.

* Remove tests multi-amdgpu for now.

* Use single-amdgpu

* Use --net=host for now.

* Remote host networking.

* Removed duplicated check_runners_amdgpu step

* Let's tag machine-types with mi210 for now.

* Machine type should be only mi210

* Remove unnecessary push.branches item

* Apply review suggestions moving from `x-amdgpu` to `x-gpu` introducing `amd-gpu` and `miXXX` labels.

* Remove amdgpu from step names.

* finalize

* delete

---------

Co-authored-by: fxmarty <9808326+fxmarty@users.noreply.github.com>
Co-authored-by: Felix Marty <felix@hf.co>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-09-20 14:48:49 +02:00
Yih-Dar
9219d1427b pin pandas==2.0.3 (#25875) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-30 18:10:01 +02:00
Yih-Dar
62399d6f35 Add flax installation in daily doctest workflow (#25860) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-30 15:13:50 +02:00
Lysandre Debut
8657ec68fc Sets the stalebot to 10 AM CEST (#25678) 
This sets the stale bot trigger time at 10 AM CEST rather than 5 PM CEST as all core maintainers on watch duty are now in the European timezone
 2023-08-23 14:21:07 +02:00
Yih-Dar
b413e0610b Remove utils/documentation_tests.txt (#25680) 
* fix

* fix

* fix

* fix

* fix

* fix

* Apply suggestions from code review

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
 2023-08-23 11:14:45 +02:00
Sylvain Gugger
2defb6b048 More utils doc (#25457) 
* Document and clean more utils.

* More documentation and fixes

* Switch to Lysandre's token

* Address review comments

* Actually put else
 2023-08-17 07:58:35 +02:00
Sylvain Gugger
d561408cc3 Skip Add model like job (#24865) 2023-07-17 15:52:04 -04:00
Yih-Dar
11cb6e0f7e Unpin DeepSpeed and require DS >= 0.9.3 (#24541) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-28 14:01:22 +02:00
Yih-Dar
e84bf1f734 ⚠️ Time to say goodbye to py37 (#24091) 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-28 07:22:39 +02:00
Sylvain Gugger
eb849f6604 Migrate doc files to Markdown. (#24376) 
* Rename index.mdx to index.md

* With saved modifs

* Address review comment

* Treat all files

* .mdx -> .md

* Remove special char

* Update utils/tests_fetcher.py

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

---------

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
 2023-06-20 18:07:47 -04:00
Yih-Dar
0b259a3b7e Clean up disk sapce during docker image build for transformers-pytorch-gpu (#24346) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-19 12:54:02 +02:00
Yih-Dar
896a58de15 Byebye pytorch 1.9 (#24080) 
byebye

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-16 16:38:23 +02:00
Mishig
ef010071ee Update delete_doc_comment_trigger.yml (#24084) 
fix base workflow name
 2023-06-07 17:55:48 +02:00
Mishig
5c9394b54c [doc build] Use secrets (#24079) 2023-06-07 17:33:39 +02:00
Yih-Dar
7631db0fdc Pin deepspeed to 0.9.2 for now (#24024) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-05 20:00:28 +02:00
Lysandre Debut
dc67da0182 Revert "Update stale.yml to use HuggingFaceBot" (#23943) 
Revert "Update stale.yml to use HuggingFaceBot (#23941)"

This reverts commit 5929f86ebb.
 2023-06-01 11:58:11 -04:00
Lysandre Debut
5929f86ebb Update stale.yml to use HuggingFaceBot (#23941) 2023-06-01 10:54:50 -04:00
Yih-Dar
e69feab8a1 Update workflow files (#23658) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-22 21:26:51 +02:00
Yih-Dar
9728f1134b Fix confusing transformers installation in CI (#23465) 
* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-19 22:10:18 +02:00
Yih-Dar
1f2c00d671 Fix DeepSpeed stuff in the nightly CI (#23478) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-19 20:31:55 +02:00
Yih-Dar
22a0769933 Update 3 docker files to use cu118 (#23406) 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-17 14:26:50 +02:00
Yih-Dar
6a6225beab Fix doctest files fetch issue (#23277) 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-11 17:14:06 +02:00
Yih-Dar
9088fcae82 Bring back the PR Refactor doctests + add CI to main (#23271) 
* Revert "Revert "[Doctests] Refactor doctests + add CI" (#23245)"

This reverts commit 69ee46243c.

* try not expose HfDocTestParser

* move into testing_utils.py

* remove pytest install

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-10 22:00:48 +02:00
Sylvain Gugger
69ee46243c Revert "[Doctests] Refactor doctests + add CI" (#23245) 
Revert "[Doctests] Refactor doctests + add CI (#22987)"

This reverts commit 627f44799a.
 2023-05-09 15:26:15 -04:00
Arthur
627f44799a [Doctests] Refactor doctests + add CI (#22987) 
* intiial commit

* new styling

* update

* just run doctest in CI

* remove more test for fast dev

* update

* update refs

* update path and fetch upstream

* update documentatyion trests

* typo

* parse pwd

* don't check for files that are in hidden folders

* just give paths relative to transformers

* update

* update

* update

* major refactoring

* make sure options is ok

* lest test that mdx is tested

* doctest glob

* nits

* update doctest nightly

* some cleaning

* run correct test on diff

* debug

* run on a single worker

* skip_cuda_test tampkate

* updates

* add rA and continue on failure

* test options

* parse `py` codeblock?

* we don't need to replace ignore results, don't remember whyu I put it

* cleanup

* more cleaning

* fix arg

* more cleaning

* clean an todo

* more pre-processing

* doctest-module has none so extra `- ` is needed

* remove logs

* nits

* doctest-modules ....

* oups

* let's use sugar

* make dataset go quiet

* add proper timeout

* nites

* spleling timeout

* update

* properly skip tests that have CUDSA

* proper skipping

* cleaning main and get tests to run

* remove make report?

* remove tee

* some updates

* tee was removed but is the full output still available?

* [all-test]

* only our tests

* don't  touch tee in this PR

* no atee-sys

* proper sub

* monkey

* only replace call

* fix sub

* nits

* nits

* fix invalid syntax

* add skip cuda doctest env variable

* make sure all packages are installed

* move file

* update check repo

* revert changes

* nit

* finish cleanup

* fix re

* findall

* update don't test init files

* ignore pycache

* `-ignore-pycache` when running pytests

* try to fix the import missmatch error

* install dec

* pytest is required as doctest_utils imports things from it

* the only log issues were dataset, ignore results should work

* more cleaning

* Update .circleci/create_circleci_config.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* [ydshieh] empty string if cuda is found

* [ydshieh] fix condition

* style

* [ydshieh] fix

* Add comment

* style

* style

* show failure

* trigger CI

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-09 20:34:48 +02:00
Yih-Dar
073baf7f22 Install accelerete@main in PyTorch Past CI jobs (#22963) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-24 21:19:06 +02:00
Yih-Dar
aa4316757d Change schedule CI time (#22884) 
* fix

* Update .github/workflows/self-nightly-past-ci-caller.yml

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-04-20 14:01:08 +02:00
Yih-Dar
648bd5a8aa Show diff between 2 CI runs on Slack reports (#22798) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-19 19:27:37 +02:00
Yih-Dar
1ebc1dee92 Fix Past CI not running against the latest main (#22823) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-18 15:41:41 +02:00
Sylvain Gugger
e13d6ef7dc Simplify update metadata job (#22811) 
* Simplify update metadata job

* Match more branch names

* Install all what is necessary

* Install all what is necessary

* Forgot the dev

* Install less stuff

* This syntax?
 2023-04-17 13:54:20 -04:00
Yih-Dar
656d41ab4c Remove DS_BUILD_AIO=1 (#22741) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-13 18:08:22 +02:00
Yih-Dar
2c22bc79c2 Make tiny model creation + pipeline testing more robust (#22500) 
* Final Tiny things

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-06 17:45:55 +02:00
Yih-Dar
0fe6c6bdca (Re-)Enable Nightly + Past CI (#22393) 
* Enable Nightly + Past CI

* put schedule

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-03-30 21:06:35 +02:00
Yih-Dar
97440e9c75 Avoid using personal HF token in CI (#22453) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-03-29 19:45:06 +02:00
Yih-Dar
8894b81742 Use real tokenizers if tiny version(s) creation has issue(s) (#22428) 
Fix some tiny model creation issues

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-03-29 16:16:23 +02:00
Yih-Dar
e8cc02555e Automatically create/update tiny models (#22275) 
* Automatically create or update tiny models

* Skip failed tests

* update workflow file

* use revision

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-03-23 19:14:17 +01:00