[CI update] Try to use dockers and no cache (#29202)

* change cis * nits * update * minor updates * [push-ci-image] * nit [push-ci-image] * nitsssss * [build-ci-image] * [push-ci-image] * [push-ci-image] * both * [push-ci-image] * this? * [push-ci-image] * pypi-kenlm needs g++ * [push-ci-image] * nit * more nits [push-ci-image] * nits [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * add vision * [push-ci-image] * [push-ci-image] * add new dummy file but will need to update them [push-ci-image] * [push-ci-image] * show package size as well * [push-ci-image] * potentially ignore failures * workflow updates * nits [push-ci-image] * [push-ci-image] * fix consistency * clean nciida triton * also show big packages [push-ci-image] * nit * update * another one * line escape? * add accelerate [push-ci-image] * updates [push-ci-image] * nits to run tests, no push-ci * try to parse skip reason to make sure nothing is skipped that should no be skippped * nit? * always show skipped reasons * nits * better parsing of the test outputs * action="store_true", * failure on failed * show matched * debug * update short summary with skipped, failed and errors * nits * nits * coolu pdates * remove docbuilder * fix * always run checks * oups * nits * don't error out on library printing * non zero exi codes * no warning * nit * WAT? * format nit * [push-ci-image] * fail if fail is needed * [push-ci-image] * sound file for torch light? * [push-ci-image] * order is important [push-ci-image] * [push-ci-image] reduce even further * [push-ci-image] * use pytest rich ! * yes [push-ci-image] * oupsy * bring back the full traceback, but pytest rich should help * nit * [push-ci-image] * re run * nit * [push-ci-image] * [push-ci-image] * [push-ci-image] * empty push to trigger * [push-ci-image] * nit? [push-ci-image] * empty * try to install timm with no deps * [push-ci-image] * oups [push-ci-image] * [push-ci-image] * [push-ci-image] ? * [push-ci-image] open ssh client for git checkout fast * empty for torch light * updates [push-ci-image] * nit * @v4 for checkout * [push-ci-image] * [push-ci-image] * fix fetch tests with parallelism * [push-ci-image] * more parallelism * nit * more nits * empty to re-trigger * empty to re-trigger * split by timing * did not work with previous commit * junit.xml * no path? * mmm this? * junitxml format * split by timing * nit * fix junit family * now we can test if the xunit1 is compatible! * this? * fully list tests * update * update * oups * finally * use classname * remove working directory to make sure the path does not interfere * okay no juni should have the correct path * name split? * sort by classname is what make most sense * some testing * naem * oups * test something fun * autodetect * 18? * nit * file size? * uip * 4 is best * update to see versions * better print * [push-ci-image] * [push-ci-image] * please install the correct keras version * [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * uv is fucking me up * [push-ci-image] * [push-ci-image] * [push-ci-image] * nits * [push-ci-image] * [push-ci-image] * install issues an pins * tapas as well * nits * more paralellism * short tb * soundfile * soundfile * [push-ci-image] * [push-ci-image] * [push-ci-image] * oups * [push-ci-image] * fix some things * [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * use torch-light for hub * small git lfs for hub job * [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * fix tf tapas * [push-ci-image] * nits * [push-ci-image] * don't update the test * [push-ci-image] * [push-ci-image] * [push-ci-image] * no use them * [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * update tf proba * [push-ci-image] * [push-ci-image] * woops * [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * test with built dockers * [push-ci-image] * skip annoying tests * revert fix copy * update test values * update * last skip and fixup * nit * ALL GOOOD * quality * Update tests/models/layoutlmv2/test_image_processing_layoutlmv2.py * Update docker/quality.dockerfile Co-authored-by: Lysandre Debut <hi@lysand.re> * Update src/transformers/models/tapas/modeling_tf_tapas.py Co-authored-by: Lysandre Debut <hi@lysand.re> * Apply suggestions from code review Co-authored-by: Lysandre Debut <hi@lysand.re> * use torch-speed * updates * [push-ci-image] * [push-ci-image] * [push-ci-image] * [push-ci-image] * fuck ken-lm [push-ci-image] * [push-ci-image] * [push-ci-image] --------- Co-authored-by: Lysandre Debut <hi@lysand.re>
2024-05-06 10:10:32 +02:00
parent 91d155ea92
commit 307f632bb2
25 changed files with 430 additions and 291 deletions
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -12,7 +12,7 @@ jobs:
    # Ensure running with CircleCI/huggingface
    check_circleci_user:
        docker:
-            - image: cimg/python:3.8.12
+            - image: python:3.10-slim
        parallelism: 1
        steps:
            - run: echo $CIRCLE_PROJECT_USERNAME
@@ -26,13 +26,11 @@ jobs:
    fetch_tests:
        working_directory: ~/transformers
        docker:
-            - image: cimg/python:3.8.12
+            - image: huggingface/transformers-quality
        parallelism: 1
        steps:
            - checkout
-            - run: pip install --upgrade --upgrade-strategy eager pip
-            - run: pip install -U --upgrade-strategy eager GitPython
-            - run: pip install -U --upgrade-strategy eager .
+            - run: uv pip install -U -e .
            - run: mkdir -p test_preparation
            - run: python utils/tests_fetcher.py | tee tests_fetched_summary.txt
            - store_artifacts:
@@ -88,25 +86,22 @@ jobs:
                      echo "No tests to run, exiting early!"
                      circleci-agent step halt
                  fi
-            - run: cp test_preparation/generated_config.yml test_preparation/generated_config.txt
            - store_artifacts:
-                  path: test_preparation/generated_config.txt
+                path: test_preparation/generated_config.yml
            - store_artifacts:
-                  path: test_preparation/filtered_test_list_cross_tests.txt
+                path: test_preparation/filtered_test_list_cross_tests.txt
            - continuation/continue:
-                  configuration_path: test_preparation/generated_config.yml
+                configuration_path: test_preparation/generated_config.yml

    # To run all tests for the nightly build
    fetch_all_tests:
        working_directory: ~/transformers
        docker:
-            - image: cimg/python:3.8.12
+            - image: huggingface/transformers-consistency
        parallelism: 1
        steps:
            - checkout
-            - run: pip install --upgrade --upgrade-strategy eager pip
-            - run: pip install -U --upgrade-strategy eager GitPython
-            - run: pip install -U --upgrade-strategy eager .
+            - run: uv pip install -e .
            - run: |
                  mkdir test_preparation
                  echo -n "tests" > test_preparation/test_list.txt
@@ -126,7 +121,7 @@ jobs:
    check_code_quality:
        working_directory: ~/transformers
        docker:
-            - image: cimg/python:3.8.12
+            - image: huggingface/transformers-quality
        resource_class: large
        environment:
            TRANSFORMERS_IS_CI: yes
@@ -134,24 +129,7 @@ jobs:
        parallelism: 1
        steps:
            - checkout
-            - restore_cache:
-                  keys:
-                      - v0.7-code_quality-pip-{{ checksum "setup.py" }}
-                      - v0.7-code-quality-pip
-            - restore_cache:
-                  keys:
-                      - v0.7-code_quality-site-packages-{{ checksum "setup.py" }}
-                      - v0.7-code-quality-site-packages
-            - run: pip install --upgrade --upgrade-strategy eager pip
-            - run: pip install -U --upgrade-strategy eager .[all,quality]
-            - save_cache:
-                  key: v0.7-code_quality-pip-{{ checksum "setup.py" }}
-                  paths:
-                      - '~/.cache/pip'
-            - save_cache:
-                  key: v0.7-code_quality-site-packages-{{ checksum "setup.py" }}
-                  paths:
-                      - '~/.pyenv/versions/'
+            - run: uv pip install -e .
            - run:
                name: Show installed libraries and their versions
                command: pip freeze | tee installed.txt
@@ -167,7 +145,7 @@ jobs:
    check_repository_consistency:
        working_directory: ~/transformers
        docker:
-            - image: cimg/python:3.8.12
+            - image: huggingface/transformers-consistency
        resource_class: large
        environment:
            TRANSFORMERS_IS_CI: yes
@@ -175,24 +153,7 @@ jobs:
        parallelism: 1
        steps:
            - checkout
-            - restore_cache:
-                  keys:
-                      - v0.7-repository_consistency-pip-{{ checksum "setup.py" }}
-                      - v0.7-repository_consistency-pip
-            - restore_cache:
-                  keys:
-                      - v0.7-repository_consistency-site-packages-{{ checksum "setup.py" }}
-                      - v0.7-repository_consistency-site-packages
-            - run: pip install --upgrade --upgrade-strategy eager pip
-            - run: pip install -U --upgrade-strategy eager .[all,quality]
-            - save_cache:
-                  key: v0.7-repository_consistency-pip-{{ checksum "setup.py" }}
-                  paths:
-                      - '~/.cache/pip'
-            - save_cache:
-                  key: v0.7-repository_consistency-site-packages-{{ checksum "setup.py" }}
-                  paths:
-                      - '~/.pyenv/versions/'
+            - run: uv pip install -e .
            - run:
                name: Show installed libraries and their versions
                command: pip freeze | tee installed.txt