[ci] Load pretrained models into the default (long-lived) cache

There's an inconsistency right now where:
- we load some models into CACHE_DIR
- and some models in the default cache
- and often, in both for the same models

When running the RUN_SLOW tests, this takes a lot of disk space, time, and bandwidth.

I'd rather always use the default cache

tests/utils.py

@@ -1,13 +1,10 @@
 import os
-import tempfile
 import unittest
 from distutils.util import strtobool
 
 from transformers.file_utils import _tf_available, _torch_available
 
 
-CACHE_DIR = os.path.join(tempfile.gettempdir(), "transformers_test")
-
 SMALL_MODEL_IDENTIFIER = "julien-c/bert-xsmall-dummy"
 DUMMY_UNKWOWN_IDENTIFIER = "julien-c/dummy-unknown"
 # Used to test Auto{Config, Model, Tokenizer} model_type detection.