[ci] Load pretrained models into the default (long-lived) cache

There's an inconsistency right now where:
- we load some models into CACHE_DIR
- and some models in the default cache
- and often, in both for the same models

When running the RUN_SLOW tests, this takes a lot of disk space, time, and bandwidth.

I'd rather always use the default cache

tests/test_modeling_tf_distilbert.py

@@ -219,5 +219,5 @@ class TFDistilBertModelTest(TFModelTesterMixin, unittest.TestCase):
     # @slow
     # def test_model_from_pretrained(self):
     #     for model_name in list(DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
-    #         model = DistilBertModesss.from_pretrained(model_name, cache_dir=CACHE_DIR)
+    #         model = DistilBertModesss.from_pretrained(model_name)
     #         self.assertIsNotNone(model)