From daa5dd12025764943ff4a8d788859d320ffbf116 Mon Sep 17 00:00:00 2001 From: Stas Bekman Date: Sun, 26 Jul 2020 11:09:14 -0700 Subject: [PATCH] add a summary report flag for run_examples on CI (#6035) Currently, it's hard to derive which example tests were run on CI, and which weren't. Adding `-rA` flag to `pytest`, will now include a summary like: ``` ==================================================================== short test summary info ===================================================================== PASSED examples/test_examples.py::ExamplesTests::test_generation PASSED examples/test_examples.py::ExamplesTests::test_run_glue PASSED examples/test_examples.py::ExamplesTests::test_run_language_modeling PASSED examples/test_examples.py::ExamplesTests::test_run_squad FAILED examples/test_examples.py::ExamplesTests::test_run_pl_glue - AttributeError: 'Namespace' object has no attribute 'gpus' ============================================================ 1 failed, 4 passed, 8 warnings in 42.96s ============================================================ ``` which makes it easier to validate whether some example is being covered by CI or not. --- .circleci/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 4f681c6454..4d76bb6ae2 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -73,7 +73,7 @@ jobs: - checkout - run: sudo pip install .[sklearn,torch,testing] - run: sudo pip install -r examples/requirements.txt - - run: python -m pytest -n 8 --dist=loadfile -s ./examples/ | tee output.txt + - run: python -m pytest -n 8 --dist=loadfile -rA -s ./examples/ | tee output.txt - store_artifacts: path: ~/transformers/output.txt destination: test_output.txt