SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Warn if debug requested without TPU fixes (#6308) (#6390)

Browse Source 
* Warn if debug requested without TPU fixes (#6308)
Check whether a PyTorch compatible TPU is available before attempting to print TPU metrics after training has completed. This way, users who apply `--debug` without reading the documentation aren't suprised by a stacktrace.

* Style

Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
This commit is contained in:
David LaPalomento
2020-08-11 05:31:26 -04:00
committed by GitHub
parent cdf1f7edb2
commit 87e124c245
1 changed files with 8 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
src/transformers/trainer.py
View File

@@ -624,8 +624,14 @@ class Trainer:
train_iterator.close() train_iterator.close()
break break
if self.args.tpu_metrics_debug or self.args.debug: if self.args.tpu_metrics_debug or self.args.debug:
if is_torch_tpu_available():
# tpu-comment: Logging debug metrics for PyTorch/XLA (compile, execute times, ops, etc.) # tpu-comment: Logging debug metrics for PyTorch/XLA (compile, execute times, ops, etc.)
xm.master_print(met.metrics_report()) xm.master_print(met.metrics_report())
else:
logger.warning(
"You enabled PyTorch/XLA debug metrics but you don't have a TPU "
"configured. Check your training configuration if this is unexpected."
)
if self.tb_writer: if self.tb_writer:
self.tb_writer.close() self.tb_writer.close()