SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[FIX] not training when epoch is small (#3006)

Browse Source 
* solving bug where for small epochs and large gradient_accumulation_steps we never train

* black formatting

* no need to change these files
This commit is contained in:
mataney
2020-03-19 17:21:21 +02:00
committed by GitHub
parent ad7233fc01
commit c44a17db1b
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
examples/run_glue.py
View File

@@ -233,7 +233,11 @@ def train(args, train_dataset, model, tokenizer):
loss.backward() loss.backward()
tr_loss += loss.item() tr_loss += loss.item()
if (step + 1) % args.gradient_accumulation_steps == 0: if (step + 1) % args.gradient_accumulation_steps == 0 or (
# last step in epoch but step is always smaller than gradient_accumulation_steps
len(epoch_iterator) <= args.gradient_accumulation_steps
and (step + 1) == len(epoch_iterator)
):
if args.fp16: if args.fp16:
torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm) torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
else: else: