SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

New features for CodeParrot training script (#16851)

Browse Source 
* add tflops logging and fix grad accumulation

* add accelerate tracking and checkpointing

* scale loss of last batch correctly

* fix typo

* compress loss computation

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

* add resume from checkpoint argument

* add load_state accelerate from checkpoint, register lr scheduler and add tflops function

* reformat code

* reformat code

* add condition on path for resume checkpoint

* combine if conditions

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

* add source for tflops formula

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
This commit is contained in:
Loubna Ben Allal
2022-04-21 18:43:46 +02:00
committed by GitHub
parent eef2422e96
commit d91841315a
3 changed files with 69 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
examples/research_projects/codeparrot/README.md
View File

@@ -82,7 +82,7 @@ Now that the dataset, tokenizer, and model are ready we can start training the m
First you need to configure `accelerate` and login to Weights & Biases:
```bash
acclerate config
accelerate config
wandb login
```