SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Allow to set Adam beta1, beta2 in TrainingArgs (#5592)

Browse Source 
* Add Adam beta1, beta2 to trainier

* Make style consistent
This commit is contained in:
Gong Linyuan
2020-07-27 17:31:37 +08:00
committed by GitHub
parent 7969e96f4a
commit b21993b362
4 changed files with 21 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/training_args.py
View File

@@ -160,6 +160,8 @@ class TrainingArguments:
learning_rate: float = field(default=5e-5, metadata={"help": "The initial learning rate for Adam."})
weight_decay: float = field(default=0.0, metadata={"help": "Weight decay if we apply some."})
adam_beta1: float = field(default=0.9, metadata={"help": "Beta1 for Adam optimizer"})
adam_beta2: float = field(default=0.999, metadata={"help": "Beta2 for Adam optimizer"})
adam_epsilon: float = field(default=1e-8, metadata={"help": "Epsilon for Adam optimizer."})
max_grad_norm: float = field(default=1.0, metadata={"help": "Max gradient norm."})