SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[deepspeed] offload + non-cpuadam optimizer exception doc (#22044)

Browse Source 
* [deepspeed] offload + non-cpuadam optimizer exception doc

* deps
This commit is contained in:
Stas Bekman
2023-03-21 17:00:05 -07:00
committed by GitHub
parent 5990743fdd
commit 89a0a9eace
3 changed files with 13 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
docs/source/en/main_classes/deepspeed.mdx
View File

@@ -1293,8 +1293,17 @@ If you want to use another optimizer which is not listed above, you will have to
}
```
Similarly to `AdamW`, you can configure other officially supported optimizers. Just remember that may have different
config values. e.g. for Adam you will want `weight_decay` around `0.01`.
Similarly to `AdamW`, you can configure other officially supported optimizers. Just remember that those may have different config values. e.g. for Adam you will want `weight_decay` around `0.01`.
Additionally, offload works the best when it's used with Deepspeed's CPU Adam optimizer. If you want to use a different optimizer with offload, since `deepspeed==0.8.3` you need to also add:
```json
{
"zero_force_ds_cpu_optimizer": false
}
```
to the top level configuration.