Update docs around mixing hf scheduler with deepspeed optimizer (#28223)
update docs around mixing hf scheduler with deepspeed optimizer
This commit is contained in:
@@ -1221,12 +1221,7 @@ Therefore you have two ways to take advantage of this very beneficial feature:
|
||||
### Optimizer and Scheduler
|
||||
|
||||
As long as you don't enable `offload_optimizer` you can mix and match DeepSpeed and HuggingFace schedulers and
|
||||
optimizers, with the exception of using the combination of HuggingFace scheduler and DeepSpeed optimizer:
|
||||
|
||||
| Combos | HF Scheduler | DS Scheduler |
|
||||
|:-------------|:-------------|:-------------|
|
||||
| HF Optimizer | Yes | Yes |
|
||||
| DS Optimizer | No | Yes |
|
||||
optimizers.
|
||||
|
||||
It is possible to use a non-DeepSpeed optimizer when `offload_optimizer` is enabled, as long as it has both CPU and
|
||||
GPU implementation (except LAMB).
|
||||
|
||||
Reference in New Issue
Block a user