Update docs around mixing hf scheduler with deepspeed optimizer (#28223)

update docs around mixing hf scheduler with deepspeed optimizer
This commit is contained in:
Dean Wyatte
2024-01-02 04:48:17 -07:00
committed by GitHub
parent 3cefac1d97
commit cad9f5c6cc
2 changed files with 2 additions and 14 deletions

View File

@@ -1221,12 +1221,7 @@ Therefore you have two ways to take advantage of this very beneficial feature:
### Optimizer and Scheduler
As long as you don't enable `offload_optimizer` you can mix and match DeepSpeed and HuggingFace schedulers and
optimizers, with the exception of using the combination of HuggingFace scheduler and DeepSpeed optimizer:
| Combos | HF Scheduler | DS Scheduler |
|:-------------|:-------------|:-------------|
| HF Optimizer | Yes | Yes |
| DS Optimizer | No | Yes |
optimizers.
It is possible to use a non-DeepSpeed optimizer when `offload_optimizer` is enabled, as long as it has both CPU and
GPU implementation (except LAMB).