Add inverse sqrt learning rate scheduler (#21495)

* added inverse sqrt lr scheduler

* Updated get_scheduler in src/transformers/optimization.py

* Updated src/transformers/__init__.py

* Added inverse sqrt lr scheduler test

* Updated docs/source/en/main_classes/optimizer_schedules.mdx

* Ran style and quality scripts

* Fix get_inverse_sqrt_schedule docstring

* Comment implementation URL
This commit is contained in:
Adrian Sager La Ganga
2023-02-07 21:00:50 +01:00
committed by GitHub
parent b9af152efb
commit a3034c7004
6 changed files with 54 additions and 0 deletions

View File

@@ -60,6 +60,8 @@ The `.optimization` module provides:
[[autodoc]] get_polynomial_decay_schedule_with_warmup
[[autodoc]] get_inverse_sqrt_schedule
### Warmup (TensorFlow)
[[autodoc]] WarmUp