Stas Bekman 580dd87c55 [Deepspeed] add support for bf16 mode (#14569) ...

* [WIP] add support for bf16 mode

* prep for bf16

* prep for bf16

* fix; zero2/bf16 is ok

* check bf16 is available

* test fixes

* enable zero3_bf16

* config files

* docs

* split stage_dtype; merge back to non-dtype-specific config file

* fix doc

* cleanup

* cleanup

* bfloat16 => bf16 to match the PR changes

* s/zero_gather_fp16_weights_on_model_save/zero_gather_16bit_weights_on_model_save/; s/save_fp16_model/save_16bit_model/

* test fixes/skipping

* move

* fix

* Update docs/source/main_classes/deepspeed.mdx

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* backticks

* cleanup

* cleanup

* cleanup

* new version

* add note about grad accum in bf16

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

2022-03-11 17:53:53 -08:00

..

internal

Constrained Beam Search [*With* Disjunctive Decoding] (#15761)

2022-03-04 18:18:34 +01:00

main_classes

[Deepspeed] add support for bf16 mode (#14569)

2022-03-11 17:53:53 -08:00

model_doc

Add Document Image Transformer (DiT) (#15984)

2022-03-10 11:34:44 +01:00

tasks

Audio/vision task guides (#15808)

2022-03-11 16:43:49 -06:00

_config.py

Force default brnahc name via the config

2022-03-11 10:09:15 -05:00

_toctree.yml

Audio/vision task guides (#15808)

2022-03-11 16:43:49 -06:00

accelerate.mdx

Fix code format for Accelerate doc (#15335)

2022-01-27 13:49:04 -06:00

add_new_model.mdx

added link to our writing-doc document (#15756)

2022-02-22 09:57:28 +01:00

add_new_pipeline.mdx

Doc styler examples (#14953)

2021-12-27 19:07:46 -05:00

autoclass_tutorial.mdx

Update tutorial docs (#15165)

2022-02-01 18:31:35 -06:00

benchmarks.mdx

[Benchmark tools] Deprecate all (#15848)

2022-03-01 11:26:20 +01:00

bertology.mdx

Convert rst files (#14888)

2021-12-22 16:14:35 -05:00

community.mdx

add t5 ner finetuning (#15432)

2022-01-31 17:03:06 +01:00

contributing.md

Update installation page and add contributing to the doc (#5084)

2020-06-17 14:01:10 -04:00

converting_tensorflow_models.mdx

Convert rst files (#14888)

2021-12-22 16:14:35 -05:00

create_a_model.mdx

Create a custom model guide (#15489)

2022-02-07 12:34:56 -06:00

custom_datasets.mdx

Added missing code in exemplary notebook - custom datasets fine-tuning (#15300)

2022-01-25 17:26:17 -05:00

custom_models.mdx

[doc] custom_models: mention security features of the Hub (#15768)

2022-02-23 11:40:06 -05:00

debugging.mdx

add a network debug script and document it (#15652)

2022-02-15 08:48:00 -08:00

fast_tokenizers.mdx

Convert rst files (#14888)

2021-12-22 16:14:35 -05:00

glossary.mdx

Doc styler examples (#14953)

2021-12-27 19:07:46 -05:00

index.mdx

Add Document Image Transformer (DiT) (#15984)

2022-03-10 11:34:44 +01:00

installation.mdx

Get started docs (#15098)

2022-01-28 19:01:37 -06:00

migration.mdx

Doc styler examples (#14953)

2021-12-27 19:07:46 -05:00

model_sharing.mdx

Update model share tutorial (#15288)

2022-01-28 18:49:26 -06:00

model_summary.mdx

Add "open in hf spaces" gradio button issue #73 (#15106)

2022-01-14 10:12:30 -05:00

multilingual.mdx

Inference for multilingual models (#15836)

2022-03-01 15:10:31 -06:00

notebooks.md

Update notebooks (#3620)

2020-04-06 14:32:39 -04:00

parallelism.mdx

[deepspeed docs] Megatron-Deepspeed info (#15488)

2022-02-04 11:15:13 -08:00

performance.mdx

add model scaling section (#15119)

2022-02-09 15:27:30 +01:00

perplexity.mdx

Doc styler examples (#14953)

2021-12-27 19:07:46 -05:00

philosophy.mdx

Convert rst files (#14888)

2021-12-22 16:14:35 -05:00

pipeline_tutorial.mdx

Update tutorial docs (#15165)

2022-02-01 18:31:35 -06:00

pr_checks.mdx

[docs] fix wrong file name in pr_check (#15380)

2022-01-28 07:52:01 -05:00

preprocessing.mdx

Update tutorial docs (#15165)

2022-02-01 18:31:35 -06:00

quicktour.mdx

[Doctests] Move doctests to new GPU & Fix bugs (#15969)

2022-03-09 13:09:56 +01:00

run_scripts.mdx

Update training scripts docs (#15931)

2022-03-07 13:29:14 -06:00

sagemaker.mdx

Convert rst files (#14888)

2021-12-22 16:14:35 -05:00

serialization.mdx

Add ONNX export for ViT (#15658)

2022-03-09 17:36:59 +01:00

task_summary.mdx

Re-enable doctests for the quicktour (#15828)

2022-02-25 17:46:38 +01:00

testing.mdx

[doc] normalize HF Transformers string (#15023)

2022-01-10 08:44:33 -08:00

tokenizer_summary.mdx

Fix grammar in tokenizer_summary (#15614)

2022-02-11 16:51:30 -05:00

training.mdx

Update fine-tune docs (#15259)

2022-02-01 18:28:12 -06:00

troubleshooting.mdx

Update troubleshoot guide (#16001)

2022-03-11 13:05:44 -06:00