[doc] DP/PP/TP/etc parallelism (#12524)

Browse Source

* wip

* complete the doc

* missing img

* improve

* correction

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

...

This commit is contained in:

Stas Bekman

2021-07-09 17:39:09 -07:00

committed by GitHub

parent 4cdbf63c03

commit 0dcc3c86e4

12 changed files with 311 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

BIN

docs/source/imgs/parallelism-deepspeed-3d.png Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 22 KiB

BIN

docs/source/imgs/parallelism-flexflow.jpeg Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 342 KiB

BIN

docs/source/imgs/parallelism-gpipe-bubble.png Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 47 KiB

BIN

docs/source/imgs/parallelism-sagemaker-interleaved-pipeline.png Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 20 KiB

BIN

docs/source/imgs/parallelism-tp-independent-gelu.png Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 5.7 KiB

BIN

docs/source/imgs/parallelism-tp-parallel_gemm.png Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 4.4 KiB

BIN

docs/source/imgs/parallelism-tp-parallel_self_attention.png Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 162 KiB

BIN

docs/source/imgs/parallelism-tp-parallel_shard_processing.png Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 99 KiB

BIN

docs/source/imgs/parallelism-zero-dp-pp.png Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 54 KiB

BIN

docs/source/imgs/parallelism-zero.png Normal file

Binary file not shown.

Side by Side After Width:  |  Height:  |  Size: 159 KiB