SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[docs] Improve visualization for vertical parallelism (#28583)

Browse Source 
The documentation says "We refer to this Model parallelism as “Vertical” because of how models are typically visualized.", but then visualizes the model horizontally. This change visualizes the model indeed vertically.
This commit is contained in:
Peter Götz
2024-01-25 18:55:11 +01:00
committed by GitHub
parent 4cbd876e42
commit 2875195887
1 changed files with 13 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/source/en/perf_train_gpu_many.md
View File

@@ -285,10 +285,19 @@ following diagram shows an 8-layer model split vertically into two slices, placi
GPU0 and 4-7 to GPU1: GPU0 and 4-7 to GPU1:
``` ```
=================== =================== ================
| 0 | 1 | 2 | 3 | | 4 | 5 | 6 | 7 | | Layer | |
=================== =================== | 0 | |
GPU0 GPU1 | 1 | GPU0 |
| 2 | |
| 3 | |
================
| Layer | |
| 4 | |
| 5 | GPU1 |
| 6 | |
| 7 | |
================
``` ```
In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data