From 28751958874eccb155fa2ab10a79bf8068d9ae29 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Peter=20G=C3=B6tz?= Date: Thu, 25 Jan 2024 18:55:11 +0100 Subject: [PATCH] [`docs`] Improve visualization for vertical parallelism (#28583) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The documentation says "We refer to this Model parallelism as “Vertical” because of how models are typically visualized.", but then visualizes the model horizontally. This change visualizes the model indeed vertically. --- docs/source/en/perf_train_gpu_many.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/source/en/perf_train_gpu_many.md b/docs/source/en/perf_train_gpu_many.md index 3045b98952..92c2fe9bbf 100644 --- a/docs/source/en/perf_train_gpu_many.md +++ b/docs/source/en/perf_train_gpu_many.md @@ -285,10 +285,19 @@ following diagram shows an 8-layer model split vertically into two slices, placi GPU0 and 4-7 to GPU1: ``` -=================== =================== -| 0 | 1 | 2 | 3 | | 4 | 5 | 6 | 7 | -=================== =================== - GPU0 GPU1 +================ +| Layer | | +| 0 | | +| 1 | GPU0 | +| 2 | | +| 3 | | +================ +| Layer | | +| 4 | | +| 5 | GPU1 | +| 6 | | +| 7 | | +================ ``` In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data