SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[docs] Tensor parallelism (#38241)

Browse Source 
* updates

* feedback

* badges

* fix?

* fix?

* fix?

* fix?
This commit is contained in:
Steven Liu
2025-06-26 14:40:45 -07:00
committed by GitHub
parent 84e8696cae
commit a52478253b
22 changed files with 209 additions and 206 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/en/perf_train_gpu_many.md
View File

@@ -91,6 +91,8 @@ Tensor parallelism distributes large tensor computations across multiple GPUs. T
Tensor parallelism is effective for training large models that don't fit into the memory of a single GPU. It is also faster and more efficient because each GPU can process its tensor slice in parallel, and it can be combined with other parallelism methods. Like other parallelism methods though, tensor parallelism adds communication overhead between GPUs.
Refer to the [Tensor parallelism](./perf_infer_gpu_multi) guide to learn how to use it for inference.
## Hybrid parallelism
Parallelism methods can be combined to achieve even greater memory savings and more efficiently train models with billions of parameters.