SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Update doc re list of models supporting TP (#35864)

Browse Source 
Update doc about models' TP support
This commit is contained in:
Ke Wen
2025-02-12 06:53:27 -08:00
committed by GitHub
parent 281c0c8b5b
commit f869d486d3
1 changed files with 10 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/source/en/perf_infer_gpu_multi.md
View File

@@ -54,6 +54,16 @@ torchrun --nproc-per-node 4 demo.py
PyTorch tensor parallel is currently supported for the following models: PyTorch tensor parallel is currently supported for the following models:
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel) * [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [Gemma](https://huggingface.co/docs/transformers/en/model_doc/gemma), [Gemma2](https://huggingface.co/docs/transformers/en/model_doc/gemma2)
* [Granite](https://huggingface.co/docs/transformers/en/model_doc/granite)
* [Mistral](https://huggingface.co/docs/transformers/en/model_doc/mistral)
* [Qwen2](https://huggingface.co/docs/transformers/en/model_doc/qwen2), [Qwen2MoE](https://huggingface.co/docs/transformers/en/model_doc/qwen2_moe), [Qwen2-VL](https://huggingface.co/docs/transformers/v4.48.0/en/model_doc/qwen2_vl)
* [Starcoder2](https://huggingface.co/docs/transformers/en/model_doc/starcoder2)
* [Cohere](https://huggingface.co/docs/transformers/en/model_doc/cohere), [Cohere2](https://huggingface.co/docs/transformers/en/model_doc/cohere2)
* [GLM](https://huggingface.co/docs/transformers/en/model_doc/glm)
* [Mixtral](https://huggingface.co/docs/transformers/en/model_doc/mixtral)
* [OLMo](https://huggingface.co/docs/transformers/en/model_doc/olmo), [OLMo2](https://huggingface.co/docs/transformers/en/model_doc/olmo2)
* [Phi](https://huggingface.co/docs/transformers/en/model_doc/phi), [Phi-3](https://huggingface.co/docs/transformers/en/model_doc/phi3)
You can request to add tensor parallel support for another model by opening a GitHub Issue or Pull Request. You can request to add tensor parallel support for another model by opening a GitHub Issue or Pull Request.