From f869d486d3aed92f75a9578bdcb272c55f97d30c Mon Sep 17 00:00:00 2001 From: Ke Wen Date: Wed, 12 Feb 2025 06:53:27 -0800 Subject: [PATCH] Update doc re list of models supporting TP (#35864) Update doc about models' TP support --- docs/source/en/perf_infer_gpu_multi.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/source/en/perf_infer_gpu_multi.md b/docs/source/en/perf_infer_gpu_multi.md index ea9421747c..7f5d52363e 100644 --- a/docs/source/en/perf_infer_gpu_multi.md +++ b/docs/source/en/perf_infer_gpu_multi.md @@ -54,6 +54,16 @@ torchrun --nproc-per-node 4 demo.py PyTorch tensor parallel is currently supported for the following models: * [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel) +* [Gemma](https://huggingface.co/docs/transformers/en/model_doc/gemma), [Gemma2](https://huggingface.co/docs/transformers/en/model_doc/gemma2) +* [Granite](https://huggingface.co/docs/transformers/en/model_doc/granite) +* [Mistral](https://huggingface.co/docs/transformers/en/model_doc/mistral) +* [Qwen2](https://huggingface.co/docs/transformers/en/model_doc/qwen2), [Qwen2MoE](https://huggingface.co/docs/transformers/en/model_doc/qwen2_moe), [Qwen2-VL](https://huggingface.co/docs/transformers/v4.48.0/en/model_doc/qwen2_vl) +* [Starcoder2](https://huggingface.co/docs/transformers/en/model_doc/starcoder2) +* [Cohere](https://huggingface.co/docs/transformers/en/model_doc/cohere), [Cohere2](https://huggingface.co/docs/transformers/en/model_doc/cohere2) +* [GLM](https://huggingface.co/docs/transformers/en/model_doc/glm) +* [Mixtral](https://huggingface.co/docs/transformers/en/model_doc/mixtral) +* [OLMo](https://huggingface.co/docs/transformers/en/model_doc/olmo), [OLMo2](https://huggingface.co/docs/transformers/en/model_doc/olmo2) +* [Phi](https://huggingface.co/docs/transformers/en/model_doc/phi), [Phi-3](https://huggingface.co/docs/transformers/en/model_doc/phi3) You can request to add tensor parallel support for another model by opening a GitHub Issue or Pull Request.