SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Docs - update formatting of llama3 model card (#33438)

Browse Source 
update formatting of llama3 content
This commit is contained in:
Michael Currin
2024-09-12 11:24:56 +02:00
committed by GitHub
parent d7a553b89f
commit e0ff4321d1
1 changed files with 14 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/source/en/model_doc/llama3.md
View File

@@ -78,4 +78,5 @@ come in several checkpoints they each contain a part of each weight of the model
- When using Flash Attention 2 via `attn_implementation="flash_attention_2"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type. - When using Flash Attention 2 via `attn_implementation="flash_attention_2"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.
## Resources ## Resources
A ton of cool resources are already available on the documentation page of [Llama2](./llama2), inviting contributors to add new resources curated for Llama3 here! 🤗 A ton of cool resources are already available on the documentation page of [Llama2](./llama2), inviting contributors to add new resources curated for Llama3 here! 🤗