From ade9e0fe41a414c6a24a03a79c15798db609a6c9 Mon Sep 17 00:00:00 2001 From: Alan Kashkash Date: Thu, 26 Sep 2024 01:20:51 +0800 Subject: [PATCH] Corrected max number for bf16 in transformer/docs (#33658) Update perf_train_gpu_one.md per issue https://github.com/huggingface/hub-docs/issues/1425 max number for bf16 should be 65,504 not 65,535 --- docs/source/en/perf_train_gpu_one.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/en/perf_train_gpu_one.md b/docs/source/en/perf_train_gpu_one.md index 364fc46544..5d7ae312ee 100644 --- a/docs/source/en/perf_train_gpu_one.md +++ b/docs/source/en/perf_train_gpu_one.md @@ -186,7 +186,7 @@ If you prefer to use 🤗 Accelerate, find the 🤗 Accelerate example [further If you have access to an Ampere or newer hardware you can use bf16 for mixed precision training and evaluation. While bf16 has a worse precision than fp16, it has a much bigger dynamic range. In fp16 the biggest number you can have -is `65535` and any number above that will result in an overflow. A bf16 number can be as large as `3.39e+38` (!) which +is `65504` and any number above that will result in an overflow. A bf16 number can be as large as `3.39e+38` (!) which is about the same as fp32 - because both have 8-bits used for the numerical range. You can enable BF16 in the 🤗 Trainer with: