SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[Docs] Add 4-bit serialization docs (#28182)

Browse Source 
* add 4-bit serialization docs

* up

* up
This commit is contained in:
Younes Belkada
2023-12-22 10:18:32 +01:00
committed by GitHub
parent 3657748b4d
commit 3a8769f6a9
1 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/source/en/quantization.md
View File

@@ -345,7 +345,7 @@ model_4bit = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", load_in_4
model_4bit.model.decoder.layers[-1].final_layer_norm.weight.dtype
```
Once a model is quantized to 4-bit, you can't push the quantized weights to the Hub.
If you have `bitsandbytes>=0.41.3`, you can serialize 4-bit models and push them on Hugging Face Hub. Simply call `model.push_to_hub()` after loading it in 4-bit precision. You can also save the serialized 4-bit models locally with `model.save_pretrained()` command.
</hfoption>
</hfoptions>
@@ -468,6 +468,7 @@ Try 4-bit quantization in this [notebook](https://colab.research.google.com/driv
This section explores some of the specific features of 4-bit models, such as changing the compute data type, using the Normal Float 4 (NF4) data type, and using nested quantization.
#### Compute data type
To speedup computation, you can change the data type from float32 (the default value) to bf16 using the `bnb_4bit_compute_dtype` parameter in [`BitsAndBytesConfig`]: