SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

[docs] add a comment that offloading requires CUDA GPU (#35055)

Browse Source 
* add commen to offloading

* Update docs/source/en/kv_cache.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
This commit is contained in:
Fanli Lin
2024-12-04 23:48:34 +08:00
committed by GitHub
parent 1da1e0d7f2
commit baa3b22137
1 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/source/en/kv_cache.md
View File

@@ -180,7 +180,7 @@ Fun fact: The shortest war in history was between Britain and Zanzibar on August
<Tip warning={true}> <Tip warning={true}>
Cache offloading requires a GPU and can be slower than dynamic KV cache. Use it if you are getting CUDA out of memory errors. Cache offloading requires a CUDA GPU and can be slower than dynamic KV cache. Use it if you are getting CUDA out of memory errors.
</Tip> </Tip>
@@ -261,6 +261,7 @@ This will use the [`~OffloadedStaticCache`] implementation instead.
>>> tokenizer.batch_decode(out, skip_special_tokens=True)[0] >>> tokenizer.batch_decode(out, skip_special_tokens=True)[0]
"Hello, my name is [Your Name], and I am a [Your Profession] with [Number of Years] of" "Hello, my name is [Your Name], and I am a [Your Profession] with [Number of Years] of"
``` ```
Cache offloading requires a CUDA GPU.
### Sliding Window Cache ### Sliding Window Cache