Nikos Karampatziakis ca59d6f77c Offloaded KV Cache (#31325) ...

* Initial implementation of OffloadedCache

* enable usage via cache_implementation

* Address feedback, add tests, remove legacy methods.

* Remove flash-attn, discover synchronization bugs, fix bugs

* Prevent usage in CPU only mode

* Add a section about offloaded KV cache to the docs

* Fix typos in docs

* Clarifications and better explanation of streams

2024-08-01 14:42:07 +02:00

26 KiB

Raw Blame History

View Raw