[Core generation] Adds support for static KV cache (#27931)

Co-authored-by: fxmarty <9808326+fxmarty@users.noreply.github.com> Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
2024-02-08 19:50:34 +09:00
parent 4b236aed76
commit 115ac94d06
19 changed files with 474 additions and 232 deletions
--- a/docs/source/en/internal/generation_utils.md
+++ b/docs/source/en/internal/generation_utils.md
@@ -373,3 +373,7 @@ A [`Constraint`] can be used to force the generation to include specific tokens
    - update
    - get_seq_length
    - reorder_cache
+
+[[autodoc]] StaticCache
+    - update
+    - get_seq_length