VLMs: patch_size -> num_image_tokens in processing (#33424)

* use num additional tokens

* fix copies + docs

* another fix copies :)

* add docs

* move order for BC
This commit is contained in:
Raushan Turganbay
2024-11-18 13:21:07 +01:00
committed by GitHub
parent 3ee24e2208
commit 1646ffb4d1
17 changed files with 131 additions and 15 deletions

View File

@@ -35,6 +35,10 @@ The original code can be found [here](https://github.com/salesforce/LAVIS/tree/m
- The model was trained by sampling 4 frames per video, so it's recommended to sample 4 frames
> [!NOTE]
> BLIP models after release v4.46 will raise warnings about adding `processor.num_query_tokens = {{num_query_tokens}}` and expand model embeddings layer to add special `<image>` token. It is strongly recommended to add the attributes to the processor if you own the model checkpoint, or open a PR if it is not owned by you. Adding these attributes means that BLIP will add the number of query tokens required per image and expand the text with as many `<image>` placeholders as there will be query tokens. Usually it is around 500 tokens per image, so make sure that the text is not truncated as otherwise there wil be failure when merging the embeddings.
The attributes can be obtained from model config, as `model.config.num_query_tokens` and model embeddings expansion can be done by following [this link](https://gist.github.com/zucchini-nlp/e9f20b054fa322f84ac9311d9ab67042).
## InstructBlipVideoConfig
[[autodoc]] InstructBlipVideoConfig