Pablo Montalvo
1360801a69
Add PaliGemma (#30814)
* add new model like
* add state dict slicing + new model config
* update palma config and weights, passes vision activations
* fix
* update
* reorder loading/unpacking
* clean up
* add debug statements
* change device
* fix
* debugging
* fix noncausal mask
* fixup sdpa + causal mask
* fix activation function
* remove debug before changing modeling file
* add variants
* debug attention mask in generate
* revert to non-debug sdpa
* revert gemma modifications
* add custom language modeling
* use Processor
* add language modeling file to init
* try thin wrapper around generate
* Update
* update mask
* breakpoints galore
* remove conflict
* switch to left-padding
* add incomplete model doc
* add paligemma global files
* batch rename paligemma
* make generation match outputs and captioning
* style
* style
* remove copied from + doc
* remove more copied from
* remove copy from projector
* minor fix
* update config and style
* add readme - dummy
* CORRECT image captioning
* moving to args
* add siglip proper + fix merging image + text features
* take update_causal_mask from upstream
* remove breakpoint
* leverage AutoModel
* fix input_ids slicing
* make siglip head conditional
* remove encoder_decoder value
* remove unneeded modeling file
* add commented 4d attention mask
* FIXED generation with 4D mask
* Update src/transformers/models/siglip/modeling_siglip.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* fix left padding detection
* shuffle order of verifications
* fix missing labels for training
* fix
* vectorize merging of features, improve slicing
* improve testing before conversion
* handle merging in processor
* image token index depends on checkpoint
* add variants, save processor too
* save processors, base tokenizer off spm file
* expand model embeddings due to additional image token
* pass image processing args
* add convert rgb to siglip processor
* add \n token separately
* fix tokenizer and prompts
* fix docstrings
* change to camel
* fix casing
* debug pos_ids and sdpa
* pass and use cache_position
* add flag for newline tokenization
* Update src/transformers/models/paligemma/processing_paligemma.py
Co-authored-by: Merve Noyan <merveenoyan@gmail.com>
* simplify conversion script
* add copied from
* add precision to conversion script
* Update src/transformers/models/paligemma/modeling_paligemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* clean up
* Shift attention mask from `1:`
After discussion with @molbap
* add docs, fix quality
* quality, tied weights inheritance, and logits/label alignment
* fix more tests
* pass attn_implementation to language model correctly
* add SiglipVisionTransformer to no split modules
* skip paligemma test for sdpa dispatch to flash
* skip incompatible tests
* quality
* [broken archive maps]
* Apply suggestions
- remove archive lists
- style
- take shape of inputs_embeds for batch
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update src/transformers/utils/dummy_pt_objects.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* simplify conversion script
* add suggestions
* add suggestions
* add copied from
* fix
* move labels out
* revert
* fix
* remove placeholder labels if None
* use cache_position
* fix quality + docstrings
* fix quality
* fix paligemma 4d gemma mask incompatibility
* fix config docstring
* fix query and attn_mask dtype
---------
Co-authored-by: ArthurZucker <arthur.zucker@gmail.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Co-authored-by: Merve Noyan <merveenoyan@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
2024-05-14 22:07:15 +02:00
..
2023-11-16 11:44:36 -08:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-15 10:13:52 -08:00
2024-03-20 15:41:03 +00:00
2023-11-03 10:57:03 -04:00
2023-12-09 05:38:14 +09:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-16 08:16:58 +01:00
2023-11-03 10:57:03 -04:00
2024-04-26 16:23:44 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-06 10:38:21 -08:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-01-29 15:46:32 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2024-02-14 08:41:31 +01:00
2023-11-03 10:57:03 -04:00
2023-11-10 13:49:10 +00:00
2024-05-09 22:57:52 +02:00
2024-04-17 12:19:18 +02:00
2024-03-15 14:29:11 +01:00
2024-05-08 11:42:07 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-04-26 20:52:24 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-05-08 11:42:07 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-19 15:22:29 +01:00
2024-05-08 11:42:07 +01:00
2024-05-08 11:42:07 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-23 17:44:08 +00:00
2024-02-16 08:16:58 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-23 17:44:08 +00:00
2023-11-03 10:57:03 -04:00
2023-06-20 18:07:47 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-16 08:16:58 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-09-04 17:18:34 +01:00
2024-03-29 14:31:31 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-03-12 10:16:21 +00:00
2024-02-21 14:21:28 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-03-28 09:31:24 +00:00
2024-02-16 08:16:58 +01:00
2023-12-09 05:38:14 +09:00
2023-11-03 10:57:03 -04:00
2023-12-09 05:38:14 +09:00
2024-01-15 09:09:22 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-04-19 21:03:07 +02:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-04-22 18:30:38 +01:00
2023-11-03 10:57:03 -04:00
2024-04-15 17:03:03 +01:00
2024-05-13 15:59:46 +01:00
2023-11-03 10:57:03 -04:00
2024-02-02 08:45:00 +01:00
2023-11-03 10:57:03 -04:00
2024-04-18 11:04:02 +02:00
2024-05-14 16:32:01 +02:00
2024-02-02 08:45:00 +01:00
2023-10-30 21:42:19 +01:00
2023-11-03 10:57:03 -04:00
2024-02-12 10:48:31 -08:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-12 10:48:31 -08:00
2023-12-20 14:25:07 +05:30
2024-05-10 12:40:57 +01:00
2024-02-06 03:41:42 +01:00
2024-04-22 10:41:03 +01:00
2024-04-11 10:38:24 +02:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-04-18 10:27:58 +02:00
2023-11-28 13:19:50 +00:00
2024-03-11 09:46:24 +01:00
2023-11-03 10:57:03 -04:00
2024-02-02 08:45:00 +01:00
2023-11-03 10:57:03 -04:00
2024-02-02 08:45:00 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-08 14:13:35 -08:00
2024-02-22 11:48:01 +01:00
2024-02-22 11:48:01 +01:00
2023-11-03 10:57:03 -04:00
2024-02-02 08:45:00 +01:00
2023-11-06 19:45:03 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-01 03:53:49 +01:00
2024-03-18 13:06:12 +00:00
2024-02-12 10:48:31 -08:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2024-04-18 10:27:58 +02:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-04-17 17:59:07 +02:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-09 05:38:14 +09:00
2024-05-14 08:36:11 +02:00
2023-12-11 18:03:42 +00:00
2024-05-14 22:07:15 +02:00
2024-02-19 15:22:29 +01:00
2024-02-19 15:22:29 +01:00
2024-02-08 14:13:35 -08:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-05-02 15:30:21 +02:00
2024-04-15 14:10:59 +02:00
2023-11-03 10:57:03 -04:00
2024-02-26 08:42:24 -08:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-12 10:48:31 -08:00
2023-11-03 10:57:03 -04:00
2024-03-13 19:05:20 +00:00
2024-02-08 14:13:35 -08:00
2024-02-16 08:16:58 +01:00
2024-03-30 17:49:03 +01:00
2024-02-23 10:43:31 +01:00
2023-11-03 10:57:03 -04:00
2023-06-20 18:07:47 -04:00
2024-04-15 18:30:59 +02:00
2024-02-02 08:45:00 +01:00
2023-11-06 19:45:03 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-07-13 11:46:54 -04:00
2023-11-06 19:45:03 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-02 08:45:00 +01:00
2024-04-19 21:03:07 +02:00
2023-12-14 15:14:13 +00:00
2023-12-14 15:14:13 +00:00
2023-11-03 10:57:03 -04:00
2024-04-26 19:40:12 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-19 15:22:29 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-16 08:16:58 +01:00
2023-08-03 14:12:07 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-04-15 14:10:59 +02:00
2024-04-15 14:10:59 +02:00
2024-03-20 15:41:03 +00:00
2024-04-19 18:31:43 +01:00
2023-06-20 18:07:47 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2024-04-15 14:10:59 +02:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-06 19:45:03 +00:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-02-16 08:16:58 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-23 17:02:16 +00:00
2024-04-10 16:02:50 +02:00
2023-11-03 10:57:03 -04:00
2024-02-02 08:45:00 +01:00
2024-02-02 08:45:00 +01:00
2023-11-03 10:57:03 -04:00
2023-11-22 17:21:36 +01:00
2023-11-03 10:57:03 -04:00
2024-02-02 08:45:00 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-12-15 20:16:47 +01:00
2024-02-16 08:16:58 +01:00
2023-11-03 10:57:03 -04:00
2024-02-16 08:16:58 +01:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-01-18 13:37:34 +00:00
2023-11-03 10:57:03 -04:00
2024-04-22 18:30:38 +01:00
2024-02-02 08:45:00 +01:00
2024-02-19 15:22:29 +01:00
2023-06-20 18:07:47 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2023-11-03 10:57:03 -04:00
2024-05-08 11:42:07 +01:00
2023-11-03 10:57:03 -04:00