Tony Wu
f33a0cebb3
Add ColPali to 🤗 transformers (#33736)
* feat: run `add-new-model-like`
* feat: add paligemma code with "copied from"
* feat: add ColPaliProcessor
* feat: add ColPaliModel
* feat: add ColPaliConfig
* feat: rename `ColPaliForConditionalGeneration` to `ColPaliModel`
* fixup modeling colpali
* fix: fix root import shortcuts
* fix: fix `modeling_auto` dict
* feat: comment out ColPali test file
* fix: fix typos from `add-new-model-like`
* feat: explicit the forward input args
* feat: move everything to `modular_colpali.py`
* fix: put back ColPaliProcesor
* feat: add auto-generated files
* fix: run `fix-copies`
* fix: remove DOCStRING constants to make modular converter work
* fix: fix typo + modular converter
* fix: add missing imports
* feat: no more errors when loading ColPaliModel
* fix: remove unused args in forward + tweak doc
* feat: rename `ColPaliModel` to `ColPaliForRetrieval`
* fix: apply `fix-copies`
* feat: add ColPaliProcessor to `modular_colpali`
* fix: run make quality + make style
* fix: remove duplicate line in configuration_auto
* feat: make ColPaliModel inehrit from PaliGemmaForConditionalGeneration
* fix: tweak and use ColPaliConfig
* feat: rename `score` to `post_process_retrieval`
* build: run modular formatter + make style
* feat: convert colpali weights + fixes
* feat: remove old weight converter file
* feat: add and validate tests
* feat: replace harcoded path to "vidore/colpali-v1.2-hf" in tests
* fix: add bfloat16 conversion in weight converter
* feat: replace pytest with unittest in modeling colpali test
* feat: add sanity check for weight conversion (doesn't work yet)
* feat: add shape sanity check in weigth converter
* feat: make ColPaliProcessor args explicit
* doc: add doc for ColPali
* fix: trying to fix output mismatch
* feat: tweaks
* fix: ColPaliModelOutput inherits from ModelOutput instead of PaliGemmaCausalLMOutputWithPast
* fix: address comments on PR
* fix: adapt tests to the Hf norm
* wip: try things
* feat: add `__call__` method to `ColPaliProcessor`
* feat: remove need for dummy image in `process_queries`
* build: run new modular converter
* fix: fix incorrect method override
* Fix tests, processing, modular, convert
* fix tokenization auto
* hotfix: manually fix processor -> fixme once convert modular is fixed
* fix: convert weights working
* feat: rename and improve convert weight script
* feat: tweaks
* fest: remove `device` input for `post_process_retrieval`
* refactor: remove unused `get_torch_device`
* Fix all tests
* docs: update ColPali model doc
* wip: fix convert weights to hf
* fix logging modular
* docs: add acknowledgements in model doc
* docs: add missing docstring to ColPaliProcessor
* docs: tweak
* docs: add doc for `ColPaliForRetrievalOutput.forward`
* feat: add modifications from colpali-engine v0.3.2 in ColPaliProcessor
* fix: fix and upload colapli hf weights
* refactor: rename `post_process_retrieval` to `score_retrieval`
* fix: fix wrong typing for `score_retrieval`
* test: add integration test for ColPali
* chore: rerun convert modular
* build: fix root imports
* Update docs/source/en/index.md
Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com>
* fix: address PR comments
* wip: reduce the prediction gap in weight conversion
* docs: add comment in weight conversion script
* docs: add example for `ColPaliForRetrieval.forward`
* tests: change dataset path to the new one in hf-internal
* fix: colpali weight conversion works
* test: add fine-grained check for ColPali integration test
* fix: fix typos in convert weight script
* docs: move input docstring in a variable
* fix: remove hardcoded torch device in test
* fix: run the new modular refactor
* docs: fix python example for ColPali
* feat: add option to choose `score_retrieval`'s output dtype and device
* docs: update doc for `score_retrieval`
* feat: add `patch_size` property in ColPali model
* chore: run `make fix-copies`
* docs: update description for ColPali cookbooks
* fix: remove `ignore_index` methods
* feat: remove non-transformers specific methods
* feat: update `__init__.py` to new hf format
* fix: fix root imports in transformers
* feat: remove ColPali's inheritance from PaliGemma
* Fix CI issues
* nit remove prints
* feat: remove ColPali config and model from `modular_colpali.py`
* feat: add `ColPaliPreTrainedModel` and update modeling and configuration code
* fix: fix auto-removed imports in root `__init__.py`
* fix: various fixes
* fix: fix `_init_weight`
* temp: comment `AutoModel.from_config` for experiments
* fix: add missing `output_attentions` arg in ColPali's forward
* fix: fix `resize_token_embeddings`
* fix: make `input_ids` optional in forward
* feat: rename `projection_layer` to `embedding_proj_layer`
* wip: fix convert colpali weight script
* fix tests and convert weights from original repo
* fix unprotected import
* fix unprotected torch import
* fix style
* change vlm_backbone_config to vlm_config
* fix unprotected import in modular this time
* fix: load config from Hub + tweaks in convert weight script
* docs: move example usage from model docstring to model markdown
* docs: fix input docstring for ColPali's forward method
* fix: use `sub_configs` for ColPaliConfig
* fix: remove non-needed sanity checks in weight conversion script + tweaks
* fix: fix issue with `replace_return_docstrings` in ColPali's `forward`
* docs: update docstring for `ColPaliConfig`
* test: change model path in ColPali test
* fix: fix ColPaliConfig
* fix: fix weight conversion script
* test: fix expected weights for ColPali model
* docs: update ColPali markdown
* docs: fix minor typo in ColPaliProcessor
* Fix tests and add _no_split_modules
* add text_config to colpali config
* [run slow] colpali
* move inputs to torch_device in integration test
* skip test_model_parallelism
* docs: clarify quickstart snippet in ColPali's model card
* docs: update ColPali's model card
---------
Co-authored-by: yonigozlan <yoni.gozlan@huggingface.co>
Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com>
2024-12-17 11:26:43 +01:00
..
2024-11-05 16:10:42 +01:00
2024-11-05 11:34:01 +01:00
2024-11-05 11:34:01 +01:00
2024-12-06 12:17:34 +01:00
2024-12-13 14:33:45 +01:00
2024-12-15 14:00:36 -05:00
2024-06-07 19:40:26 +01:00
2024-06-26 21:59:08 +01:00
2024-10-30 10:59:08 +00:00
2024-06-26 21:59:08 +01:00
2024-03-13 14:53:27 +01:00
2024-12-13 14:33:45 +01:00
2024-10-30 10:59:08 +00:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-10-07 18:11:13 +02:00
2024-10-09 16:46:11 +01:00
2024-08-16 11:20:41 +01:00
2024-06-19 10:18:08 +01:00
2024-08-23 11:12:53 +01:00
2024-10-09 16:46:11 +01:00
2024-11-05 11:34:01 +01:00
2024-11-25 10:41:55 +01:00
2024-09-19 12:04:24 +02:00
2024-10-02 16:43:43 +02:00
2024-10-09 16:46:11 +01:00
2024-06-26 21:59:08 +01:00
2024-11-04 16:37:51 +01:00
2024-06-26 21:59:08 +01:00
2024-10-31 15:48:11 -04:00
2024-09-25 19:05:01 +01:00
2024-12-13 14:33:45 +01:00
2024-12-13 14:33:45 +01:00
2024-11-05 11:34:01 +01:00
2024-12-13 14:33:45 +01:00
2024-06-26 21:59:08 +01:00
2024-09-24 16:40:48 +01:00
2024-12-13 09:35:50 +01:00
2024-12-13 09:35:50 +01:00
2024-12-17 11:26:43 +01:00
2024-12-13 14:33:45 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-06-26 21:59:08 +01:00
2024-10-09 16:46:11 +01:00
2024-06-26 21:59:08 +01:00
2024-10-31 16:36:13 +01:00
2024-06-19 10:18:08 +01:00
2024-12-13 14:33:45 +01:00
2024-07-23 14:54:38 +08:00
2024-07-29 10:52:13 +01:00
2024-11-25 10:43:16 +01:00
2024-11-25 10:43:16 +01:00
2024-08-23 11:12:53 +01:00
2024-11-19 11:18:58 -05:00
2024-08-06 11:33:05 +01:00
2024-11-20 11:31:21 +01:00
2024-12-15 14:00:36 -05:00
2024-06-26 21:59:08 +01:00
2024-09-21 01:58:00 +01:00
2024-11-05 13:41:48 +01:00
2023-06-29 10:17:36 +01:00
2024-10-02 16:43:43 +02:00
2024-03-25 10:33:38 +01:00
2024-11-20 11:31:21 +01:00
2024-08-27 11:58:27 +01:00
2024-10-07 18:11:13 +02:00
2024-12-13 14:33:45 +01:00
2024-10-24 18:42:03 +02:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-12-11 11:16:52 +01:00
2024-09-24 16:40:48 +01:00
2024-06-26 21:59:08 +01:00
2024-10-09 16:46:11 +01:00
2024-11-05 11:34:01 +01:00
2024-10-09 16:46:11 +01:00
2024-06-19 10:18:08 +01:00
2024-06-26 21:59:08 +01:00
2024-05-22 06:40:15 +02:00
2024-11-21 11:00:22 +01:00
2024-10-30 10:59:08 +00:00
2024-11-27 11:50:48 +01:00
2024-10-31 15:48:11 -04:00
2024-11-05 16:10:42 +01:00
2024-10-28 19:44:20 +01:00
2024-10-31 16:36:13 +01:00
2024-10-31 16:36:13 +01:00
2024-09-24 16:40:48 +01:00
2024-12-11 11:16:52 +01:00
2024-09-09 10:59:04 +02:00
2024-07-24 17:36:32 +01:00
2024-10-30 10:59:08 +00:00
2024-11-05 16:10:42 +01:00
2024-11-05 16:10:42 +01:00
2024-12-13 14:33:45 +01:00
2024-11-05 11:34:01 +01:00
2024-06-26 21:59:08 +01:00
2024-09-05 16:17:34 +02:00
2024-12-10 14:18:23 +01:00
2024-07-25 15:12:23 +02:00
2024-12-13 14:33:45 +01:00
2024-12-13 14:33:45 +01:00
2024-11-05 11:34:01 +01:00
2024-12-06 13:42:51 +00:00
2024-10-29 10:48:57 +01:00
2024-09-25 12:21:53 +01:00
2024-11-25 10:41:55 +01:00
2024-11-05 11:34:01 +01:00
2024-10-30 10:59:08 +00:00
2024-10-30 10:59:08 +00:00
2024-11-21 11:00:22 +01:00
2024-06-26 21:59:08 +01:00
2024-11-05 15:10:15 +00:00
2024-11-05 15:10:15 +00:00
2024-11-05 15:10:15 +00:00
2024-10-09 16:46:11 +01:00
2024-08-06 11:33:05 +01:00
2024-10-09 16:46:11 +01:00
2024-12-11 11:16:52 +01:00
2024-11-18 13:21:07 +01:00
2024-12-13 14:33:45 +01:00
2024-12-13 14:33:45 +01:00
2024-12-13 14:33:45 +01:00
2024-10-09 16:46:11 +01:00
2024-10-22 08:23:53 +02:00
2024-10-09 16:46:11 +01:00
2024-07-26 10:33:02 +02:00
2024-10-09 16:46:11 +01:00
2024-10-01 09:28:40 +02:00
2024-10-30 10:59:08 +00:00
2024-08-23 11:12:53 +01:00
2024-12-13 14:33:45 +01:00
2024-12-13 14:33:45 +01:00
2024-12-13 14:33:45 +01:00
2024-10-09 16:46:11 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-09-09 13:20:27 +01:00
2024-12-02 16:21:04 +01:00
2024-10-31 16:36:13 +01:00
2024-10-30 10:59:08 +00:00
2024-11-28 15:05:56 +01:00
2024-08-05 15:19:42 +01:00
2024-10-29 16:14:31 +01:00
2024-08-06 11:33:05 +01:00
2024-08-06 11:33:05 +01:00
2024-08-23 11:12:53 +01:00
2024-06-07 19:40:26 +01:00
2024-11-04 16:37:51 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-10-09 11:23:50 +01:00
2024-10-30 10:59:08 +00:00
2024-12-02 16:21:04 +01:00
2024-12-13 14:33:45 +01:00
2024-10-09 16:46:11 +01:00
2024-10-07 10:56:24 +02:00
2024-11-05 16:10:42 +01:00
2024-06-26 21:59:08 +01:00
2024-10-09 16:46:11 +01:00
2024-08-06 11:33:05 +01:00
2024-05-22 06:40:15 +02:00
2024-10-25 11:55:07 +01:00
2024-11-25 16:31:22 +01:00
2024-10-25 11:55:07 +01:00
2024-10-01 23:52:03 +02:00
2024-12-13 14:33:45 +01:00
2024-10-09 16:46:11 +01:00
2024-10-25 11:55:07 +01:00
2024-11-05 11:34:01 +01:00
2024-11-05 11:34:01 +01:00
2024-12-05 18:15:48 +01:00
2024-06-26 21:59:08 +01:00
2024-06-07 19:40:26 +01:00
2024-08-23 11:12:53 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-12-11 11:16:52 +01:00
2024-12-11 11:16:52 +01:00
2024-10-10 11:50:39 +02:00
2024-10-04 22:45:37 +02:00
2024-03-13 14:53:27 +01:00
2024-12-13 14:33:45 +01:00
2024-12-13 14:33:45 +01:00
2024-10-09 16:46:11 +01:00
2024-08-06 11:33:05 +01:00
2024-12-13 14:33:45 +01:00
2024-10-09 16:46:11 +01:00
2024-08-06 11:33:05 +01:00
2024-06-26 21:59:08 +01:00
2024-11-25 15:53:04 +01:00
2024-10-31 16:36:13 +01:00
2024-10-30 10:59:08 +00:00
2024-12-13 14:33:45 +01:00
2024-10-31 16:36:13 +01:00
2024-10-30 10:59:08 +00:00
2024-10-21 10:00:14 +02:00
2024-10-07 18:11:13 +02:00
2024-11-04 16:37:51 +01:00
2024-10-07 18:11:13 +02:00
2024-10-30 08:36:45 +00:00
2024-10-07 18:11:13 +02:00
2024-10-09 16:46:11 +01:00
2024-10-09 16:46:11 +01:00
2024-12-15 14:00:36 -05:00
2024-09-24 16:40:48 +01:00
2024-10-31 16:36:13 +01:00
2024-12-13 14:33:45 +01:00
2024-12-11 15:38:42 +01:00
2024-12-13 14:33:45 +01:00
2024-12-13 14:33:45 +01:00
2024-09-17 08:05:21 +02:00
2024-09-17 08:05:21 +02:00
2024-11-05 11:34:01 +01:00
2024-10-24 18:42:03 +02:00
2024-12-13 14:33:45 +01:00
2024-12-13 14:33:45 +01:00
2024-10-09 16:46:11 +01:00
2024-07-26 10:33:02 +02:00
2024-12-11 11:16:52 +01:00
2024-10-30 10:59:08 +00:00
2024-12-13 14:33:45 +01:00
2024-06-07 19:40:26 +01:00
2024-06-19 10:18:08 +01:00
2024-08-06 11:33:05 +01:00
2024-06-26 18:46:48 +01:00
2024-10-22 08:23:53 +02:00
2024-10-30 10:59:08 +00:00
2024-06-26 21:59:08 +01:00
2024-11-05 15:10:15 +00:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-12-11 12:40:30 +00:00
2024-11-29 11:58:11 +00:00
2024-08-06 11:33:05 +01:00
2024-11-05 15:10:15 +00:00
2024-10-30 10:59:08 +00:00
2024-07-23 14:54:38 +08:00
2024-07-23 14:54:38 +08:00
2024-12-13 14:33:45 +01:00
2024-06-19 10:18:08 +01:00
2024-11-18 13:21:07 +01:00
2024-11-05 16:10:42 +01:00
2024-08-06 11:33:05 +01:00
2024-11-18 13:21:07 +01:00
2024-10-24 18:42:03 +02:00
2024-12-15 14:00:36 -05:00
2024-05-22 06:40:15 +02:00
2024-10-07 18:11:13 +02:00
2024-09-30 18:47:13 +02:00
2024-06-07 19:40:26 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-10-03 14:01:24 +01:00
2024-10-15 11:27:54 +02:00
2024-12-13 14:33:45 +01:00
2024-10-02 10:41:08 +02:00
2024-07-23 14:54:38 +08:00
2024-09-26 19:38:20 +02:00
2024-06-17 17:29:13 +01:00
2024-07-23 14:54:38 +08:00
2024-12-13 14:33:45 +01:00
2024-11-05 11:34:01 +01:00
2024-10-31 16:36:13 +01:00
2024-10-09 16:46:11 +01:00
2024-08-28 10:26:00 +02:00
2024-10-25 11:55:07 +01:00
2024-10-09 16:46:11 +01:00
2024-10-09 16:46:11 +01:00
2024-12-13 14:33:45 +01:00
2024-06-26 21:59:08 +01:00
2024-10-30 10:59:08 +00:00
2024-10-22 15:50:54 +02:00
2022-05-03 14:42:02 +02:00