Files
HuggingFace_transformer/docs/source/en/model_doc
Shuming Hu bf607f6d3b PerceptionLM (#37878)
* plm template

* A working plm with fixed image features

* hacked processor

* First version that reproduced PLM output using PE from timm.

* Simplify and fix tie_word_embeddings

* Use PIL resize. Simplify converstion.

* First version that works with video input.

* simplifed image preprocessing (not batched)

* Minor fixes after rebasing on main.

* Video processor based on new API.

* Revert to use _preprocess for image processor.

* refactor with modular

* fix tie_word_embedding

* Testing with timm PE

* check in missed converstion from modular to model.py

* First working version of PLM with Eva PE. PLM-1B and 3B outputs are exactly the same as before. PLM-8B output has some differences.

* address review comments

* Fixed batching if video and image examples mixed.

* Simplify PE configuration.

* Enable AutoModel for PerceptionEncoder.

* Update PE config style.

* update all headers

* Minor fixes.

* Move lm_head to PerceptionLMForConditionalGeneration.
Fix vit_G model specification.

* Fix for testing_modeling_perception_lm.py

* Image processing refactoring to use more common parts.

* Fix processor test.

* update tests to use model from hub

* More test fixes.

* integration test GT update after rebasing; probably due to video preprocessing

* update test media path to hub

* Stop tracking local scripts

* address some review comments

* refactor image processing.

* small fixes

* update documentation and minor fixes

* remove scripts

* Minor fix for CI

* Fix image processing

* CI and doc fix

* CI formatting fix

* ruff fix

* ruff formatting

* ran utils/sort_auto_mappings.py

* update docstring

* more docstring udpates

* add vision_input_type default fallback for image processing

* more verbose variable naming

* test update

* Remove PE and PEConfig use AutoModel(TimmWrapper) instead

* Minor cleanup.

* Minor Fix: remove any ref to PE. Ruff format and check.

* fix docstring

* Fix modular/model consistency.Improvex docstringfor  .

* Fix PerceptionLMForConditionalGenerationModelTest

* ruff fix

* fix for check_repo

* minor formatting

* dummy size arg to fix for processor test.

* Update docstring for PerceptionLMConfig

* Minor fixes from review feedback.

* Revert some minor changes per reviewer feedback.

* update base_model_prefix

* address reviewer feedback

* fix comment in modeling file

* address reviewer feedback

* ruff format

* Pre-merge test update.

* reapply modular and fix checkpoint name

* processor test path

* use modular a bit more

* remove dead code

* add token decorator

---------

Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co>
Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>
2025-07-11 11:07:32 +02:00
..
2025-07-08 11:53:21 +02:00
2025-06-13 11:07:09 +00:00
2025-06-24 15:05:29 +02:00
2025-06-05 14:36:54 -07:00
2025-06-26 11:04:23 +00:00
2025-06-13 11:07:09 +00:00
2025-06-16 10:46:30 -07:00
2025-06-18 16:01:25 -07:00
2025-03-03 10:33:46 -08:00
2025-05-27 11:51:41 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-05-23 13:03:47 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-20 13:46:19 -07:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-17 19:37:18 +01:00
2025-06-13 11:07:09 +00:00
2025-06-10 09:30:05 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-02 12:58:01 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-05-27 17:03:55 +02:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-17 19:37:18 +01:00
2025-07-03 10:02:58 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-12 10:26:50 -07:00
2025-07-08 11:44:29 +02:00
2025-06-25 11:38:25 +02:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-07-02 12:25:26 +01:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-05-21 10:43:11 +02:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-07 17:47:51 +02:00
2025-06-26 14:40:45 -07:00
2025-07-07 14:41:33 +02:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-07-08 08:22:04 +02:00
2025-07-08 08:22:04 +02:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-04 09:56:47 -07:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-07-07 15:56:57 -07:00
2025-06-13 11:07:09 +00:00
2025-07-10 16:07:33 +02:00
2025-06-17 18:10:23 +02:00
2025-06-13 11:07:09 +00:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-04-15 11:33:09 +01:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-05-27 16:24:36 -07:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-07-11 11:07:32 +02:00
2025-03-03 10:33:46 -08:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-26 14:40:45 -07:00
2025-06-26 14:40:45 -07:00
2025-03-31 09:50:49 +02:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-17 11:02:18 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-25 15:12:15 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-04-30 12:15:43 +01:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-21 15:35:22 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-07-02 07:56:29 -07:00
2025-04-15 13:16:05 -07:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-06-13 11:07:09 +00:00
2025-06-13 11:07:09 +00:00
2025-03-03 10:33:46 -08:00
2025-03-03 10:33:46 -08:00