Ryan Mullins
487dab1b2b
Shieldgemma2 ( #36678 )
...
* single commit
* correct config
* fixup
* dummy pt
* Use ShieldGemma2Config in conversion script
* Update src/transformers/models/shieldgemma2/configuration_shieldgemma2.py
* Adding shieldgemma2 to models.__init__.py
* Adding ShieldGemma2 to main __init__.py
* Update shieldgemma2.md
* Update shieldgemma2.md
* Adding tests. Addressing review feedback.
* Minor docs update
* Fixing code quality feedback from CI
* Fixing empty messages bug reported by ghunkins
---------
Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com >
Co-authored-by: Ren Pang <ain-soph@live.com >
2025-03-20 15:14:38 +01:00
Joao Gante
957b05b413
[qwen2 audio] remove redundant code and update docs ( #36282 )
2025-03-20 10:54:51 +00:00
HDCharles
94555437e2
Disable inductor config setter by default ( #36608 )
...
* Disable inductor config setter by default
This is hard to debug and should be off by default
* remove default settings in autoquant too
* Add info to torchao.md about recommended settings
* satisfying Ruff format
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
2025-03-20 11:23:14 +01:00
Matt
9be4728af8
Just import torch AdamW instead ( #36177 )
...
* Just import torch AdamW instead
* Update docs too
* Make AdamW undocumented
* make fixup
* Add a basic wrapper class
* Add it back to the docs
* Just remove AdamW entirely
* Remove some AdamW references
* Drop AdamW from the public init
* make fix-copies
* Cleanup some references
* make fixup
* Delete lots of transformers.AdamW references
* Remove extra references to adamw_hf
2025-03-19 18:29:40 +00:00
Mohamed Mekkouri
258dd9cc69
Add Space to Bitsandbytes doc ( #36834 )
...
* add space
* address review
2025-03-19 18:56:07 +01:00
Driss Guessous
e8d960329e
Add option for ao base configs ( #36526 )
2025-03-19 14:59:47 +01:00
Yoni Gozlan
12f2ebef63
Support custom dosctrings in modular ( #36726 )
...
* Override docstrings in modular if not none
* Update doc
2025-03-18 14:00:54 -04:00
Yoni Gozlan
30580f035b
Fix Mistral3 tests ( #36797 )
...
* fix processor tests
* fix modeling tests
* fix test processor chat template
* revert modeling test changes
2025-03-18 13:08:12 -04:00
Cyril Vallez
e959530b8f
Add Mistral3 ( #36790 )
...
Release - Conda / build_and_package (push) Has been cancelled
Secret Leaks / trufflehog (push) Has been cancelled
* initial start
* style and dummies
* Create convert_mistral3_weights_to_hf.py
* update
* typo
* typo
* Update convert_mistral3_weights_to_hf.py
* Update convert_mistral3_weights_to_hf.py
* Update convert_mistral3_weights_to_hf.py
* Update convert_mistral3_weights_to_hf.py
* up
* Update convert_mistral3_weights_to_hf.py
* Update convert_mistral3_weights_to_hf.py
* update
* update
* Update image_processing_mistral3.py
* Update convert_mistral3_weights_to_hf.py
* fix patch merger
* Update convert_mistral3_weights_to_hf.py
* Update convert_mistral3_weights_to_hf.py
* up
* update modular to fit
* style
* Update convert_mistral3_weights_to_hf.py
* typo
* Update modular_mistral3.py
* simplify a lot all shape shenanigans
* simplify
* add working test processor
* Add partially working common modeling tests
* All tests working and remove mistral3 image processors
* add docs and fixup
* fix inference with image size >1540
* 🚨 fix test image proc pixtral
* Remove vision_feature_select_strategy
* Update convert_mistral3_weights_to_hf.py
* Update convert_mistral3_weights_to_hf.py
* Update convert_mistral3_weights_to_hf.py
* Update convert_mistral3_weights_to_hf.py
* clean
* fix test checkpoints
* Update test_modeling_mistral3.py
* Update test_modeling_mistral3.py
* style
* Use Pixtral processor
* up
* finish cleaning processor to use pixtral directly
* Update __init__.py
* Update processing_pixtral.py
* doc
* Update __init__.py
* Update mistral3.md
* Update _toctree.yml
---------
Co-authored-by: yonigozlan <yoni.gozlan@huggingface.co >
Co-authored-by: yonigozlan <yoni.gozlan10@gmail.com >
2025-03-18 12:04:42 +01:00
Steven Liu
ac1a1b66b9
[docs] Update README ( #36265 )
...
* update
* feedback
* feedback
* update versions
2025-03-17 09:37:19 -07:00
Christopher Akiki
e3af4fec91
[MINOR:TYPO] Update hubert.md ( #36733 )
...
* [MINOR:TYPO] Update hubert.md
- typo fix (wave2vec instead of hubert)
- make code snippet copiable and runnable
* Run tests
2025-03-17 09:07:51 -07:00
MaCAT
25992b493c
🌐 [i18n-KO] Translated codegen.md to Korean ( #36698 )
...
* Initial translation
* Add _toctree.yml
2025-03-14 09:31:18 -07:00
Yoni Gozlan
69bc848480
Add support for fast image processors in add-new-model-like CLI ( #36313 )
...
* add support for fast image processors in add-new-model-like
* fix header not found add-fast-image-processor-cli
* Encourage adding fast image processor
* nit
* start improve doc
* update docs
* make requested modifs
2025-03-13 14:16:37 -04:00
Arthur
2829013d2d
fix block mask typing ( #36661 )
...
* fix block mask typing
* updated
Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com >
* gemma
* fix
---------
Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com >
2025-03-12 11:29:11 +01:00
Ryan Mullins
50d3530aa0
Gemma3 ( #36658 )
...
* Fix converter
* [Broken] Adds Gemma 3 to Hugging Face Transformers
* Consolidating Config and Processor params across impls
* Sorting out configuration parameters. Adds qk_norm before RoPE. Still not sure if RoPE is right.
* Additional plumbing for CausalLM and ConditionalGeneration variants
* incomplete draft of Orbax conversion script
* More complete checkpoint conversion
* Supporting Gemma 3 1B checkpoints
* Updating RoPE for multiple frequencies
* Adjustments to rotary embedder
* Proof of life for text-only operation
* Updating the conversion script to handle multimodal projection weights
* Fixing tet-only conversions
* Cleaner conversion script with multimodal support and a simpler processor
* Additional refatcors to the Gemma3Processor
* Simplified Processor to work over text representations
* Updated conversion script to join text and vision embeddings at converion time
* Logging for debugging
* Update src/transformers/models/gemma2/modeling_gemma2.py
Co-authored-by: Joshua Lochner <admin@xenova.com >
* Removed extraneous Config params
* Switching to fast tokenizer for checkpoint conversions
* isolating siglip for performance tetsing
* Minor changes for debugging tests against baselines
* Adding average pooling for soft tokens
* Updating processor code to enable simpler embedding interleaving for arbitrary number of images in prompts
* Updating conversion script for ShieldGemma 2 conversion compatibility
* Allow disable_compile to be provided as a kwarg
* Refresh from modular
* Updated conversion script and corrected sliding window
* Fix type mismatch in cache_position (#4 )
* Fix dtype (#5 )
* Fix type mismatch in cache_position
* Actually fix in the modular file
Co-authored-by: Aritra Roy Gosthipaty <aritra.born2fly@gmail.com >
---------
Co-authored-by: Aritra Roy Gosthipaty <aritra.born2fly@gmail.com >
* fixes for embedding table overflow and missing image_soft_token_mask from Gemma3Processor
* Adding 2D pooling for image embeddings
* Revert "Adding 2D pooling for image embeddings"
This reverts commit 65350cf531296f050b2078a5b8e46f61642b2648.
* Gemma3 average pooling changed from 1D to 2D
* Major refactor to Gemma3MultimodalInputProjection
* Updating Gemm 3 Auto* registrations
* Add option to save Gemma 3 chat template with tokenizer during weights conversion
* Removing unused imports
* Moving out-of-vocab handling from Gemma3Processor to Gemma3ForConditionalGeneration
* Removing duplicate config property
* Removing final logit softcapping and 1-indexing of position ids
* Fixing image processor config and none --> None typo
* Fixing sliding window size for 1B
* Updating image_mean and image_std in Image Processor
* Attention masking changed to lower triangular
* Moving image special tokens to conversion script
* Mirror image processor defaults from conversion script into Gemma3ProcessorKwargs
* Remove special token variables from symbol space
* Moving image soft token mask computation from Gemma3Processor to Gemma3ForConditionalGeneration
* tie lm_head and embedding weights
Co-authored-by: Matthew Douglas <38992547+matthewdouglas@users.noreply.github.com >
* Correct tied weights in Gemma3CausalLM
* iterative bidirectional attention
* resolving merge conflicts
* Reverting to Gemma 2 HybridCache with sldiing window support and a sliding_window_pattern of 6
* Correcting RoPE scaling
* clean up first pass, dummy model geenration works
* final clean up before fixing tests
* causal lm test works, so fine
* Fix conversion
* Update src/transformers/models/gemma3/processing_gemma3.py
* model tests are happy
* processor tests are happy
* image processing tests added
* fixup
* Fix pre-processing in conversion
* Inputs merging
* Do not normalize vision embeddings
* Apply Ryan's (and team) changes to attention
* token type ids + mask
* template
* move embed scale, add rope scale, fix tests
* Add chat template to tokenizer
* Use prefix for causal model loading
* use existing code for sliding mask from gemma2
* self.embed_tokens already normalizes
* Correcting Gemma3TextConfig parameters in conversion script
* typo, modular overwrites my fixes
* enable device map for text model
* Conversion updates
* ultra nit: no einsums
* update image token
* copy deepcopy config + some docs
* add some test, still WIP
* Refactoring --include_chat_tempalte logic in converter
* Update src/transformers/models/gemma3/modular_gemma3.py
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com >
* Add eos tokens for instruct models
* dump so i can work on dgx
* Removing add_bos by default
* dump
* add fast im proc
* docs for PaS + fixup
* another fixup
* one more fixup
* fix tests
* Inverting prior BOS change
* ultra nit
* Reverting to Tokenizer saved with add_bos_token=True and chat template starting with BOS
* resize embeds, remove sqrt, add slow test outputs
* FA2 but quality is meh
* nit
* skip FA2, no idea what happened
* last bit for green CI
* please, green CI for docs
* T_T
* Fix for Gemma3 logits
* Support both options for system prompt
* Update src/transformers/models/gemma3/image_processing_gemma3_fast.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* Update docs/source/en/model_doc/gemma3.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* Update docs/source/en/model_doc/gemma3.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* Update docs/source/en/model_doc/gemma3.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* Update docs/source/en/model_doc/gemma3.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* Update docs/source/en/model_doc/gemma3.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* Docs updates now that assets are live
* Style fixes
---------
Co-authored-by: Joshua Lochner <admin@xenova.com >
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
Co-authored-by: Aritra Roy Gosthipaty <aritra.born2fly@gmail.com >
Co-authored-by: Mayank Chaturvedi <imayank@google.com >
Co-authored-by: Matthew Douglas <38992547+matthewdouglas@users.noreply.github.com >
Co-authored-by: raushan <raushan@huggingface.co >
Co-authored-by: Raushan Turganbay <raushan.turganbay@alumni.nu.edu.kz >
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com >
Co-authored-by: Lysandre <hi@lysand.re >
2025-03-12 09:06:17 +01:00
Afanti
81aa9b2e07
fix typos in the docs directory ( #36639 )
...
* chore: fix typos in the docs directory
* chore: fix typos in the docs directory
* chore: fix typos in the docs directory
2025-03-11 09:41:41 -07:00
Marc Sun
cb384dcd7a
Fix gguf docs ( #36601 )
...
* update
* doc
* update
* Update docs/source/en/gguf.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* fix
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
2025-03-11 15:29:14 +01:00
Matt
1e4286fd59
Remove research projects ( #36645 )
...
* Remove research projects
* Add new README to explain where the projects went
* Trigger tests
* Cleanup all references to research_projects
2025-03-11 13:47:38 +00:00
Steven Liu
e9756cdbc7
[docs] Serving LLMs ( #36522 )
...
* initial
* fix
* model-impl
2025-03-10 13:14:19 -07:00
Krishnakumar Kannan
1b9978c360
Update chat_extras.md with content correction ( #36599 )
...
Update chat_extras.md - content
Fixed a typo in the content, that may confuse the readers.
2025-03-07 13:09:02 +00:00
Nouamane Tazi
51ed61e2f0
Mention UltraScale Playbook 🌌 in docs ( #36589 )
2025-03-06 14:48:11 -08:00
Aritra Roy Gosthipaty
159445d044
fix: argument ( #36558 )
...
752ef3fd4e/utils/modular_model_converter.py (L1729)
2025-03-06 13:11:19 -08:00
Shaohon Chen
0440dbc0e1
Integrate SwanLab for offline/online experiment tracking and local visualization ( #36433 )
...
* add swanlab integration
* feat(integrate): add SwanLab as an optional experiment tracking tool in transformers
- Integrated SwanLab into the transformers library as an alternative for experiment tracking.
- Users can now log training metrics, hyperparameters, and other experiment details to SwanLab by setting `report_to="swanlab"` in the `TrainingArguments`.
- Added necessary dependencies and documentation for SwanLab integration.
* Fix the spelling error of SwanLabCallback in callback.md
* Apply suggestions from code review
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
* Fix typo in comment
* Fix typo in comment
* Fix typos and update comments
* fix annotation
* chore: opt some comments
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
Co-authored-by: AAssets <20010618@qq.com >
Co-authored-by: ZeYi Lin <944270057@qq.com >
Co-authored-by: KAAANG <79990647+SAKURA-CAT@users.noreply.github.com >
2025-03-06 17:35:30 +01:00
Mohamed Mekkouri
89d27fa6ff
Fix links in quantization doc ( #36528 )
...
fix quantization doc
2025-03-04 16:43:03 +01:00
co63oc
37508816d6
chore: Fix typos in docs and examples ( #36524 )
...
Fix typos in docs and examples
Signed-off-by: co63oc <co63oc@users.noreply.github.com >
2025-03-04 13:47:41 +00:00
Arthur
84f0186e89
Add aya ( #36521 )
...
Release - Conda / build_and_package (push) Has been cancelled
Secret Leaks / trufflehog (push) Has been cancelled
* initial commit
* small fix
* move stuff to image processing file
* remove stuff in validate turn and fix return tensor
* remove liquid stuff
* in the process of addressing comments
* changes to get the right tokenization
* new __init__ works
* fixing defulat std and mean
* works
* small testing scipt -- to be deleted before merge
* remove redundant code
* addressing comments
* fix inits, add docs templates
* refactor processor, switch to gotocr image processor
* remove image proc from init
* refactor to working llava-style architecture
* Change AyaVisionModel to AyaVisionForConditionalGeneration
* add tests
* fixups
* update doc
* Adding logits_to_keep explicitly in ayavision forward to enable compatibility with cohere model
* better variable names + remove code paths
* Updates to aya_vision.md
* address comments
* adding copied from
* make style and remove unused projector_hidden_act from config
* sort init
* include usage of fast image proc and proc on cuda in doc
* update checkpoint iin test processor
* update checkpoint in test processor 2
* remove test_model and update docstring
* skip failing tests
---------
Co-authored-by: Saurabh Dash <saurabh@cohere.com >
Co-authored-by: yonigozlan <yoni.gozlan@huggingface.co >
2025-03-04 12:24:33 +01:00
Steven Liu
c0f8d055ce
[docs] Redesign ( #31757 )
...
* toctree
* not-doctested.txt
* collapse sections
* feedback
* update
* rewrite get started sections
* fixes
* fix
* loading models
* fix
* customize models
* share
* fix link
* contribute part 1
* contribute pt 2
* fix toctree
* tokenization pt 1
* Add new model (#32615 )
* v1 - working version
* fix
* fix
* fix
* fix
* rename to correct name
* fix title
* fixup
* rename files
* fix
* add copied from on tests
* rename to `FalconMamba` everywhere and fix bugs
* fix quantization + accelerate
* fix copies
* add `torch.compile` support
* fix tests
* fix tests and add slow tests
* copies on config
* merge the latest changes
* fix tests
* add few lines about instruct
* Apply suggestions from code review
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com >
* fix
* fix tests
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com >
* "to be not" -> "not to be" (#32636 )
* "to be not" -> "not to be"
* Update sam.md
* Update trainer.py
* Update modeling_utils.py
* Update test_modeling_utils.py
* Update test_modeling_utils.py
* fix hfoption tag
* tokenization pt. 2
* image processor
* fix toctree
* backbones
* feature extractor
* fix file name
* processor
* update not-doctested
* update
* make style
* fix toctree
* revision
* make fixup
* fix toctree
* fix
* make style
* fix hfoption tag
* pipeline
* pipeline gradio
* pipeline web server
* add pipeline
* fix toctree
* not-doctested
* prompting
* llm optims
* fix toctree
* fixes
* cache
* text generation
* fix
* chat pipeline
* chat stuff
* xla
* torch.compile
* cpu inference
* toctree
* gpu inference
* agents and tools
* gguf/tiktoken
* finetune
* toctree
* trainer
* trainer pt 2
* optims
* optimizers
* accelerate
* parallelism
* fsdp
* update
* distributed cpu
* hardware training
* gpu training
* gpu training 2
* peft
* distrib debug
* deepspeed 1
* deepspeed 2
* chat toctree
* quant pt 1
* quant pt 2
* fix toctree
* fix
* fix
* quant pt 3
* quant pt 4
* serialization
* torchscript
* scripts
* tpu
* review
* model addition timeline
* modular
* more reviews
* reviews
* fix toctree
* reviews reviews
* continue reviews
* more reviews
* modular transformers
* more review
* zamba2
* fix
* all frameworks
* pytorch
* supported model frameworks
* flashattention
* rm check_table
* not-doctested.txt
* rm check_support_list.py
* feedback
* updates/feedback
* review
* feedback
* fix
* update
* feedback
* updates
* update
---------
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com >
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com >
Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com >
2025-03-03 10:33:46 -08:00
co63oc
acb8586dd9
Fix some typos in docs ( #36502 )
...
Co-authored-by: Matt <Rocketknight1@users.noreply.github.com >
2025-03-03 17:53:53 +00:00
Yoni Gozlan
2c5d038f92
Add Got-OCR 2 Fast image processor and refactor slow one ( #36185 )
...
* refactor image processor slow got ocr
* add working image processor fast
* fix fast image processor, update doc
* use one big loop for processing patches
2025-03-01 00:56:00 -05:00
Fanli Lin
51083d1bac
[docs] fix bug in deepspeed config ( #36081 )
...
bug fix
2025-02-28 07:09:54 -08:00
Nicolas Patry
b4965cecc5
Fixing the docs corresponding to the breaking change in torch 2.6. ( #36420 )
2025-02-26 14:11:52 +01:00
Aymeric Roucher
9a217fc327
Deprecate transformers.agents ( #36415 )
2025-02-26 11:38:47 +01:00
jiqing-feng
9d6abf9778
enable torchao quantization on CPU ( #36146 )
...
* enable torchao quantization on CPU
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix int4
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix format
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* enable CPU torchao tests
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix cuda tests
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix cpu tests
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* update tests
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix style
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix cuda tests
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix torchao available
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix torchao available
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix torchao config cannot convert to json
* fix docs
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* rm to_dict to rebase
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* limited torchao version for CPU
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix format
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix skip
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix format
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* Update src/transformers/testing_utils.py
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
* fix cpu test
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix format
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
---------
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
Co-authored-by: Mohamed Mekkouri <93391238+MekkCyber@users.noreply.github.com >
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
2025-02-25 11:06:52 +01:00
Jerry Zhang
2af272c101
Add autoquant support for torchao quantizer ( #35503 )
...
* Add autoquant support for torchao quantizer
Summary:
att, also verified that autoquantized model can be saved and loaded:
save: https://gist.github.com/jerryzh168/01d367aaf44dbbbfd4068a4a10a00061
load: https://gist.github.com/jerryzh168/d5c6c401b2abdf18e0b6771341f1525c
Test Plan:
tested locally with above script
model uploaded to https://huggingface.co/jerryzh168/llama3-8b-autoquant
Reviewers:
Subscribers:
Tasks:
Tags:
* add test
* ruff fix
* ruff reformat
* add docs and min_sqnr support
* format
* format
* fix test
* update doc
* format
* remove disable_compile
* format
2025-02-24 15:54:16 +01:00
Pavel Iakubovskii
a957b7911a
Add SigLIP 2 ( #36323 )
...
Release - Conda / build_and_package (push) Has been cancelled
Secret Leaks / trufflehog (push) Has been cancelled
* Docs
* Inits
* Auto classes
* Add siglip base
* Add base tests
* Fix Siglip V1 for fix res version
* Add image processor
* Update conversion
* Experimenting with vectorized embeddings
* Fixup
* Add modular Siglip2Processor
* Add modular configuration
* Rename num patches
* Correct image and text features merging
* Working conversion script
* Refactoring conversion script
* Remove unused code in conversion script
* Shorten dict a bit
* Refactoring conversion
* Done conversion refactoring
* Fixup
* Modular siglip2
* Make model exportable and compilable without graph breaks
* Remove position_ids from image_processor
* REmove position ids from modeling file
* Update modular
* Type hint
* Fixup
* Set defaults to processor
* Add integration test
* Revert spatial shapes back to tensor
* Change order
* Fix most of the tests
* Fix docstring
* Remove interpolate_pos_encoding arg (not needed)
* Update docs
* Standardize processing
* Fix attention_mask in vision head
* Siglip v1: remove double transpose in FA2
* Update modular file
* Update FA2 test
* Update expected logits
* Fix interpolation for siglip2 image processor
* Skip init test
* Skip dispatch on flash test
* Fix modeling tests
* Fixup
* Add dummy objects
* Fix some docstrings
* Add siglip2 in index.md
* Fix consistency
* Add docs
* Remove size and data format
* Add image processor tests
* Fix
* Add fast image processor
* Fix style
* Fix
* Docs
* Set lowercase for tokenizer
* Adjust head size for Siglip v1
* Update siglip2 for consistency with siglip1
* Update siglip2 conversion
* Update pipeline
* Update checkpoints in tests
* Update checkpoint name
* Fix pooling for image classification model
* Fix FA2 test
* Update processor
* Fix check repo
* Update docs
* Fix typos
* Fix docstring for fast image processor
* Add siglip2 to FA2 docs
* Fix fast ip tests
* Fix constitency
* Fix tokenizer class for siglip v1
* Fix missing header
* Refactor scaling for clip, siglip, siglip2
* Remove unused imports
* Make fast IP default for siglip2
* Update docs
* Update checkpoints
* Update modular
* Update paper link
* Fixup
* Fix name in toctree
* Fix test
2025-02-21 09:04:19 +00:00
Joao Gante
27d1707586
[smolvlm] make CI green ( #36306 )
...
* add smolvlm to toctree
* add requirements
* dev-ci
* no docker changes
* dev-ci
* update torch-light.dockerfile
* derp
* dev-ci
2025-02-20 18:56:11 +01:00
12v
5412ff1a13
Fix typo in Pixtral example ( #36302 )
...
Fix typo
2025-02-20 14:13:48 +00:00
Orr Zohar
4397dfcb71
SmolVLM2 ( #36126 )
...
Release - Conda / build_and_package (push) Has been cancelled
Secret Leaks / trufflehog (push) Has been cancelled
* smolvlm init
* updates
* fixing bugs
* minimal run, no checks
* minimal run, no checks
* passing first check + adding url support
* updating video dataloading logic
* fixing image logic
* trying modular, but fails
* modular is working, changing processor to match PR comments and general transformers logic
* fixing kwargs
* offloading video loading logic to image_util
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* update
* add idefics3-based tests
* add keyword to all
* add PreTrainedModel
* updateing video loading logic
* working inference
* updates for PR comments
* updates for PR comments
* moving SmolVLMPretrainedModel higher to fix import error
* CI test pass
* CI test pass
* removing lambda
* CI test pass
* CI test pass
* CI test pass
* CI test pass
* CI test pass
* CI test pass
* processor tests
* add example in docs
* typo
* fix copies
* skip compile tests - sdpa for VisionTransformer
* fix init
* raise import error for num2words
* update doc for FA2
* more doc fix
* CI
* updates for PR comments
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Joshua Lochner <admin@xenova.com >
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* fixing processor -- tokenizer not defined properly, (gpt2 tokenizer), and does not have the attributes of fake image token, etc
* adding smolvlm to VQA models
* removing vqa auto class
* Update src/transformers/models/smolvlm/processing_smolvlm.py
Co-authored-by: Joshua Lochner <admin@xenova.com >
* removing smolvlmvisiontransformer from index.md
* my bad, video processing had typos
* fixing docs
* renaming params in SmolVLMModel.inputs_merger
* removing un-needed dtype/device in model forward
* ruff for CI
* update docs
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
* return cache position
* return cache position
* return cache also in modular
* needed to run modular again
* fix training tests
* push vectorized inputs merger
* format
* format
* reduce number of mappings
* addressing PR comments
* happy CI, happy me :)
* skip non-nested images
* adjust integration test for smaller GPUs
* format
* fix kwargs in chat template apply
* skip this for now
---------
Co-authored-by: raushan <raushan@huggingface.co >
Co-authored-by: Pablo <pablo.montalvo.leroux@gmail.com >
Co-authored-by: Pedro Cuenca <pedro@huggingface.co >
Co-authored-by: Joshua Lochner <admin@xenova.com >
2025-02-20 15:00:26 +01:00
Joao Gante
99adc74462
[tests] remove flax-pt equivalence and cross tests ( #36283 )
2025-02-19 15:13:27 +00:00
Joao Gante
0863eef248
[tests] remove pt_tf equivalence tests ( #36253 )
2025-02-19 11:55:11 +00:00
Mehant Kammakomati
c3ba53303b
feat: add support for tensor parallel training workflow with accelerate ( #34194 )
...
* feat: add support for tensor parallel flow using accelerate
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com >
* fix: add tp degree to env variable
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com >
* fix: add version check for accelerate to allow TP
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com >
* docs: tensor parallelism
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com >
* nit: rename plugin name
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com >
* fix: guard accelerate version before allow tp
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com >
* docs: add more docs and updates related to TP
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com >
---------
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com >
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
2025-02-18 14:05:46 +01:00
Mayank Mishra
a570e2ba87
add shared experts for upcoming Granite 4.0 language models ( #35894 )
...
* Modular GraniteMoE with shared Experts.
Signed-off-by: Shawn Tan <shawntan@ibm.com >
* Modified
* Import order.
* Modified for style
* Fix space.
* Test
* Remove extra granitemoe file.
* New converted file and tests
* Modified __init__ files.
* Formatting.
* Dummy PT objects
* register granitemoe shared model
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
* fix linting of a file
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
* fix import in modeling file
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
* update generated modeling file
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
* add documentation
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
* update docstrings
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
* update generated modeling file
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
* fix docstrings in config class
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
* merge main
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
---------
Signed-off-by: Shawn Tan <shawntan@ibm.com >
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
Co-authored-by: Shawn Tan <shawntan@ibm.com >
Co-authored-by: Shawn Tan <shawn@wtf.sg >
Co-authored-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com >
Co-authored-by: Sukriti Sharma <Ssukriti@users.noreply.github.com >
2025-02-14 16:55:28 +01:00
Isotr0py
33d1d715b0
Add ImageProcessorFast to Qwen2.5-VL processor ( #36164 )
...
* add qwen2 fast image processor to modular file
Signed-off-by: isotr0py <2037008807@qq.com >
* fix modular
Signed-off-by: isotr0py <2037008807@qq.com >
* fix circle import
Signed-off-by: isotr0py <2037008807@qq.com >
* add docs
Signed-off-by: isotr0py <2037008807@qq.com >
* fix typo
Signed-off-by: isotr0py <2037008807@qq.com >
* add modular generated files
Signed-off-by: isotr0py <2037008807@qq.com >
* revert qwen2vl fast image processor
Signed-off-by: isotr0py <2037008807@qq.com >
* remove qwen2.5-vl image processor from modular
Signed-off-by: isotr0py <2037008807@qq.com >
* re-generate qwen2.5-vl files
Signed-off-by: isotr0py <2037008807@qq.com >
* remove unnecessary test
Signed-off-by: isotr0py <2037008807@qq.com >
* fix auto map
Signed-off-by: isotr0py <2037008807@qq.com >
* cleanup
Signed-off-by: isotr0py <2037008807@qq.com >
* fix model_input_names
Signed-off-by: isotr0py <2037008807@qq.com >
* remove import
Signed-off-by: isotr0py <2037008807@qq.com >
* make fix-copies
Signed-off-by: isotr0py <2037008807@qq.com >
---------
Signed-off-by: isotr0py <2037008807@qq.com >
2025-02-14 17:34:55 +08:00
Raushan Turganbay
1931a35140
Chat template docs ( #36163 )
...
* decompose chat template docs
* add docs
* update model docs
* qwen2-5
* pixtral
* remove old chat template
* also video as list frames supported
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/chat_template_multimodal.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* remove audio for now
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
2025-02-14 10:32:14 +01:00
Elvir Crnčević
845b0a2616
Efficient Inference Kernel for SpQR ( #34976 )
...
* Resolve vptq conflict
* Rename spqr package to spqr_quant
* Get rid of aqlm mention
* Start working on tests
* Resolve ruff code checks
* Ruff format
* Isort
* Test updates
* Add gpu tag
* Rename to modules_to_not_convert
* Config update
* Docs and config update
* Docs and config update
* Update to update_torch_dtype
* spqr config parameter validation
* Ruff update
* Apply ruff fixes
* Test fixes
* Ruff update
* Mark tests as @slow again; Ruff; Docstring update
* Ruff
* Remove absolute path
* Resolve typo
* Remove redundandt log
* Check accelerate/spqr availability
* Ruff fix
* Check if the config contains proper shapes
* Ruff test
* Documentation update
* overview update
* Ruff checks
* Ruff code quality
* Make style
* Update docs/source/en/quantization/spqr.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update spqr.md
* Enable gptqmodel (#35012 )
* gptqmodel
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix format
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* update readme
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* gptqmodel need use checkpoint_format (#1 )
* gptqmodel need use checkpoint_format
* fix quantize
* Update quantization_config.py
* Update quantization_config.py
* Update quantization_config.py
---------
Co-authored-by: ZX-ModelCloud <zx@modelcloud.ai >
Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai >
* Revert quantizer_gptq.py (#2 )
* revert quantizer_gptq.py change
* pass **kwargs
* limit gptqmodel and optimum version
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix format
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix warning
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix version check
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* revert unrelated changes
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* enable gptqmodel tests
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix requires gptq
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* Fix Transformer compat (#3 )
* revert quantizer_gptq.py change
* pass **kwargs
* add meta info
* cleanup
* cleanup
* Update quantization_config.py
* hf_select_quant_linear pass checkpoint_format and meta
* fix GPTQTestCUDA
* Update test_gptq.py
* gptqmodel.hf_select_quant_linear() now does not select ExllamaV2
* cleanup
* add backend
* cleanup
* cleanup
* no need check exllama version
* Update quantization_config.py
* lower checkpoint_format and backend
* check none
* cleanup
* Update quantization_config.py
* fix self.use_exllama == False
* spell
* fix unittest
* fix unittest
---------
Co-authored-by: LRL <lrl@lbx.dev >
Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai >
* fix format
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix format again
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* update gptqmodel version (#6 )
* update gptqmodel version
* update gptqmodel version
* fix unit test (#5 )
* update gptqmodel version
* update gptqmodel version
* "not self.use_exllama" is not equivalent to "self.use_exllama==False"
* fix unittest
* update gptqmodel version
* backend is loading_attibutes (#7 )
* fix format and tests
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix memory check
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix device mismatch
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* fix result check
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* Update src/transformers/quantizers/quantizer_gptq.py
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
* Update src/transformers/quantizers/quantizer_gptq.py
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
* Update src/transformers/quantizers/quantizer_gptq.py
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
* update tests
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* review: update docs (#10 )
* review: update docs (#12 )
* review: update docs
* fix typo
* update tests for gptqmodel
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
* update document (#9 )
* update overview.md
* cleanup
* Update overview.md
* Update overview.md
* Update overview.md
* update gptq.md
* Update gptq.md
* Update gptq.md
* Update gptq.md
* Update gptq.md
* Update gptq.md
* Update gptq.md
---------
Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai >
* typo
* doc note for asymmetric quant
* typo with apple silicon(e)
* typo for marlin
* column name revert: review
* doc rocm support
* Update docs/source/en/quantization/gptq.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/quantization/gptq.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/quantization/gptq.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/quantization/gptq.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/quantization/overview.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/quantization/overview.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
---------
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
Co-authored-by: LRL-ModelCloud <165116337+LRL-ModelCloud@users.noreply.github.com >
Co-authored-by: ZX-ModelCloud <zx@modelcloud.ai >
Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai >
Co-authored-by: ZX-ModelCloud <165115237+ZX-ModelCloud@users.noreply.github.com >
Co-authored-by: LRL <lrl@lbx.dev >
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
Co-authored-by: Mohamed Mekkouri <93391238+MekkCyber@users.noreply.github.com >
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Fix : Nemotron Processor in GGUF conversion (#35708 )
* fixing nemotron processor
* make style
* Update docs/source/en/quantization/spqr.md
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com >
* Add missing TOC to doc
---------
Signed-off-by: jiqing-feng <jiqing.feng@intel.com >
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
Co-authored-by: jiqing-feng <jiqing.feng@intel.com >
Co-authored-by: LRL-ModelCloud <165116337+LRL-ModelCloud@users.noreply.github.com >
Co-authored-by: ZX-ModelCloud <zx@modelcloud.ai >
Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai >
Co-authored-by: ZX-ModelCloud <165115237+ZX-ModelCloud@users.noreply.github.com >
Co-authored-by: LRL <lrl@lbx.dev >
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
Co-authored-by: Mohamed Mekkouri <93391238+MekkCyber@users.noreply.github.com >
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com >
2025-02-13 16:22:58 +01:00
Mohamed Mekkouri
b41591d847
Fix : fix doc fp8 ( #36173 )
...
* fix
* fix
2025-02-13 15:29:59 +01:00
Mohamed Mekkouri
efe72fe21f
Adding FP8 Quantization to transformers ( #36026 )
...
* first commit
* adding kernels
* fix create_quantized_param
* fix quantization logic
* end2end
* fix style
* fix imports
* fix consistency
* update
* fix style
* update
* udpate after review
* make style
* update
* update
* fix
* update
* fix docstring
* update
* update after review
* update
* fix scheme
* update
* update
* fix
* update
* fix docstring
* add source
* fix test
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
2025-02-13 13:01:19 +01:00
Lysandre Debut
c82319b493
Helium documentation fixes ( #36170 )
...
* Helium documentation fixes
* Update helium.md
* Update helium.md
* Update helium.md
2025-02-13 12:20:53 +01:00
Thomas Bauwens
8f137b2427
Move DataCollatorForMultipleChoice from the docs to the package ( #34763 )
...
* Add implementation for DataCollatorForMultipleChoice based on docs.
* Add DataCollatorForMultipleChoice to import structure.
* Remove custom DataCollatorForMultipleChoice implementations from example scripts.
* Remove custom implementations of DataCollatorForMultipleChoice from docs in English, Spanish, Japanese and Korean.
* Refactor torch version of DataCollatorForMultipleChoice to be more easily understandable.
* Apply suggested changes and run make fixup.
* fix copies, style and fixup
* add missing documentation
* nits
* fix docstring
* style
* nits
* isort
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com >
Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com >
2025-02-13 12:01:28 +01:00
Ke Wen
f869d486d3
Update doc re list of models supporting TP ( #35864 )
...
Update doc about models' TP support
2025-02-12 15:53:27 +01:00
