* Normalize image - cast input images to float32.
This is done if the input image isn't of floating type. Issues can occur when do_rescale=False is set in an image processor. When this happens, the image passed to the call is of type uint8 becuase of the type casting that happens in resize because of the PIL image library. As the mean and std values are cast to match the image dtype, this can cause NaNs and infs to appear in the normalized image, as the floating values being used to divide the image are now set to 0.
The reason the mean and std values are cast is because previously they were set as float32 by default. However, if the input image was of type float16, the normalization would result in the image being upcast to float32 too.
* Add tests
* Remove float32 cast
* only dir not even init
* init
* tokenizer removed and reference of codegen added
* modeling file updated a lot remaining app_rotary_emb
* conversion script done
* conversion script fixed, a lot of factoring done and most tests pass
* added token_clf and extractive_QA_head
* integration tests pass
* flash attn tests pass!
* config done
* more docs in modeling file
* some style fix
* style and others
* doc test error fix
* more doc fix
* some attention fixes
* most fixes
* style and other fixes
* docs fix and config
* doc fix
* some comments
* conversion script updated
* conversion script updated
* Revert "conversion script updated"
This reverts commit e92378c54084ec0747041b113083d1746ecb6c7f.
* final comments
* add Phi to language_modeling.md
* edit phi.md file
* rebase and fix
* removed phi-1.5 example
* changed model_type from 'phi'->'mixformer-sequential'
* small change
* small change
* revert \small change
* changed mixformer-sequential->phi
* small change
* added phi-1.5 example instead of phi-1
* doc test might pass now
* rebase and small change
* added the dropout layer
* more fixes
* modified .md file
* very very small doc change
* fix?
* actual fix
* fixups
* add dataclass to the attention mask converter
* refine testing suite
* make sure there are no overflows
* update the test
* init commit
* attention arch done except rotary emb
* rotary emb done
* text encoder working
* outputs matching
* arch first pass done
* make commands done, tests and docs remaining
* all tests passed, only docs remaining
* docs done
* doc-builder fix
* convert script removed(not relevant)
* minor comments done
* added ckpt conversion script
* tokenizer done
* very minor fix of index.md 2
* mostly make fixup related
* all done except fe and rotary emb
* very small change
* removed unidecode dependency
* style changes
* tokenizer removed require_backends
* added require_inflect to tokenizer tests
* removed VOCAB_FILES in tokenizer test
* inflect dependency removed
* added rotary pos emb cache and simplified the apply method
* style
* little doc change
* more comments
* feature extractor added
* added processor
* auto-regressive config added
* added CLVPConditioningEncoder
* comments done except the test one
* weights added successfull(NOT tested)
* tokenizer fix with numbers
* generate outputs matching
* almost tests passing Integ tests not written
* Integ tests added
* major CUDA error fixed
* docs done
* rebase and multiple fixes
* fixed rebase overwrites
* generate code simplified and tests for AutoRegressive model added
* minor changes
* refectored gpt2 code in clvp file
* weights done and all code refactored
* mostly done except the fast_tokenizer
* doc test fix
* config file's doc fixes
* more config fix
* more comments
* tokenizer comments mostly done
* modeling file mostly refactored and can load modules
* ClvpEncoder tested
* ClvpDecoder, ClvpModel and ClvpForCausalLM tested
* integration and all tests passed
* more fixes
* docs almost done
* ckpt conversion refectored
* style and some failing tests fix
* comments
* temporary output fix but test_assisted_decoding_matches_greedy_search test fails
* majority changes done
* use_cache outputs same now! Along with the asisted_greedy_decoding test fix
* more comments
* more comments
* prepare_inputs_for_generation fixed and _prepare_model_inputs added
* style fix
* clvp.md change
* moved clvpconditionalencoder norms
* add model to new index
* added tokenizer input_ids_with_special_tokens
* small fix
* config mostly done
* added config-tester and changed conversion script
* more comments
* comments
* style fix
* some comments
* tokenizer changed back to prev state
* small commnets
* added output hidden states for the main model
* style fix
* comments
* small change
* revert small change
* .
* Update clvp.md
* Update test_modeling_clvp.py
* :)
* some minor change
* new fixes
* remove to_dict from FE
* add audio_utils usage in the FE of SpeechToText
* clean unecessary parameters of AudioSpectrogramTransformer FE
* add audio_utils usage in AST
* add serialization tests and function to FEs
* make style
* remove use_torchaudio and move to_dict to FE
* test audio_utils usage
* make style and fix import (remove torchaudio dependency import)
* fix torch dependency for jax and tensor tests
* fix typo
* clean tests with suggestions
* add lines to test if is_speech_availble is False
* Use Llama RoPE implementation for Falcon
+ Add copy functionalities
* Use standard cache format for Falcon
* Simplify apply_rotary_pos_emb, copy from Llama
* Remove unnecessary cache conversion test
We don't need to convert any caches anymore!
* Resolve copy complaint
* Fixing m4t.
* Trying to remove comparison ? Odd test failure.
* Adding shared. But why on earth does it hang ????
* Putting back the model weights checks the test is silently failing on
cuda.
* Fix style + unremoved comment.
* Fix Fuyu image scaling bug
It could produce negative padding and hence inference errors for certain
image sizes.
* initial rework commit
* add batching capabilities, refactor image processing
* add functional batching for a list of images and texts
* make args explicit
* Fuyu processing update (#27133)
* Add file headers
* Add file headers
* First pass - preprocess method with standard args
* First pass image processor rework
* Small tweaks
* More args and docstrings
* Tidying iterating over batch
* Tidying up
* Modify to have quick tests (for now)
* Fix up
* BatchFeature
* Passing tests
* Add tests for processor
* Sense check when patchifying
* Add some tests
* FuyuBatchFeature
* Post-process box coordinates
* Update to `size` in processor
* Remove unused and duplicate constants
* Store unpadded dims after resize
* Fix up
* Return FuyuBatchFeature
* Get unpadded sizes after resize
* Update exception
* Fix return
* Convert input `<box>` coordinates to model format.
* Post-process point coords, support multiple boxes/points in a single
sequence
* Replace constants
* Update src/transformers/models/fuyu/image_processing_fuyu.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Preprocess List[List[image]]
* Update src/transformers/models/fuyu/image_processing_fuyu.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update to Amy's latest state.
* post-processing returns a list of tensors
* Fix error when target_sizes is None
Co-authored-by: Pablo Montalvo <pablo.montalvo.leroux@gmail.com>
* Update src/transformers/models/fuyu/image_processing_fuyu.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update src/transformers/models/fuyu/image_processing_fuyu.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update src/transformers/models/fuyu/image_processing_fuyu.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update src/transformers/models/fuyu/image_processing_fuyu.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Review comments
* Update src/transformers/models/fuyu/image_processing_fuyu.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Fix up
* Fix up
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-72-126.ec2.internal>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Pablo Montalvo <pablo.montalvo.leroux@gmail.com>
* Fix conflicts in fuyu_follow_up_image_processing (#27228)
fixing conflicts and updating on main
* Revert "Fix conflicts in fuyu_follow_up_image_processing" (#27232)
Revert "Fix conflicts in fuyu_follow_up_image_processing (#27228)"
This reverts commit acce10b6c653dc7041fb9d18cfed55775afd6207.
---------
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-72-126.ec2.internal>
* add whisper fa2
* correct
* change all
* correct
* correct
* fix more
* fix more
* fix more
* fix more
* fix more
* fix more
* Apply suggestions from code review
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* fix more
* fix more
* fix more
* fix more
* fix more
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Add type annotations to TFConvNextDropPath
* Use tf.debugging.assert_equal for TFConvNextEmbeddings shape check
* Add TensorFlow implementation of ConvNeXTV2
* check_docstrings: add TFConvNextV2Model to exclusions
TFConvNextV2Model and TFConvNextV2ForImageClassification have docstrings
which are equivalent to their PyTorch cousins, but a parsing issue prevents them
from passing the test.
Adding exclusions for these two classes as discussed in #25558.
* Safetensors serialization by default
* First pass on the tests
* Second pass on the tests
* Third pass on the tests
* Fix TF weight loading from TF-format safetensors
* Specific encoder-decoder fixes for weight crossloading
* Add VisionEncoderDecoder fixes for TF too
* Change filename test for pt-to-tf
* One missing fix for TFVisionEncoderDecoder
* Fix the other crossload test
* Support for flax + updated tests
* Apply suggestions from code review
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
* Sanchit's comments
* Sanchit's comments 2
* Nico's comments
* Fix tests
* cleanup
* Apply suggestions from code review
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
---------
Co-authored-by: Matt <rocketknight1@gmail.com>
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* stronger GC tests
* better tests and skip failing tests
* break down into 3 sub-tests
* break down into 3 sub-tests
* refactor a bit
* more refactor
* fix
* last nit
* credits contrib and suggestions
* credits contrib and suggestions
---------
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
