* Adding support for `microphone` streaming within pipeline.
- Uses `ffmpeg` to get microphone data.
- Makes sure alignment is made to `size_of_sample`.
- Works by sending `{"raw": ..data.., "stride": (n, left, right),
"partial": bool}`
directly to the pipeline enabling to stream partial results and still
get inference.
- Let's `partial` information flow through the pipeline to enable caller
to get it back and choose to display text or not.
- The striding reconstitution is bound to have errors since CTC does not
keep previous state. Currently most of the errors are we don't know if
there's a space or not between two chunks.
Since we have some left striding info, we could use that during decoding
to choose what to do with those spaces and even extra letters maybe (if
the stride is long enough, it's bound to cover at least a few symbols)
Fixing tests.
Protecting with `require_torch`.
`raw_ctc` support for nicer demo.
Post rebase fixes.
Revamp to split raw_mic_data from it's live chunking.
- Requires a refactor to make everything a bit cleaner.
Automatic resampling.
Small fix.
Small fix.
* Post rebase fix (need to let super handle more logic, reorder args.)
* Update docstrings
* Docstring format.
* Remove print.
* Prevent flow of `input_values`.
* Fixing `stride` too.
* Fixing the PR by removing `raw_ctc`.
* Better docstrings.
* Fixing init.
* Update src/transformers/pipelines/audio_utils.py
Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>
* Update tests/test_pipelines_automatic_speech_recognition.py
Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>
* Quality.
Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>
* Add torchvision's resize
* Rename torch_resize to default_to_square
* Apply suggestions from code review
* Add support for default_to_square and tuple of length 1
* add new test
* update test
* remove `tokenizer_file` from `additional_files_names` in `tokenization_utils_base.py`
* add `tokenizer_file` for the fast only tokenizer
* change global variables layoutxml
* remove `"tokenizer_file"` from DPR tokenizer's Global variables
* remove `tokenizer_file` from herbert slow tokenizer init
* `"tokenizer_file"` from LED tokenizer's Global variables
* remove `tokenizer_file` from mbart slow tokenizer init
* remove `tokenizer_file` from slow tokenizer template
* adapt to versioning
* adapt the `test_tokenizer_mismatch_warning` test
* clean test
* clarify `VOCAB_FILES_NAMES` in tokenization_utils_fast.py
* Revert "remove `tokenizer_file` from mbart slow tokenizer init"
This reverts commit 0dbb723fa9c7599d4640fe30b3647a74eb4a64e1.
* Revert "`"tokenizer_file"` from LED tokenizer's Global variables"
This reverts commit 5a3f879bdd651233f3d74a3d1146c34cde82b0c2.
* Revert "remove `tokenizer_file` from herbert slow tokenizer init"
This reverts commit f5e10007b7b0ec5345e015b9de7ffec72c5407fd.
* Revert "remove `"tokenizer_file"` from DPR tokenizer's Global variables"
This reverts commit da0895330bedfafc81ae3073470a9348c669f032.
* set `tokenizer_file` in super `__init__` of mbart
* replace assert with exception for `padding_side` arg in `PreTrainedTokenizerBase` `__init__`
* add test
* fix kwargs
* reformat test
* format
* format
* fix typo to render the documentation
* Update modeling_wav2vec2.py
With very tiny sound files (less than 0.1 seconds) the num_masked_span can be too long. The issue is described in issue #15366 and discussed with @patrickvonplaten.
* correct errors with mask time indices
* remove bogus file
* make fix-copies
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Fix the inconsistency of loss calculation between PT/TF XLNetLMHeadModel
* overwrite test_loss_computation
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
* add xlm roberta xl
* add convert xlm xl fairseq checkpoint to pytorch
* fix init and documents for xlm-roberta-xl
* fix indention
* add test for XLM-R xl,xxl
* fix model hub name
* fix some stuff
* up
* correct init
* fix more
* fix as suggestions
* add torch_device
* fix default values of doc strings
* fix leftovers
* merge to master
* up
* correct hub names
* fix docs
* fix model
* up
* finalize
* last fix
* Apply suggestions from code review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* add copied from
* make style
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Fixing support `batch_size` and `num_return_Sequences` in
`text-generation` pipeline
And `text2text-generation` too.
The bug was caused by the batch_size containing both the incoming batch
**and** the generated `num_sequences`.
The fix simply consists into splitting both of these again into
different dimensions.
* TF support.
* Odd backward compatibility script in the way.
* add new test
* add a feature to same the sentencepiece tokenizer model when the init file was deleted
* update marian
* update m2m_100
* fix marian
* update speech to text
* override test for layoutxlm
* fix saving bartpho
* remove harcoded values bartpho
* special token string version
* finish bartpho
* override layoutxml test
* add mbart
* move special tokens list
* format
* Revert "format"
This reverts commit 37a40df37903a932c2f951cbd33acb684246bae7.
* simplify list of string of special tokens
* Re-write `self.fairseq_tokens_to_ids ` initialization logic with special tokens
Co-authored-by: Sylvain Gugger <sylvain.gugger@gmail.com>
Co-authored-by: Sylvain Gugger <sylvain.gugger@gmail.com>
* Avoid using get_list_of_files in config
* Wip, change tokenizer file getter
* Remove call in tokenizer files
* Remove last call to get_list_model_files
* Better tests
* Unit tests for new function
* Document bad API
* Add new model like command
* Bad doc-styler
* black and doc-styler, stop fighting!
* black and doc-styler, stop fighting!
* At last
* Clean up
* Typo
* Bad doc-styler
* Bad doc-styler
* All good maybe?
* Use constants
* Add doc and type hints
* More cleaning
* Add doc
* Fix Copied from
* Doc template
* Use typing.Pattern instead
* Framework-specific files
* Fixes
* Select frameworks clean model init
* Deal with frameworks in main init
* fixes
* Last fix
* Prompt user for info
* Delete exemple config
* Last fixes
* Add test config
* Fix bug with model_type included in each other
* Fixes
* More fixes
* More fixes
* Adapt config
* Remove print statements
* Will fix tokenization later, leave it broken for now
* Add test
* Quality
* Try this way
* Debug
* Maybe by setting the path?
* Let's try another way
* It should go better when actually passing the arg...
* Remove debug statements and style
* Fix config
* Add tests
* Test require the three backends
* intermediate commit
* Revamp pattern replacements and start work on feature extractors
* Adapt model info
* Finalize code for processors
* Fix in main init additions
* Finish questionnaire for processing classes
* Fix file name
* Fix for real
* Fix patterns
* Style
* Remove needless warnings
* Copied from should work now.
* Include Copied form in blocks
* Add test
* More fixes and tests
* Apply suggestions from code review
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* Address review comment
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* better
* save intermediate
* finish code
* up
* docs
* Apply suggestions from code review
* up
* add compute transition beam scores function to model and make sure scores are correct with eos
* apply nicos comments
* Apply suggestions from code review
* another fix
* Refine errors for pretrained objects
* PoC to avoid using get_list_of_files
* Adapt tests to use new errors
* Quality + Fix PoC
* Revert "PoC to avoid using get_list_of_files"
This reverts commit cb93b7cae8504ef837c2a7663cb7955e714f323e.
* Revert "Quality + Fix PoC"
This reverts commit 3ba6d0d4ca546708b31d355baa9e68ba9736508f.
* Fix doc
* Revert PoC
* Add feature extractors
* More tests and PT model
* Adapt error message
* Feature extractor tests
* TF model
* Flax model and test
* Merge flax auto tests
* Add tokenization
* Fix test
* Add missing __spec__ for transformers.models.auto
* Moves the __spec__-test to the UnitTest class
* Adds module_spec to all instances of _LazyModule
* Refactors an old test from pytest to unittest
* [EncoderDecoder] Add test for usage of extra kwargs
* [EncoderDecoder] Fix usage of extra kwargs in from pretrained
* [EncoderDecoder] apply suggested changes (passing **kwargs_encoder)
* [EncoderDecoder] create new test function and make sure it passes
Co-authored-by: jonas <jsnfly@gmx.de>
* [WIP] Make chuking smartly (long files) work on asr ctc_with_lm.
* Slow test with functionality.
* Fixing regular test.
* fix for batch size 1
* Handling batch outside `rescale_Stride`.
- Renamed to `rescale_stride`.
* Disable equality in the test.
* Remove print.
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* First commit
* Add conversion script
* Make conversion script work for base model
* More improvements
* Update conversion script, works for vqa
* Add indexing argument to meshgrid
* Make conversion script work for ViltForPreTraining
* Add ViltForPreTraining to docs
* Fix device issue
* Add processor
* Add MinMaxResize to feature extractor
* Implement call method of ViltProcessor
* Fix tests
* Add integration test
* Add loss calculation for VQA
* Improve tests
* Improve some more tests
* Debug tests
* Small improvements
* Add support for attention_mask
* Remove mask_it
* Add pixel_mask
* Add tests for ViltFeatureExtractor
* Improve tests
* Add ViltForNaturalLanguageVisualReasoning
* Add ViltForNaturalLanguageVisualReasoning to conversion script
* Minor fixes
* Add support for image_embeds, update docstrings to markdown
* Update docs to markdown
* Improve conversion script
* Rename ViltForPreTraining to ViltForMaskedLM
* Improve conversion script
* Convert docstrings to markdown
* Fix code example of retrieval model
* Properly convert masked language model
* Add integration test for nlvr
* Fix code quality
* Apply suggestions from code review
* Add copied from statements
* Fix pretrained_config_archive_map
* Fix docs
* Add model to README
* Apply suggestions from code review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Apply more suggestions from code review
* Make code more readable
* Add ViltForNaturalLanguageVisualReasoning to the tests
* Rename ViltForVisualQuestionAnswering to ViltForQuestionAnswering
* Replace pixel_values_2 by single tensor
* Add hidden_states and attentions
* Fix one more test
* Fix all tests
* Update year
* Fix rebase issues
* Fix another rebase issue
* Remove ViltForPreTraining from auto mapping
* Rename ViltForImageRetrievalTextRetrieval to ViltForImageAndTextRetrieval
* Make it possible to use BertTokenizerFast in the processor
* Use BertTokenizerFast by default
* Rename ViltForNaturalLanguageVisualReasoning, define custom model output
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Rename compute_loss to hf_compute_loss to avoid conflicts with the new Keras method
* make style
* Adding deprecation warning to `compute_loss`
* Fix sneaky reference to compute_loss
* Replace logger.warning with warnings.warn
* Clarifying warning and deprecation timeline
* First draft
* More improvements
* More improvements
* More improvements
* Fix embeddings
* Add conversion script
* Finish conversion script
* More improvements
* Fix forward pass
* Remove print statements
* Add weights initialization
* Add initialization of decoder weights
* Add support for other models in the conversion script
* Fix patch_size for huge model
* Fix most of the tests
* Fix integration test
* Fix docs
* Fix archive_list
* Apply suggestions from code review
* Improve documentation
* Apply more suggestions
* Skip some tests due to non-deterministic behaviour
* Fix test_initialization
* Remove unneccessary initialization of nn.Embedding
* Improve docs
* Fix dummies
* Remove ViTMAEFeatureExtractor from docs
* Add model to README and table of contents
* Delete inference file
* Fix deprecation warnings for int div
Co-authored-by: mgoldey <matthew.goldey@gmail.com>
* Fix import
* ensure that tensor output is python scalar
* make backward compatible
* make code more readable
* adapt test functions
Co-authored-by: mgoldey <matthew.goldey@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
