Files
HuggingFace_transformer/docs/source
Jonatan Kłosko deafc24388 Add WhisperTokenizerFast (#21222)
* Add WhisperTokenizerFast

* Fixup

* Up

* Up

* Improve tests

* Update src/transformers/models/whisper/tokenization_whisper_fast.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Keep stride in whisper pipelien test

* Remove unknown token special case

* Reduce vocabulary size in tests

* Fix vocab size assertion

* Sync copied changes from WhisperTokenizer

* Skip pipeline tests

* Update assertion

* Remove Whisper tokenizer dependency on sentencepiece

* Format

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-02-21 06:58:54 +01:00
..
2023-02-20 16:37:11 +03:00
2023-02-21 06:58:54 +01:00
2023-02-20 16:37:11 +03:00
2023-02-20 16:37:11 +03:00
2023-02-20 16:37:11 +03:00
2023-02-20 16:37:11 +03:00
2023-02-20 16:37:11 +03:00
2023-01-30 16:50:57 -05:00