[Whisper] Make tokenizer normalization public (#28136)

* [Whisper] Make tokenizer normalization public * add to docs
2024-01-29 16:07:35 +00:00
parent e694e985d7
commit da3c79b245
3 changed files with 41 additions and 5 deletions
--- a/docs/source/en/model_doc/whisper.md
+++ b/docs/source/en/model_doc/whisper.md
@@ -102,6 +102,8 @@ python convert_hf_to_openai.py \
    - save_vocabulary
    - batch_decode
    - decode
+    - basic_normalize
+    - normalize

 ## WhisperTokenizerFast

@@ -113,6 +115,8 @@ python convert_hf_to_openai.py \
    - save_vocabulary
    - batch_decode
    - decode
+    - basic_normalize
+    - normalize

 ## WhisperFeatureExtractor