SaulLu 1025a9b742 add a warning in SpmConverter for sentencepiece's model using the byte fallback feature (#16629) ...

* update proto sentencepiece model

* Revert "update proto sentencepiece model"

This reverts commit b07f671747fec35773d0b3d4788b8b15aefa0229.

* add check

* add test

* Revert "Revert "update proto sentencepiece model""

This reverts commit 46108257b8927b73627ec8f4f3eed53a95fc700d.

* test for log level

* test for log level 2

* warning at the warning level

* clean

* format

* add explanation in docstring

2022-04-11 11:06:10 +02:00

..

tests_samples

Fix img classification tests (#13456)

2021-09-07 05:58:45 -04:00

add_distilbert_like_config.json

Add model like (#14992)

2022-01-24 15:25:10 -05:00

dummy_feature_extractor_config.json

[AutoProcessor] Add Wav2Vec2WithLM & small fix (#14675)

2021-12-08 15:51:28 +01:00

dummy-config.json

AutoConfig + other Auto classes honor model_type

2020-01-11 02:46:17 +00:00

empty.txt

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

input.txt

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

merges.txt

[AutoTokenizer] Allow creation of tokenizers by tokenizer type (#13668)

2021-09-22 00:29:38 +02:00

preprocessor_config.json

Auto processor (#14465)

2021-11-22 12:17:38 -05:00

sample_text_no_unicode.txt

[Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659)

2020-10-18 20:51:24 +02:00

sample_text.txt

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

spiece.model

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_entity_vocab.json

Feature/fix slow test in mluke (#14749)

2021-12-22 06:35:59 -05:00

test_sentencepiece_bpe.model

Conversion from slow to fast for BPE spm vocabs contained an error. (#10120)

2021-02-13 08:24:53 -05:00

test_sentencepiece_no_bos.model

[pegasus] Faster tokenizer tests (#7672)

2020-10-09 11:10:32 -04:00

test_sentencepiece_with_bytefallback.model

add a warning in SpmConverter for sentencepiece's model using the byte fallback feature (#16629)

2022-04-11 11:06:10 +02:00

test_sentencepiece.model

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

vocab.json

[AutoTokenizer] Allow creation of tokenizers by tokenizer type (#13668)

2021-09-22 00:29:38 +02:00

vocab.txt

[AutoTokenizer] Allow creation of tokenizers by tokenizer type (#13668)

2021-09-22 00:29:38 +02:00