add a warning in SpmConverter for sentencepiece's model using the byte fallback feature (#16629)

Browse Source

* update proto sentencepiece model

* Revert "update proto sentencepiece model"

This reverts commit b07f671747fec35773d0b3d4788b8b15aefa0229.

* add check

* add test

* Revert "Revert "update proto sentencepiece model""

This reverts commit 46108257b8927b73627ec8f4f3eed53a95fc700d.

* test for log level

* test for log level 2

* warning at the warning level

* clean

* format

* add explanation in docstring

...

This commit is contained in:

SaulLu

2022-04-11 11:06:10 +02:00

committed by GitHub

parent 7c5d79912a

commit 1025a9b742

4 changed files with 486 additions and 158 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

BIN

tests/fixtures/test_sentencepiece_with_bytefallback.model vendored Normal file

Binary file not shown.