add tokenizer and tests

Browse Source

...

This commit is contained in:

thomwolf

2019-06-21 11:09:51 +02:00

parent 45709d7532

commit 32da75486b

11 changed files with 511 additions and 57 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

BIN

samples/test_sentencepiece.model Normal file

Binary file not shown.