SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

add a template to add missing tokenization test (#16553)

Browse Source 
* add a template to add missing tokenization test

* add cookiecutter setting

* improve doc

* Update templates/adding_a_missing_tokenization_test/README.md

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
This commit is contained in:
SaulLu
2022-04-05 10:50:22 +02:00
committed by GitHub
parent 765bafb8e4
commit 02214cb3cc
3 changed files with 127 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
templates/adding_a_missing_tokenization_test/cookiecutter.json Normal file
View File

@@ -0,0 +1,10 @@
{
"modelname": "BrandNewBERT",
"uppercase_modelname": "BRAND_NEW_BERT",
"lowercase_modelname": "brand_new_bert",
"camelcase_modelname": "BrandNewBert",
"has_slow_class": ["True", "False"],
"has_fast_class": ["True", "False"],
"slow_tokenizer_use_sentencepiece": ["True", "False"],
"authors": "The HuggingFace Team"
}