Make bert_japanese and cpm independent of their inherited modules (#19431)

* Make cpm tokenization independent of xlnet

* Make bert japanese tokenization independent of bert
This commit is contained in:
David Yang
2022-10-12 00:09:17 +08:00
committed by GitHub
parent 462cd641d9
commit d0d5aee1dd
4 changed files with 738 additions and 19 deletions

View File

@@ -19,10 +19,10 @@ import pickle
import unittest
from transformers import AutoTokenizer
from transformers.models.bert.tokenization_bert import BertTokenizer
from transformers.models.bert_japanese.tokenization_bert_japanese import (
VOCAB_FILES_NAMES,
BertJapaneseTokenizer,
BertTokenizer,
CharacterTokenizer,
JumanppTokenizer,
MecabTokenizer,