Tomohiro Endo 08d609bfb8 Add tokenizers class mismatch detection between cls and checkpoint (#12619) ...

* Detect mismatch by analyzing config

* Fix comment

* Fix import

* Update src/transformers/tokenization_utils_base.py

Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>

* Revise based on reviews

* remove kwargs

* Fix exception

* Fix handling exception again

* Disable mismatch test in PreTrainedTokenizerFast

Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>

2021-07-17 15:52:21 +02:00

12 KiB

Raw Blame History

View Raw