Li-Huai (Allan) Lin 66ea739168 Improve tokenizer tests (#13594) ...

* Use new method to acquire tokenizers

* Resolve TODOs.

* Style

* Fix

* Enable do_lower_case in test_tokenize_special_tokens

* Apply suggestion from code review

* Fix mask token handling

* Revert "Fix mask token handling"

This reverts commit daaa3f5291b1f71e5bc3604ca281c000000c4648.

* Fix FNet mask token tokenization

* Complete everything

* Apply suggestions from code review

2021-12-03 08:39:10 +01:00

185 KiB

Raw Blame History

View Raw