Li-Huai (Allan) Lin da8beaaf76 Fix special tokens not correctly tokenized (#13489) ...

* Fix special tokens not correctly tokenized

* Add testing

* Fix

* Fix

* Use user workflows instead of directly assigning variables

* Enable test of fast tokenizers

* Update test of canine tokenizer

2021-09-17 10:28:28 -04:00

14 KiB

Raw Blame History

View Raw