Nicolas Patry 08cb5718ec Enabling tokenizers upgrade. (#14941) ...

* Enabling `tokenizers` upgrade.

* Moved ugly comment.

* Tokenizers==0.11.1 needs an update to keep borrow checker

happy in highly contiguous calls.

* Support both 0.11.1 and 0.11.0

2021-12-30 17:30:58 +01:00

13 KiB

Raw Blame History

View Raw