Thomas Wolf 315f464b0a [tokenizers] Several small improvements and bug fixes (#5287) ...

* avoid recursion in id checks for fast tokenizers

* better typings and fix #5232

* align slow and fast tokenizers behaviors for Roberta and GPT2

* style and quality

* fix tests - improve typings

2020-06-25 22:17:14 +02:00

..

transformers

[tokenizers] Several small improvements and bug fixes (#5287)

2020-06-25 22:17:14 +02:00