Funtowicz Morgan 9e03364999 Ability to pickle/unpickle BatchEncoding pickle (reimport) (#5039) ...

* Added is_fast property on BatchEncoding to indicate if the object comes from a Fast Tokenizer.

* Added __get_state__() & __set_state__() to be pickable.

* Correct tokens() return type from List[int] to List[str]

* Added unittest for BatchEncoding pickle/unpickle

* Added unittest for BatchEncoding is_fast

* More careful checking on BatchEncoding unpickle tests.

* Formatting.

* is_fast should assertTrue on Rust tokenizers.

* Ensure tensorflow has correct way of checking array_equal

* More formatting.

2020-06-16 09:25:25 +02:00

5.7 KiB

Raw Blame History

View Raw