Arthur f9cc333805 [ PreTrainedTokenizerFast] Keep properties from fast tokenizer (#25053) ...

* draft solution

* use `setdefault`

* nits

* add tests and fix truncation issue

* fix test

* test passes locally

* quality

* updates

* update tsets

2023-07-25 18:45:01 +02:00

..

__init__.py

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

test_tokenization_fast.py

[ PreTrainedTokenizerFast] Keep properties from fast tokenizer (#25053)

2023-07-25 18:45:01 +02:00

test_tokenization_utils.py

Update quality tooling for formatting (#21480)

2023-02-06 18:10:56 -05:00