Nicolas Patry a8c3f9aa76 Warning about too long input for fast tokenizers too (#8799) ...

* Warning about too long input for fast tokenizers too

If truncation is not set in tokenizers, but the tokenization is too long
for the model (`model_max_length`), we used to trigger a warning that

The input would probably fail (which it most likely will).

This PR re-enables the warning for fast tokenizers too and uses common
code for the trigger to make sure it's consistent across.

* Checking for pair of inputs too.

* Making the function private and adding it's doc.

* Remove formatting ?? in odd place.

* Missed uppercase.

2020-12-02 10:18:28 -05:00

141 KiB

Raw Blame History

View Raw