Joe Davison f1e8a51f08 Preserve spaces in GPT-2 tokenizers (#2778) ...

* Preserve spaces in GPT-2 tokenizers

Preserves spaces after special tokens in GPT-2 and inhereted (RoBERTa)
tokenizers, enabling correct BPE encoding. Automatically inserts a space
in front of first token in encode function when adding special tokens.

* Add tokenization preprocessing method

* Add framework argument to pipeline factory

Also fixes pipeline test issue. Each test input now treated as a
distinct sequence.

2020-02-13 13:29:43 -05:00

5.7 KiB

Raw Blame History

View Raw