T5Tokenizer adds EOS token if not already added (#5866)

Browse Source

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

...

This commit is contained in:

Sam Shleifer

2020-08-25 14:56:08 -04:00

committed by GitHub

parent e11d923bfc

commit 624495706c

3 changed files with 117 additions and 50 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

39

tests/test_modeling_t5.py

File diff suppressed because one or more lines are too long