SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Fix for #3865. PretrainedTokenizer mapped " do not" into " don't" when .decode(...) is called. Removed the " do not" --> " don't" mapping from clean_up_tokenization(...). (#4024)

Browse Source
This commit is contained in:
Denis
2020-05-13 14:32:57 +02:00
committed by GitHub
parent 241759101e
commit 1e51bb717c
1 changed files with 0 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
src/transformers/tokenization_utils.py
View File

@@ -2195,7 +2195,6 @@ class PreTrainedTokenizer(SpecialTokensMixin):
.replace(" ' ", "'") .replace(" ' ", "'")
.replace(" n't", "n't") .replace(" n't", "n't")
.replace(" 'm", "'m") .replace(" 'm", "'m")
.replace(" do not", " don't")
.replace(" 's", "'s") .replace(" 's", "'s")
.replace(" 've", "'ve") .replace(" 've", "'ve")
.replace(" 're", "'re") .replace(" 're", "'re")