SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

make save_pretrained work with added tokens

Browse Source 
right now it's dumping the *decoder* when it should be dumping the *encoder*. this fixes that.
This commit is contained in:
Joel Grus
2019-07-24 16:54:48 -07:00
committed by GitHub
parent 067923d326
commit ae152cec09
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
pytorch_transformers/tokenization_utils.py
View File

@@ -266,7 +266,7 @@ class PreTrainedTokenizer(object):
with open(added_tokens_file, 'w', encoding='utf-8') as f: with open(added_tokens_file, 'w', encoding='utf-8') as f:
if self.added_tokens_encoder: if self.added_tokens_encoder:
out_str = json.dumps(self.added_tokens_decoder, ensure_ascii=False) out_str = json.dumps(self.added_tokens_encoder, ensure_ascii=False)
else: else:
out_str = u"{}" out_str = u"{}"
f.write(out_str) f.write(out_str)