SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Merge pull request #1162 from huggingface/xlnet-bias

Browse Source 
XLNet bias fix on resize embeddings (cf #1124)
This commit is contained in:
Thomas Wolf
2019-09-02 23:14:04 +02:00
committed by GitHub
parent 7f522437bc ea86bef545
commit 0287d264e9
1 changed files with 8 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
pytorch_transformers/modeling_utils.py
View File

@@ -331,6 +331,14 @@ class PreTrainedModel(nn.Module):
else: else:
first_module.weight = second_module.weight first_module.weight = second_module.weight
if hasattr(first_module, 'bias') and first_module.bias is not None:
first_module.bias.data = torch.nn.functional.pad(
first_module.bias.data,
(0, first_module.weight.shape[0] - first_module.bias.shape[0]),
'constant',
0
)
def resize_token_embeddings(self, new_num_tokens=None): def resize_token_embeddings(self, new_num_tokens=None):
""" Resize input token embeddings matrix of the model if new_num_tokens != config.vocab_size. """ Resize input token embeddings matrix of the model if new_num_tokens != config.vocab_size.
Take care of tying weights embeddings afterwards if the model class has a `tie_weights()` method. Take care of tying weights embeddings afterwards if the model class has a `tie_weights()` method.