SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Fix gradient overflow issue during attention mask

Browse Source 
This fix is in reference to issue #382. GPT2 can now be trained in mixed precision, which I've confirmed with testing. I also tested unconditional generation on multiple seeds before and after changing 1e10 to 1e4 and there was no difference. Please let me know if there is anything else I can do to make this pull request better. Thanks for all your work!
This commit is contained in:
Abhi Sharma
2019-04-16 11:42:34 -07:00
committed by GitHub
parent 3d78e226e6
commit 9e666aaa29
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
pytorch_pretrained_bert/modeling_gpt2.py
View File

@@ -218,7 +218,7 @@ class Attention(nn.Module):
w = w / math.sqrt(v.size(-1)) w = w / math.sqrt(v.size(-1))
nd, ns = w.size(-2), w.size(-1) nd, ns = w.size(-2), w.size(-1)
b = self.bias[:, :, ns-nd:ns, :ns] b = self.bias[:, :, ns-nd:ns, :ns]
w = w * b - 1e10 * (1 - b) w = w * b - 1e4 * (1 - b)
w = nn.Softmax(dim=-1)(w) w = nn.Softmax(dim=-1)(w)
return torch.matmul(w, v) return torch.matmul(w, v)