SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Merge pull request #1753 from digantamisra98/patch-1

Browse Source 
Added Mish Activation Function
This commit is contained in:
Thomas Wolf
2019-11-28 15:24:08 +01:00
committed by GitHub
parent 96e7ee7238 070dcf1c02
commit 49a69d5b78
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
transformers/modeling_bert.py
View File

@@ -138,7 +138,11 @@ def swish(x):
return x * torch.sigmoid(x) return x * torch.sigmoid(x)
ACT2FN = {"gelu": gelu, "relu": torch.nn.functional.relu, "swish": swish, "gelu_new": gelu_new} def mish(x):
return x * torch.tanh(nn.functional.softplus(x))
ACT2FN = {"gelu": gelu, "relu": torch.nn.functional.relu, "swish": swish, "gelu_new": gelu_new, "mish": mish}
BertLayerNorm = torch.nn.LayerNorm BertLayerNorm = torch.nn.LayerNorm