SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Merge branch 'pooler_end_logits_fp16_fix' of https://github.com/slayton58/pytorch-transformers into pr/1284

Browse Source
This commit is contained in:
thomwolf
2019-10-01 18:17:48 -04:00
parent 5c3b32d44d ec94f4e0f8
commit c50783e388
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
transformers/modeling_utils.py
View File

@@ -501,6 +501,9 @@ class PoolerEndLogits(nn.Module):
x = self.dense_1(x).squeeze(-1) x = self.dense_1(x).squeeze(-1)
if p_mask is not None: if p_mask is not None:
if next(self.parameters()).dtype == torch.float16:
x = x * (1 - p_mask) - 65500 * p_mask
else:
x = x * (1 - p_mask) - 1e30 * p_mask x = x * (1 - p_mask) - 1e30 * p_mask
return x return x