From bf99014c460f69ac1aeb35870bf2766936b3964e Mon Sep 17 00:00:00 2001 From: Kevin Canwen Xu Date: Mon, 10 Feb 2020 09:37:53 +0800 Subject: [PATCH] Create BERT-of-Theseus model card --- .../canwenxu/BERT-of-Theseus-MNLI/README.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 model_cards/canwenxu/BERT-of-Theseus-MNLI/README.md diff --git a/model_cards/canwenxu/BERT-of-Theseus-MNLI/README.md b/model_cards/canwenxu/BERT-of-Theseus-MNLI/README.md new file mode 100644 index 0000000000..92471d2d9a --- /dev/null +++ b/model_cards/canwenxu/BERT-of-Theseus-MNLI/README.md @@ -0,0 +1,16 @@ +# BERT-of-Theseus +See our paper ["BERT-of-Theseus: Compressing BERT by Progressive Module Replacing"](http://arxiv.org/abs/2002.02925). + +BERT-of-Theseus is a new compressed BERT by progressively replacing the components of the original BERT. + +![BERT of Theseus](https://github.com/JetRunner/BERT-of-Theseus/blob/master/bert-of-theseus.png?raw=true) + +## Load Pretrained Model on MNLI + +We provide a 6-layer pretrained model on MNLI as a general-purpose model, which can transfer to other sentence classification tasks, outperforming DistillBERT (with the same 6-layer structure) on six tasks of GLUE (dev set). + +| Method | MNLI | MRPC | QNLI | QQP | RTE | SST-2 | STS-B | +|-----------------|------|------|------|------|------|-------|-------| +| BERT-base | 83.5 | 89.5 | 91.2 | 89.8 | 71.1 | 91.5 | 88.9 | +| DistillBERT | 79.0 | 87.5 | 85.3 | 84.9 | 59.9 | 90.7 | 81.2 | +| BERT-of-Theseus | 82.1 | 87.5 | 88.8 | 88.8 | 70.1 | 91.8 | 87.8 |