From 8ed47aa10b280bc0469e94f3d31deeda2658344f Mon Sep 17 00:00:00 2001
From: David Mezzetti <561939+davidmezzetti@users.noreply.github.com>
Date: Wed, 3 Jun 2020 03:40:14 -0400
Subject: [PATCH] bert-small-cord19 model cards (#4730)

* Create README.md

* Create README.md

* Create README.md
---
 .../NeuML/bert-small-cord19-squad2/README.md  | 26 ++++++++
 model_cards/NeuML/bert-small-cord19/README.md | 25 ++++++++
 .../NeuML/bert-small-cord19qa/README.md       | 63 +++++++++++++++++++
 3 files changed, 114 insertions(+)
 create mode 100644 model_cards/NeuML/bert-small-cord19-squad2/README.md
 create mode 100644 model_cards/NeuML/bert-small-cord19/README.md
 create mode 100644 model_cards/NeuML/bert-small-cord19qa/README.md

diff --git a/model_cards/NeuML/bert-small-cord19-squad2/README.md b/model_cards/NeuML/bert-small-cord19-squad2/README.md
new file mode 100644
index 0000000000..94e86be7e9
--- /dev/null
+++ b/model_cards/NeuML/bert-small-cord19-squad2/README.md
@@ -0,0 +1,26 @@
+# BERT-Small CORD-19 fine-tuned on SQuAD 2.0
+
+[bert-small-cord19 model](https://huggingface.co/NeuML/bert-small-cord19) fine-tuned on SQuAD 2.0
+
+## Building the model
+
+```bash
+python run_squad.py
+    --model_type bert
+    --model_name_or_path bert-small-cord19
+    --do_train
+    --do_eval
+    --do_lower_case
+    --version_2_with_negative
+    --train_file train-v2.0.json
+    --predict_file dev-v2.0.json
+    --per_gpu_train_batch_size 8
+    --learning_rate 3e-5
+    --num_train_epochs 3.0
+    --max_seq_length 384
+    --doc_stride 128
+    --output_dir bert-small-cord19-squad2
+    --save_steps 0
+    --threads 8
+    --overwrite_cache
+    --overwrite_output_dir
diff --git a/model_cards/NeuML/bert-small-cord19/README.md b/model_cards/NeuML/bert-small-cord19/README.md
new file mode 100644
index 0000000000..60486535e8
--- /dev/null
+++ b/model_cards/NeuML/bert-small-cord19/README.md
@@ -0,0 +1,25 @@
+# BERT-Small fine-tuned on CORD-19 dataset
+
+[BERT L6_H-512_A-8 model](https://huggingface.co/google/bert_uncased_L-6_H-512_A-8) fine-tuned on the [CORD-19 dataset](https://www.semanticscholar.org/cord19).
+
+## CORD-19 data subset
+The training data for this dataset is stored as a [Kaggle dataset](https://www.kaggle.com/davidmezzetti/cord19-qa?select=cord19.txt). The training
+data is a subset of the full corpus, focusing on high-quality, study-design detected articles.
+
+## Building the model
+
+```bash
+python run_language_modeling.py
+    --model_type bert
+    --model_name_or_path google/bert_uncased_L-6_H-512_A-8
+    --do_train
+    --mlm
+    --line_by_line
+    --block_size 512
+    --train_data_file cord19.txt
+    --per_gpu_train_batch_size 4
+    --learning_rate 3e-5
+    --num_train_epochs 3.0
+    --output_dir bert-small-cord19
+    --save_steps 0
+    --overwrite_output_dir
diff --git a/model_cards/NeuML/bert-small-cord19qa/README.md b/model_cards/NeuML/bert-small-cord19qa/README.md
new file mode 100644
index 0000000000..f9f3c492a2
--- /dev/null
+++ b/model_cards/NeuML/bert-small-cord19qa/README.md
@@ -0,0 +1,63 @@
+# BERT-Small fine-tuned on CORD-19 QA dataset
+
+[bert-small-cord19-squad model](https://huggingface.co/NeuML/bert-small-cord19-squad2) fine-tuned on the [CORD-19 QA dataset](https://www.kaggle.com/davidmezzetti/cord19-qa?select=cord19-qa.json).
+
+## CORD-19 QA dataset
+The CORD-19 QA dataset is a SQuAD 2.0 formatted list of question, context, answer combinations covering the [CORD-19 dataset](https://www.semanticscholar.org/cord19).
+
+## Building the model
+
+```bash
+python run_squad.py \
+    --model_type bert \
+    --model_name_or_path bert-small-cord19-squad \
+    --do_train \
+    --do_lower_case \
+    --version_2_with_negative \
+    --train_file cord19-qa.json \
+    --per_gpu_train_batch_size 8 \
+    --learning_rate 5e-5 \
+    --num_train_epochs 10.0 \
+    --max_seq_length 384 \
+    --doc_stride 128 \
+    --output_dir bert-small-cord19qa \
+    --save_steps 0 \
+    --threads 8 \
+    --overwrite_cache \
+    --overwrite_output_dir
+```
+
+## Testing the model
+
+Example usage below:
+
+```python
+from transformers import pipeline
+
+qa = pipeline(
+    "question-answering",
+    model="NeuML/bert-small-cord19qa",
+    tokenizer="NeuML/bert-small-cord19qa"
+)
+
+qa({
+    "question": "What is the median incubation period?",
+    "context": "The incubation period is around 5 days (range: 4-7 days) with a maximum of 12-13 day"
+})
+
+qa({
+    "question": "What is the incubation period range?",
+    "context": "The incubation period is around 5 days (range: 4-7 days) with a maximum of 12-13 day"
+})
+
+qa({
+    "question": "What type of surfaces does it persist?",
+    "context": "The virus can survive on surfaces for up to 72 hours such as plastic and stainless steel ."
+})
+```
+
+```json
+{"score": 0.5970273583242793, "start": 32, "end": 38, "answer": "5 days"}
+{"score": 0.999555868193891, "start": 39, "end": 56, "answer": "(range: 4-7 days)"}
+{"score": 0.9992726505196998, "start": 61, "end": 88, "answer": "plastic and stainless steel"}
+```