From 3363a19b120fdb319ab8c97938cd996ad24904c9 Mon Sep 17 00:00:00 2001 From: Adriano Diniz Date: Mon, 22 Jun 2020 18:59:33 -0300 Subject: [PATCH] Create README.md (#5152) * Create README.md * Apply suggestions from code review Co-authored-by: Julien Chaumond --- .../README.md | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 model_cards/aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616_squad2/README.md diff --git a/model_cards/aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616_squad2/README.md b/model_cards/aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616_squad2/README.md new file mode 100644 index 0000000000..57b955182b --- /dev/null +++ b/model_cards/aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616_squad2/README.md @@ -0,0 +1,26 @@ +--- +datasets: +- squad_v2 +--- + +# BERT L-10 H-512 CORD-19 (2020/06/16) fine-tuned on SQuAD v2.0 + +BERT model with [10 Transformer layers and hidden embedding of size 512](https://huggingface.co/google/bert_uncased_L-10_H-512_A-8), referenced in [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://arxiv.org/abs/1908.08962), [fine-tuned for MLM](https://huggingface.co/aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616) on CORD-19 dataset (as released on 2020/06/16) and fine-tuned for QA on SQuAD v2.0. + +## Training the model + +```bash +python run_squad.py + --model_type bert + --model_name_or_path aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616 + --train_file 'train-v2.0.json' + --predict_file 'dev-v2.0.json' + --do_train + --do_eval + --do_lower_case + --version_2_with_negative + --max_seq_length 384 + --per_gpu_train_batch_size 10 + --learning_rate 3e-5 + --num_train_epochs 2 + --output_dir bert_uncased_L-10_H-512_A-8_cord19-200616_squad2