From 9209d36f9329fe9a0c1160d67ab6a1b4f4eec707 Mon Sep 17 00:00:00 2001
From: Pradhy729 <49659913+Pradhy729@users.noreply.github.com>
Date: Mon, 29 Jun 2020 01:29:14 -0700
Subject: [PATCH] Added a model card README.md for my pretrained model. (#5325)

* Create README.md

* Removed unnecessary link from README.md

* Update README.md
---
 model_cards/pradhyra/AWSBlogBert/README.md | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 model_cards/pradhyra/AWSBlogBert/README.md

diff --git a/model_cards/pradhyra/AWSBlogBert/README.md b/model_cards/pradhyra/AWSBlogBert/README.md
new file mode 100644
index 0000000000..b182a398be
--- /dev/null
+++ b/model_cards/pradhyra/AWSBlogBert/README.md
@@ -0,0 +1,8 @@
+This model is pre-trained on blog articles from AWS Blogs.
+
+## Pre-training corpora
+The input text contains around 3000 blog articles on [AWS Blogs website](https://aws.amazon.com/blogs/) technical subject matter including AWS products, tools and tutorials. 
+
+## Pre-training details
+I picked a Roberta architecture for masked language modeling (6-layer, 768-hidden, 12-heads, 82M parameters) and its corresponding ByteLevelBPE tokenization strategy. I then followed HuggingFace's Transformers [blog post](https://huggingface.co/blog/how-to-train) to train the model.
+I chose to follow the following training set-up: 28k training steps with batches of 64 sequences of length 512 with an initial learning rate 5e-5. The model acheived a training loss of 3.6 on the MLM task over 10 epochs.