From 233072fc1eec76eb2be81c476501381f71eabbd9 Mon Sep 17 00:00:00 2001 From: Nick Doiron Date: Thu, 16 Jul 2020 15:13:51 -0400 Subject: [PATCH] dv-wave (#5823) --- model_cards/monsoon-nlp/dv-wave/README.md | 28 +++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 model_cards/monsoon-nlp/dv-wave/README.md diff --git a/model_cards/monsoon-nlp/dv-wave/README.md b/model_cards/monsoon-nlp/dv-wave/README.md new file mode 100644 index 0000000000..df24a541ba --- /dev/null +++ b/model_cards/monsoon-nlp/dv-wave/README.md @@ -0,0 +1,28 @@ +--- +language: dv +--- + +# dv-wave + +This is a second attempt at a Dhivehi language model trained with +Google Research's [ELECTRA](https://github.com/google-research/electra). + +Tokenization and pre-training CoLab: https://colab.research.google.com/drive/1ZJ3tU9MwyWj6UtQ-8G7QJKTn-hG1uQ9v?usp=sharing + +Using SimpleTransformers to classify news https://colab.research.google.com/drive/1KnyQxRNWG_yVwms_x9MUAqFQVeMecTV7?usp=sharing + +V1: similar performance to mBERT on news classification task after finetuning for 3 epochs (52%) + +V2: fixed tokenizers ```do_lower_case=False``` and ```strip_accents=False``` to preserve vowel signs of Dhivehi + dv-wave: 89% to mBERT: 52% + +## Corpus + +Trained on @Sofwath's 307MB corpus of Dhivehi text: https://github.com/Sofwath/DhivehiDatasets - this repo also contains the news classification task CSV + +[OSCAR](https://oscar-corpus.com/) was considered but has not been added to pretraining; as of +this writing their web crawl has 126MB of Dhivehi text (79MB deduped). + +## Vocabulary + +Included as vocab.txt in the upload - vocab_size is 29874