From 216e167ce6a7558253bc8543bc6fa357629c8736 Mon Sep 17 00:00:00 2001 From: HUSEIN ZOLKEPLI Date: Fri, 3 Apr 2020 21:28:43 +0800 Subject: [PATCH] Added albert-base-bahasa-cased README and fixed tiny-bert-bahasa-cased README (#3613) * add bert bahasa readme * update readme * update readme * added xlnet * added tiny-bert and fix xlnet readme * added albert base --- .../albert-base-bahasa-cased/README.md | 86 +++++++++++++++++++ .../tiny-bert-bahasa-cased/README.md | 8 +- 2 files changed, 90 insertions(+), 4 deletions(-) create mode 100644 model_cards/huseinzol05/albert-base-bahasa-cased/README.md diff --git a/model_cards/huseinzol05/albert-base-bahasa-cased/README.md b/model_cards/huseinzol05/albert-base-bahasa-cased/README.md new file mode 100644 index 0000000000..27f5630860 --- /dev/null +++ b/model_cards/huseinzol05/albert-base-bahasa-cased/README.md @@ -0,0 +1,86 @@ +--- +language: malay +--- + +# Bahasa Albert Model + +Pretrained Albert base language model for Malay and Indonesian. + +## Pretraining Corpus + +`albert-base-bahasa-cased` model was pretrained on ~1.8 Billion words. We trained on both standard and social media language structures, and below is list of data we trained on, + +1. [dumping wikipedia](https://github.com/huseinzol05/Malaya-Dataset#wikipedia-1). +2. [local instagram](https://github.com/huseinzol05/Malaya-Dataset#instagram). +3. [local twitter](https://github.com/huseinzol05/Malaya-Dataset#twitter-1). +4. [local news](https://github.com/huseinzol05/Malaya-Dataset#public-news). +5. [local parliament text](https://github.com/huseinzol05/Malaya-Dataset#parliament). +6. [local singlish/manglish text](https://github.com/huseinzol05/Malaya-Dataset#singlish-text). +7. [IIUM Confession](https://github.com/huseinzol05/Malaya-Dataset#iium-confession). +8. [Wattpad](https://github.com/huseinzol05/Malaya-Dataset#wattpad). +9. [Academia PDF](https://github.com/huseinzol05/Malaya-Dataset#academia-pdf). + +Preprocessing steps can reproduce from here, [Malaya/pretrained-model/preprocess](https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/preprocess). + +## Pretraining details + +- This model was trained using Google Albert's github [repository](https://github.com/google-research/ALBERT) on v3-8 TPU. +- All steps can reproduce from here, [Malaya/pretrained-model/albert](https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/albert). + +## Load Pretrained Model + +You can use this model by installing `torch` or `tensorflow` and Huggingface library `transformers`. And you can use it directly by initializing it like this: + +```python +from transformers import AlbertTokenizer, AlbertModel + +model = BertModel.from_pretrained('huseinzol05/albert-base-bahasa-cased') +tokenizer = AlbertTokenizer.from_pretrained( + 'huseinzol05/albert-base-bahasa-cased', + do_lower_case = False, +) +``` + +## Example using AutoModelWithLMHead + +```python +from transformers import AlbertTokenizer, AutoModelWithLMHead, pipeline + +model = AutoModelWithLMHead.from_pretrained('huseinzol05/albert-base-bahasa-cased') +tokenizer = AlbertTokenizer.from_pretrained( + 'huseinzol05/albert-base-bahasa-cased', + do_lower_case = False, +) +fill_mask = pipeline('fill-mask', model = model, tokenizer = tokenizer) +print(fill_mask('makan ayam dengan [MASK]')) +``` + +Output is, + +```text +[{'sequence': '[CLS] makan ayam dengan ayam[SEP]', + 'score': 0.044952988624572754, + 'token': 629}, + {'sequence': '[CLS] makan ayam dengan sayur[SEP]', + 'score': 0.03621877357363701, + 'token': 1639}, + {'sequence': '[CLS] makan ayam dengan ikan[SEP]', + 'score': 0.034429922699928284, + 'token': 758}, + {'sequence': '[CLS] makan ayam dengan nasi[SEP]', + 'score': 0.032447945326566696, + 'token': 453}, + {'sequence': '[CLS] makan ayam dengan rendang[SEP]', + 'score': 0.028885239735245705, + 'token': 2451}] +``` + +## Results + +For further details on the model performance, simply checkout accuracy page from Malaya, https://malaya.readthedocs.io/en/latest/Accuracy.html, we compared with traditional models. + +## Acknowledgement + +Thanks to [Im Big](https://www.facebook.com/imbigofficial/), [LigBlou](https://www.facebook.com/ligblou), [Mesolitica](https://mesolitica.com/) and [KeyReply](https://www.keyreply.com/) for sponsoring AWS, Google and GPU clouds to train Albert for Bahasa. + + diff --git a/model_cards/huseinzol05/tiny-bert-bahasa-cased/README.md b/model_cards/huseinzol05/tiny-bert-bahasa-cased/README.md index 8712d0fbff..967e870e99 100644 --- a/model_cards/huseinzol05/tiny-bert-bahasa-cased/README.md +++ b/model_cards/huseinzol05/tiny-bert-bahasa-cased/README.md @@ -4,7 +4,7 @@ language: malay # Bahasa Tiny-BERT Model -General Distilled Tiny BERT base language model for Malay and Indonesian. +General Distilled Tiny BERT language model for Malay and Indonesian. ## Pretraining Corpus @@ -36,7 +36,7 @@ from transformers import AlbertTokenizer, BertModel model = BertModel.from_pretrained('huseinzol05/tiny-bert-bahasa-cased') tokenizer = AlbertTokenizer.from_pretrained( - 'huseinzol05/tiny-base-bahasa-cased', + 'huseinzol05/tiny-bert-bahasa-cased', unk_token = '[UNK]', pad_token = '[PAD]', do_lower_case = False, @@ -50,9 +50,9 @@ We use [google/sentencepiece](https://github.com/google/sentencepiece) to train ```python from transformers import AlbertTokenizer, AutoModelWithLMHead, pipeline -model = AutoModelWithLMHead.from_pretrained('huseinzol05/tiny-base-bahasa-cased') +model = AutoModelWithLMHead.from_pretrained('huseinzol05/tiny-bert-bahasa-cased') tokenizer = AlbertTokenizer.from_pretrained( - 'huseinzol05/tiny-base-bahasa-cased', + 'huseinzol05/tiny-bert-bahasa-cased', unk_token = '[UNK]', pad_token = '[PAD]', do_lower_case = False,