From 3c5c567507032da4c9dec00facf9ea4043e634eb Mon Sep 17 00:00:00 2001 From: HUSEIN ZOLKEPLI Date: Thu, 26 Mar 2020 19:50:27 +0800 Subject: [PATCH] Update model card huseinzol05/bert-base-bahasa-cased (#3425) * add bert bahasa readme * update readme * update readme * added xlnet --- .../bert-base-bahasa-cased/README.md | 47 +++++++++++++- .../xlnet-base-bahasa-cased/README.md | 64 +++++++++++++++++++ 2 files changed, 108 insertions(+), 3 deletions(-) create mode 100644 model_cards/huseinzol05/xlnet-base-bahasa-cased/README.md diff --git a/model_cards/huseinzol05/bert-base-bahasa-cased/README.md b/model_cards/huseinzol05/bert-base-bahasa-cased/README.md index 9b8cf0daee..46fe0ff442 100644 --- a/model_cards/huseinzol05/bert-base-bahasa-cased/README.md +++ b/model_cards/huseinzol05/bert-base-bahasa-cased/README.md @@ -32,13 +32,54 @@ Preprocessing steps can reproduce from here, [Malaya/pretrained-model/preprocess You can use this model by installing `torch` or `tensorflow` and Huggingface library `transformers`. And you can use it directly by initializing it like this: ```python -from transformers import XLNetTokenizer, BertModel +from transformers import AlbertTokenizer, BertModel model = BertModel.from_pretrained('huseinzol05/bert-base-bahasa-cased') -tokenizer = XLNetTokenizer.from_pretrained('huseinzol05/bert-base-bahasa-cased') +tokenizer = AlbertTokenizer.from_pretrained( + 'huseinzol05/bert-base-bahasa-cased', + unk_token = '[UNK]', + pad_token = '[PAD]', + do_lower_case = False, +) ``` -We use [google/sentencepiece](https://github.com/google/sentencepiece) to train the tokenizer, so to use it, need to load from `XLNetTokenizer`. +We use [google/sentencepiece](https://github.com/google/sentencepiece) to train the tokenizer, so to use it, need to load from `AlbertTokenizer`. + +## Example using AutoModelWithLMHead + +```python +from transformers import AlbertTokenizer, AutoModelWithLMHead, pipeline + +model = AutoModelWithLMHead.from_pretrained('huseinzol05/bert-base-bahasa-cased') +tokenizer = AlbertTokenizer.from_pretrained( + 'huseinzol05/bert-base-bahasa-cased', + unk_token = '[UNK]', + pad_token = '[PAD]', + do_lower_case = False, +) +fill_mask = pipeline('fill-mask', model = model, tokenizer = tokenizer) +print(fill_mask('makan ayam dengan [MASK]')) +``` + +Output is, + +```text +[{'sequence': '[CLS] makan ayam dengan rendang[SEP]', + 'score': 0.10812027007341385, + 'token': 2446}, + {'sequence': '[CLS] makan ayam dengan kicap[SEP]', + 'score': 0.07653367519378662, + 'token': 12928}, + {'sequence': '[CLS] makan ayam dengan nasi[SEP]', + 'score': 0.06839974224567413, + 'token': 450}, + {'sequence': '[CLS] makan ayam dengan ayam[SEP]', + 'score': 0.059544261544942856, + 'token': 638}, + {'sequence': '[CLS] makan ayam dengan sayur[SEP]', + 'score': 0.05294966697692871, + 'token': 1639}] +``` ## Results diff --git a/model_cards/huseinzol05/xlnet-base-bahasa-cased/README.md b/model_cards/huseinzol05/xlnet-base-bahasa-cased/README.md new file mode 100644 index 0000000000..a4762e6175 --- /dev/null +++ b/model_cards/huseinzol05/xlnet-base-bahasa-cased/README.md @@ -0,0 +1,64 @@ +--- +language: malay +--- + +# Bahasa XLNet Model + +Pretrained XLNet base language model for Malay and Indonesian. + +## Pretraining Corpus + +`XLNET-base-bahasa-cased` model was pretrained on ~1.8 Billion words. We trained on both standard and social media language structures, and below is list of data we trained on, + +1. [dumping wikipedia](https://github.com/huseinzol05/Malaya-Dataset#wikipedia-1). +2. [local instagram](https://github.com/huseinzol05/Malaya-Dataset#instagram). +3. [local twitter](https://github.com/huseinzol05/Malaya-Dataset#twitter-1). +4. [local news](https://github.com/huseinzol05/Malaya-Dataset#public-news). +5. [local parliament text](https://github.com/huseinzol05/Malaya-Dataset#parliament). +6. [local singlish/manglish text](https://github.com/huseinzol05/Malaya-Dataset#singlish-text). +7. [IIUM Confession](https://github.com/huseinzol05/Malaya-Dataset#iium-confession). +8. [Wattpad](https://github.com/huseinzol05/Malaya-Dataset#wattpad). +9. [Academia PDF](https://github.com/huseinzol05/Malaya-Dataset#academia-pdf). + +Preprocessing steps can reproduce from here, [Malaya/pretrained-model/preprocess](https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/preprocess). + +## Pretraining details + +- This model was trained using zihangdai XLNet's github [repository](https://github.com/zihangdai/xlnet) on 3 Titan V100 32GB VRAM. +- All steps can reproduce from here, [Malaya/pretrained-model/xlnet](https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/xlnet). + +## Load Pretrained Model + +You can use this model by installing `torch` or `tensorflow` and Huggingface library `transformers`. And you can use it directly by initializing it like this: + +```python +from transformers import XLNetTokenizer, XLNetModel + +model = XLNetModel.from_pretrained('huseinzol05/xlnet-base-bahasa-cased') +tokenizer = XLNetTokenizer.from_pretrained( + 'huseinzol05/xlnet-base-bahasa-cased', do_lower_case = False +) +``` + +## Example using AutoModelWithLMHead + +```python +from transformers import AlbertTokenizer, AutoModelWithLMHead, pipeline + +model = AutoModelWithLMHead.from_pretrained('huseinzol05/xlnet-base-bahasa-cased') +tokenizer = XLNetTokenizer.from_pretrained( + 'huseinzol05/xlnet-base-bahasa-cased', do_lower_case = False +) +fill_mask = pipeline('fill-mask', model = model, tokenizer = tokenizer) +print(fill_mask('makan ayam dengan [MASK]')) +``` + +## Results + +For further details on the model performance, simply checkout accuracy page from Malaya, https://malaya.readthedocs.io/en/latest/Accuracy.html, we compared with traditional models. + +## Acknowledgement + +Thanks to [Im Big](https://www.facebook.com/imbigofficial/), [LigBlou](https://www.facebook.com/ligblou), [Mesolitica](https://mesolitica.com/) and [KeyReply](https://www.keyreply.com/) for sponsoring AWS, Google and GPU clouds to train XLNet for Bahasa. + +