From 074340339a6d6aede30c14c94ffe7b59a01786f1 Mon Sep 17 00:00:00 2001 From: Zane Lim Date: Wed, 26 Aug 2020 00:11:24 +0800 Subject: [PATCH] Create README.md (#6721) add model card for singbert large --- .../zanelim/singbert-large-sg/README.md | 215 ++++++++++++++++++ 1 file changed, 215 insertions(+) create mode 100644 model_cards/zanelim/singbert-large-sg/README.md diff --git a/model_cards/zanelim/singbert-large-sg/README.md b/model_cards/zanelim/singbert-large-sg/README.md new file mode 100644 index 0000000000..a38f50db80 --- /dev/null +++ b/model_cards/zanelim/singbert-large-sg/README.md @@ -0,0 +1,215 @@ +--- +language: en +tags: +- singapore +- sg +- singlish +- malaysia +- ms +- manglish +- bert-large-uncased +license: mit +datasets: +- reddit singapore, malaysia +- hardwarezone +widget: +- text: "die [MASK] must try" +- text: "kopi c siew [MASK]" +--- + +# Model name + +SingBert - Bert for Singlish (SG) and Manglish (MY). + +## Model description + +Similar to [SingBert](https://huggingface.co/zanelim/singbert) but initialized from [BERT large uncased (whole word masking)](https://github.com/google-research/bert#pre-trained-models), with pre-training finetuned on +[singlish](https://en.wikipedia.org/wiki/Singlish) and [manglish](https://en.wikipedia.org/wiki/Manglish) data. + +## Intended uses & limitations + +#### How to use + +```python +>>> from transformers import pipeline +>>> nlp = pipeline('fill-mask', model='zanelim/singbert-large-sg') +>>> nlp("kopi c siew [MASK]") + +[{'sequence': '[CLS] kopi c siew dai [SEP]', + 'score': 0.9003700017929077, + 'token': 18765, + 'token_str': 'dai'}, + {'sequence': '[CLS] kopi c siew mai [SEP]', + 'score': 0.0779474675655365, + 'token': 14736, + 'token_str': 'mai'}, + {'sequence': '[CLS] kopi c siew. [SEP]', + 'score': 0.0032227332703769207, + 'token': 1012, + 'token_str': '.'}, + {'sequence': '[CLS] kopi c siew bao [SEP]', + 'score': 0.0017727474914863706, + 'token': 25945, + 'token_str': 'bao'}, + {'sequence': '[CLS] kopi c siew peng [SEP]', + 'score': 0.0012526646023616195, + 'token': 26473, + 'token_str': 'peng'}] + +>>> nlp("one teh c siew dai, and one kopi [MASK]") + +[{'sequence': '[CLS] one teh c siew dai, and one kopi. [SEP]', + 'score': 0.5249741077423096, + 'token': 1012, + 'token_str': '.'}, + {'sequence': '[CLS] one teh c siew dai, and one kopi o [SEP]', + 'score': 0.27349168062210083, + 'token': 1051, + 'token_str': 'o'}, + {'sequence': '[CLS] one teh c siew dai, and one kopi peng [SEP]', + 'score': 0.057190295308828354, + 'token': 26473, + 'token_str': 'peng'}, + {'sequence': '[CLS] one teh c siew dai, and one kopi c [SEP]', + 'score': 0.04022320732474327, + 'token': 1039, + 'token_str': 'c'}, + {'sequence': '[CLS] one teh c siew dai, and one kopi? [SEP]', + 'score': 0.01191170234233141, + 'token': 1029, + 'token_str': '?'}] + +>>> nlp("die [MASK] must try") + +[{'sequence': '[CLS] die die must try [SEP]', + 'score': 0.9921030402183533, + 'token': 3280, + 'token_str': 'die'}, + {'sequence': '[CLS] die also must try [SEP]', + 'score': 0.004993876442313194, + 'token': 2036, + 'token_str': 'also'}, + {'sequence': '[CLS] die liao must try [SEP]', + 'score': 0.000317625846946612, + 'token': 727, + 'token_str': 'liao'}, + {'sequence': '[CLS] die still must try [SEP]', + 'score': 0.0002260878391098231, + 'token': 2145, + 'token_str': 'still'}, + {'sequence': '[CLS] die i must try [SEP]', + 'score': 0.00016935862367972732, + 'token': 1045, + 'token_str': 'i'}] + +>>> nlp("dont play [MASK] leh") + +[{'sequence': '[CLS] dont play play leh [SEP]', + 'score': 0.9079819321632385, + 'token': 2377, + 'token_str': 'play'}, + {'sequence': '[CLS] dont play punk leh [SEP]', + 'score': 0.006846973206847906, + 'token': 7196, + 'token_str': 'punk'}, + {'sequence': '[CLS] dont play games leh [SEP]', + 'score': 0.004041737411171198, + 'token': 2399, + 'token_str': 'games'}, + {'sequence': '[CLS] dont play politics leh [SEP]', + 'score': 0.003728888463228941, + 'token': 4331, + 'token_str': 'politics'}, + {'sequence': '[CLS] dont play cheat leh [SEP]', + 'score': 0.0032805048394948244, + 'token': 21910, + 'token_str': 'cheat'}] + +>>> nlp("confirm plus [MASK]") + +{'sequence': '[CLS] confirm plus chop [SEP]', + 'score': 0.9749826192855835, + 'token': 24494, + 'token_str': 'chop'}, + {'sequence': '[CLS] confirm plus chopped [SEP]', + 'score': 0.017554156482219696, + 'token': 24881, + 'token_str': 'chopped'}, + {'sequence': '[CLS] confirm plus minus [SEP]', + 'score': 0.002725469646975398, + 'token': 15718, + 'token_str': 'minus'}, + {'sequence': '[CLS] confirm plus guarantee [SEP]', + 'score': 0.000900257145985961, + 'token': 11302, + 'token_str': 'guarantee'}, + {'sequence': '[CLS] confirm plus one [SEP]', + 'score': 0.0004384620988275856, + 'token': 2028, + 'token_str': 'one'}] + +>>> nlp("catch no [MASK]") + +[{'sequence': '[CLS] catch no ball [SEP]', + 'score': 0.9381157159805298, + 'token': 3608, + 'token_str': 'ball'}, + {'sequence': '[CLS] catch no balls [SEP]', + 'score': 0.060842301696538925, + 'token': 7395, + 'token_str': 'balls'}, + {'sequence': '[CLS] catch no fish [SEP]', + 'score': 0.00030917322146706283, + 'token': 3869, + 'token_str': 'fish'}, + {'sequence': '[CLS] catch no breath [SEP]', + 'score': 7.552534952992573e-05, + 'token': 3052, + 'token_str': 'breath'}, + {'sequence': '[CLS] catch no tail [SEP]', + 'score': 4.208395694149658e-05, + 'token': 5725, + 'token_str': 'tail'}] + +``` + +Here is how to use this model to get the features of a given text in PyTorch: +```python +from transformers import BertTokenizer, BertModel +tokenizer = BertTokenizer.from_pretrained('zanelim/singbert-large-sg') +model = BertModel.from_pretrained("zanelim/singbert-large-sg") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='pt') +output = model(**encoded_input) +``` + +and in TensorFlow: +```python +from transformers import BertTokenizer, TFBertModel +tokenizer = BertTokenizer.from_pretrained("zanelim/singbert-large-sg") +model = TFBertModel.from_pretrained("zanelim/singbert-large-sg") +text = "Replace me by any text you'd like." +encoded_input = tokenizer(text, return_tensors='tf') +output = model(encoded_input) +``` + +#### Limitations and bias +This model was finetuned on colloquial Singlish and Manglish corpus, hence it is best applied on downstream tasks involving the main +constituent languages- english, mandarin, malay. Also, as the training data is mainly from forums, beware of existing inherent bias. + +## Training data +Colloquial singlish and manglish (both are a mixture of English, Mandarin, Tamil, Malay, and other local dialects like Hokkien, Cantonese or Teochew) +corpus. The corpus is collected from subreddits- `r/singapore` and `r/malaysia`, and forums such as `hardwarezone`. + +## Training procedure + +Initialized with [bert large uncased (whole word masking)](https://github.com/google-research/bert#pre-trained-models) vocab and checkpoints (pre-trained weights). +Top 1000 custom vocab tokens (non-overlapped with original bert vocab) were further extracted from training data and filled into unused tokens in original bert vocab. + +Pre-training was further finetuned on training data with the following hyperparameters +* train_batch_size: 512 +* max_seq_length: 128 +* num_train_steps: 300000 +* num_warmup_steps: 5000 +* learning_rate: 2e-5 +* hardware: TPU v3-8