From 8cd57eb73107de0353d1fe94cfe99484b738172a Mon Sep 17 00:00:00 2001 From: Reshan Gomis Date: Sat, 5 Apr 2025 02:58:35 +0530 Subject: [PATCH] mobilebert model card update (#37256) * mobilebert model card update * Updates to model card mobilebert --------- Co-authored-by: Reshan Gomis --- docs/source/en/model_doc/mobilebert.md | 97 +++++++++++++++++--------- 1 file changed, 63 insertions(+), 34 deletions(-) diff --git a/docs/source/en/model_doc/mobilebert.md b/docs/source/en/model_doc/mobilebert.md index 11a2b21b61..2104d0a457 100644 --- a/docs/source/en/model_doc/mobilebert.md +++ b/docs/source/en/model_doc/mobilebert.md @@ -14,52 +14,81 @@ rendered properly in your Markdown viewer. --> -# MobileBERT -
-PyTorch -TensorFlow +
+
+ PyTorch + TensorFlow +
-## Overview +# MobileBERT -The MobileBERT model was proposed in [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://arxiv.org/abs/2004.02984) by Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny -Zhou. It's a bidirectional transformer based on the BERT model, which is compressed and accelerated using several -approaches. +[MobileBERT](https://huggingface.co/papers/2004.02984) is a lightweight and efficient variant of BERT, specifically designed for resource-limited devices such as mobile phones. It retains BERT's architecture but significantly reduces model size and inference latency while maintaining strong performance on NLP tasks. MobileBERT achieves this through a bottleneck structure and carefully balanced self-attention and feedforward networks. The model is trained by knowledge transfer from a large BERT model with an inverted bottleneck structure. -The abstract from the paper is the following: +You can find the original MobileBERT checkpoint under the [Google](https://huggingface.co/google/mobilebert-uncased) organization. +> [!TIP] +> Click on the MobileBERT models in the right sidebar for more examples of how to apply MobileBERT to different language tasks. -*Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds -of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot -be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating -the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to -various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while -equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. -To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE -model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is -4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the -natural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6 lower than BERT_BASE), and 62 ms -latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of -90.0/79.2 (1.5/2.1 higher than BERT_BASE).* +The example below demonstrates how to predict the `[MASK]` token with [`Pipeline`], [`AutoModel`], and from the command line. -This model was contributed by [vshampor](https://huggingface.co/vshampor). The original code can be found [here](https://github.com/google-research/google-research/tree/master/mobilebert). + + -## Usage tips +```py +import torch +from transformers import pipeline -- MobileBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather - than the left. -- MobileBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore - efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained - with a causal language modeling (CLM) objective are better in that regard. +pipeline = pipeline( + task="fill-mask", + model="google/mobilebert-uncased", + torch_dtype=torch.float16, + device=0 +) +pipeline("The capital of France is [MASK].") +``` + + + +```py +import torch +from transformers import AutoModelForMaskedLM, AutoTokenizer + +tokenizer = AutoTokenizer.from_pretrained( + "google/mobilebert-uncased", +) +model = AutoModelForMaskedLM.from_pretrained( + "google/mobilebert-uncased", + torch_dtype=torch.float16, + device_map="auto", +) +inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt").to("cuda") + +with torch.no_grad(): + outputs = model(**inputs) + predictions = outputs.logits + +masked_index = torch.where(inputs['input_ids'] == tokenizer.mask_token_id)[1] +predicted_token_id = predictions[0, masked_index].argmax(dim=-1) +predicted_token = tokenizer.decode(predicted_token_id) + +print(f"The predicted token is: {predicted_token}") +``` + + + + +```bash +echo -e "The capital of France is [MASK]." | transformers-cli run --task fill-mask --model google/mobilebert-uncased --device 0 +``` + + + -## Resources +## Notes -- [Text classification task guide](../tasks/sequence_classification) -- [Token classification task guide](../tasks/token_classification) -- [Question answering task guide](../tasks/question_answering) -- [Masked language modeling task guide](../tasks/masked_language_modeling) -- [Multiple choice task guide](../tasks/multiple_choice) +- Inputs should be padded on the right because BERT uses absolute position embeddings. ## MobileBertConfig