From d5d007a1a0f0c11a726a54c8f00bd71825f84d02 Mon Sep 17 00:00:00 2001 From: "SOUVIK CHAND [ZD]" <96312748+souvikchand@users.noreply.github.com> Date: Sat, 14 Jun 2025 03:28:06 +0530 Subject: [PATCH] Updated Albert model Card (#37753) * Updated Albert model Card * Update docs/source/en/model_doc/albert.md added the quotes in Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/model_doc/albert.md updated checkpoints Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/model_doc/albert.md changed !Tips description Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/model_doc/albert.md updated text Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/model_doc/albert.md updated transformer-cli implementation Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/model_doc/albert.md changed text Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/model_doc/albert.md removed repeated description Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update albert.md removed lines * Update albert.md updated pipeline code * Update albert.md updated auto model code, removed quantization as model size is not large, removed the attention visualizer part * Update docs/source/en/model_doc/albert.md updated notes Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update albert.md reduced a repeating point in notes * Update docs/source/en/model_doc/albert.md updated transformer-CLI Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/en/model_doc/albert.md removed extra notes Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --- docs/source/en/model_doc/albert.md | 146 ++++++++++++++--------------- 1 file changed, 73 insertions(+), 73 deletions(-) diff --git a/docs/source/en/model_doc/albert.md b/docs/source/en/model_doc/albert.md index d121e370da..b56fca55b2 100644 --- a/docs/source/en/model_doc/albert.md +++ b/docs/source/en/model_doc/albert.md @@ -14,100 +14,100 @@ rendered properly in your Markdown viewer. --> -# ALBERT - -
-PyTorch -TensorFlow -Flax -SDPA +
+
+ PyTorch + TensorFlow + Flax + SDPA +
-## Overview +# ALBERT -The ALBERT model was proposed in [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://huggingface.co/papers/1909.11942) by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, -Radu Soricut. It presents two parameter-reduction techniques to lower memory consumption and increase the training -speed of BERT: +[ALBERT](https://huggingface.co/papers/1909.11942) is designed to address memory limitations of scaling and training of [BERT](./bert). It adds two parameter reduction techniques. The first, factorized embedding parametrization, splits the larger vocabulary embedding matrix into two smaller matrices so you can grow the hidden size without adding a lot more parameters. The second, cross-layer parameter sharing, allows layer to share parameters which keeps the number of learnable parameters lower. -- Splitting the embedding matrix into two smaller matrices. -- Using repeating layers split among groups. +<<<<<<< HEAD +======= -The abstract from the paper is the following: +<<<<<<< HEAD +ALBERT was created to address problems like -- GPU/TPU memory limitations, longer training times, and unexpected model degradation in BERT. ALBERT uses two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT: -*Increasing model size when pretraining natural language representations often results in improved performance on -downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations, -longer training times, and unexpected model degradation. To address these problems, we present two parameter-reduction -techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows -that our proposed methods lead to models that scale much better compared to the original BERT. We also use a -self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks -with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and -SQuAD benchmarks while having fewer parameters compared to BERT-large.* +- **Factorized embedding parameterization:** The large vocabulary embedding matrix is decomposed into two smaller matrices, reducing memory consumption. +- **Cross-layer parameter sharing:** Instead of learning separate parameters for each transformer layer, ALBERT shares parameters across layers, further reducing the number of learnable weights. -This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by -[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT). +ALBERT uses absolute position embeddings (like BERT) so padding is applied at right. Size of embeddings is 128 While BERT uses 768. ALBERT can processes maximum 512 token at a time. +>>>>>>> 7ba1110083 (Update docs/source/en/model_doc/albert.md ) -## Usage tips +======= +>>>>>>> 155b733538 (Update albert.md) +You can find all the original ALBERT checkpoints under the [ALBERT community](https://huggingface.co/albert) organization. -- ALBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather - than the left. -- ALBERT uses repeating layers which results in a small memory footprint, however the computational cost remains - similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same - number of (repeating) layers. -- Embedding size E is different from hidden size H justified because the embeddings are context independent (one embedding vector represents one token), whereas hidden states are context dependent (one hidden state represents a sequence of tokens) so it's more logical to have H >> E. Also, the embedding matrix is large since it's V x E (V being the vocab size). If E < H, it has less parameters. -- Layers are split in groups that share parameters (to save memory). -Next sentence prediction is replaced by a sentence ordering prediction: in the inputs, we have two sentences A and B (that are consecutive) and we either feed A followed by B or B followed by A. The model must predict if they have been swapped or not. -- The `head_mask` argument is ignored when using all attention implementation other than "eager". If you have a `head_mask` and want it to have effect, load the model with `XXXModel.from_pretrained(model_id, attn_implementation="eager")` +> [!TIP] +> Click on the ALBERT models in the right sidebar for more examples of how to apply ALBERT to different language tasks. -### Using Scaled Dot Product Attention (SDPA) +The example below demonstrates how to predict the `[MASK]` token with [`Pipeline`], [`AutoModel`], and from the command line. -PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function -encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the -[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) -or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention) -page for more information. + + -SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set -`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used. +```py +import torch +from transformers import pipeline -``` -from transformers import AlbertModel -model = AlbertModel.from_pretrained("albert/albert-base-v1", torch_dtype=torch.float16, attn_implementation="sdpa") -... +pipeline = pipeline( + task="fill-mask", + model="albert-base-v2", + torch_dtype=torch.float16, + device=0 +) +pipeline("Plants create [MASK] through a process known as photosynthesis.", top_k=5) ``` -For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`). + + -On a local benchmark (GeForce RTX 2060-8GB, PyTorch 2.3.1, OS Ubuntu 20.04) with `float16`, we saw the -following speedups during training and inference. +```py +import torch +from transformers import AutoModelForMaskedLM, AutoTokenizer -#### Training for 100 iterations +tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") +model = AutoModelForMaskedLM.from_pretrained( + "albert/albert-base-v2", + torch_dtype=torch.float16, + attn_implementation="sdpa", + device_map="auto" +) -|batch_size|seq_len|Time per batch (eager - s)| Time per batch (sdpa - s)| Speedup (%)| Eager peak mem (MB)| sdpa peak mem (MB)| Mem saving (%)| -|----------|-------|--------------------------|--------------------------|------------|--------------------|-------------------|---------------| -|2 |256 |0.028 |0.024 |14.388 |358.411 |321.088 |11.624 | -|2 |512 |0.049 |0.041 |17.681 |753.458 |602.660 |25.022 | -|4 |256 |0.044 |0.039 |12.246 |679.534 |602.660 |12.756 | -|4 |512 |0.090 |0.076 |18.472 |1434.820 |1134.140 |26.512 | -|8 |256 |0.081 |0.072 |12.664 |1283.825 |1134.140 |13.198 | -|8 |512 |0.170 |0.143 |18.957 |2820.398 |2219.695 |27.062 | +prompt = "Plants create energy through a process known as [MASK]." +inputs = tokenizer(prompt, return_tensors="pt").to(model.device) -#### Inference with 50 batches +with torch.no_grad(): + outputs = model(**inputs) + mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1] + predictions = outputs.logits[0, mask_token_index] -|batch_size|seq_len|Per token latency eager (ms)|Per token latency SDPA (ms)|Speedup (%) |Mem eager (MB)|Mem BT (MB)|Mem saved (%)| -|----------|-------|----------------------------|---------------------------|------------|--------------|-----------|-------------| -|4 |128 |0.083 |0.071 |16.967 |48.319 |48.45 |-0.268 | -|4 |256 |0.148 |0.127 |16.37 |63.4 |63.922 |-0.817 | -|4 |512 |0.31 |0.247 |25.473 |110.092 |94.343 |16.693 | -|8 |128 |0.137 |0.124 |11.102 |63.4 |63.66 |-0.409 | -|8 |256 |0.271 |0.231 |17.271 |91.202 |92.246 |-1.132 | -|8 |512 |0.602 |0.48 |25.47 |186.159 |152.564 |22.021 | -|16 |128 |0.252 |0.224 |12.506 |91.202 |91.722 |-0.567 | -|16 |256 |0.526 |0.448 |17.604 |148.378 |150.467 |-1.388 | -|16 |512 |1.203 |0.96 |25.365 |338.293 |271.102 |24.784 | +top_k = torch.topk(predictions, k=5).indices.tolist() +for token_id in top_k[0]: + print(f"Prediction: {tokenizer.decode([token_id])}") +``` -This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by -[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT). + + + +```bash +echo -e "Plants create [MASK] through a process known as photosynthesis." | transformers run --task fill-mask --model albert-base-v2 --device 0 +``` + + + + + + +## Notes + +- Inputs should be padded on the right because BERT uses absolute position embeddings. +- The embedding size `E` is different from the hidden size `H` because the embeddings are context independent (one embedding vector represents one token) and the hidden states are context dependent (one hidden state represents a sequence of tokens). The embedding matrix is also larger because `V x E` where `V` is the vocabulary size. As a result, it's more logical if `H >> E`. If `E < H`, the model has less parameters. ## Resources