update ernie model card (#39657)
* update ernie model doc Signed-off-by: Zhang Jun <jzhang533@gmail.com> * address ruff format error reported by ci Signed-off-by: Zhang Jun <jzhang533@gmail.com> * address check_repository_consistency error reported by ci Signed-off-by: Zhang Jun <jzhang533@gmail.com> --------- Signed-off-by: Zhang Jun <jzhang533@gmail.com> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
This commit is contained in:
@@ -14,29 +14,83 @@ rendered properly in your Markdown viewer.
|
|||||||
|
|
||||||
-->
|
-->
|
||||||
|
|
||||||
# ERNIE
|
<div style="float: right;">
|
||||||
|
|
||||||
<div class="flex flex-wrap space-x-1">
|
<div class="flex flex-wrap space-x-1">
|
||||||
<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white" >
|
<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white" >
|
||||||
</div>
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
## Overview
|
# ERNIE
|
||||||
ERNIE is a series of powerful models proposed by baidu, especially in Chinese tasks,
|
|
||||||
including [ERNIE1.0](https://huggingface.co/papers/1904.09223), [ERNIE2.0](https://ojs.aaai.org/index.php/AAAI/article/view/6428),
|
|
||||||
[ERNIE3.0](https://huggingface.co/papers/2107.02137), [ERNIE-Gram](https://huggingface.co/papers/2010.12148), [ERNIE-health](https://huggingface.co/papers/2110.07244), etc.
|
|
||||||
|
|
||||||
These models are contributed by [nghuyong](https://huggingface.co/nghuyong) and the official code can be found in [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP) (in PaddlePaddle).
|
[ERNIE1.0](https://arxiv.org/abs/1904.09223), [ERNIE2.0](https://ojs.aaai.org/index.php/AAAI/article/view/6428),
|
||||||
|
[ERNIE3.0](https://arxiv.org/abs/2107.02137), [ERNIE-Gram](https://arxiv.org/abs/2010.12148), [ERNIE-health](https://arxiv.org/abs/2110.07244) are a series of powerful models proposed by baidu, especially in Chinese tasks.
|
||||||
|
|
||||||
### Usage example
|
ERNIE (Enhanced Representation through kNowledge IntEgration) is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking.
|
||||||
Take `ernie-1.0-base-zh` as an example:
|
|
||||||
|
|
||||||
```Python
|
Other ERNIE models released by baidu can be found at [Ernie 4.5](./ernie4_5.md), and [Ernie 4.5 MoE](./ernie4_5_moe.md).
|
||||||
from transformers import AutoTokenizer, AutoModel
|
|
||||||
tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
|
> [!TIP]
|
||||||
model = AutoModel.from_pretrained("nghuyong/ernie-1.0-base-zh")
|
> This model was contributed by [nghuyong](https://huggingface.co/nghuyong), and the official code can be found in [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP) (in PaddlePaddle).
|
||||||
|
>
|
||||||
|
> Click on the ERNIE models in the right sidebar for more examples of how to apply ERNIE to different language tasks.
|
||||||
|
|
||||||
|
The example below demonstrates how to predict the `[MASK]` token with [`Pipeline`], [`AutoModel`], and from the command line.
|
||||||
|
|
||||||
|
<hfoptions id="usage">
|
||||||
|
<hfoption id="Pipeline">
|
||||||
|
|
||||||
|
```py
|
||||||
|
from transformers import pipeline
|
||||||
|
|
||||||
|
pipeline = pipeline(
|
||||||
|
task="fill-mask",
|
||||||
|
model="nghuyong/ernie-3.0-xbase-zh"
|
||||||
|
)
|
||||||
|
|
||||||
|
pipeline("巴黎是[MASK]国的首都。")
|
||||||
```
|
```
|
||||||
|
|
||||||
### Model checkpoints
|
</hfoption>
|
||||||
|
<hfoption id="AutoModel">
|
||||||
|
|
||||||
|
```py
|
||||||
|
import torch
|
||||||
|
from transformers import AutoModelForMaskedLM, AutoTokenizer
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(
|
||||||
|
"nghuyong/ernie-3.0-xbase-zh",
|
||||||
|
)
|
||||||
|
model = AutoModelForMaskedLM.from_pretrained(
|
||||||
|
"nghuyong/ernie-3.0-xbase-zh",
|
||||||
|
torch_dtype=torch.float16,
|
||||||
|
device_map="auto"
|
||||||
|
)
|
||||||
|
inputs = tokenizer("巴黎是[MASK]国的首都。", return_tensors="pt").to("cuda")
|
||||||
|
|
||||||
|
with torch.no_grad():
|
||||||
|
outputs = model(**inputs)
|
||||||
|
predictions = outputs.logits
|
||||||
|
|
||||||
|
masked_index = torch.where(inputs['input_ids'] == tokenizer.mask_token_id)[1]
|
||||||
|
predicted_token_id = predictions[0, masked_index].argmax(dim=-1)
|
||||||
|
predicted_token = tokenizer.decode(predicted_token_id)
|
||||||
|
|
||||||
|
print(f"The predicted token is: {predicted_token}")
|
||||||
|
```
|
||||||
|
|
||||||
|
</hfoption>
|
||||||
|
<hfoption id="transformers CLI">
|
||||||
|
|
||||||
|
```bash
|
||||||
|
echo -e "巴黎是[MASK]国的首都。" | transformers run --task fill-mask --model nghuyong/ernie-3.0-xbase-zh --device 0
|
||||||
|
```
|
||||||
|
|
||||||
|
</hfoption>
|
||||||
|
</hfoptions>
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
Model variants are available in different sizes and languages.
|
||||||
|
|
||||||
| Model Name | Language | Description |
|
| Model Name | Language | Description |
|
||||||
|:-------------------:|:--------:|:-------------------------------:|
|
|:-------------------:|:--------:|:-------------------------------:|
|
||||||
@@ -51,18 +105,11 @@ model = AutoModel.from_pretrained("nghuyong/ernie-1.0-base-zh")
|
|||||||
| ernie-health-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
|
| ernie-health-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
|
||||||
| ernie-gram-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
|
| ernie-gram-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
|
||||||
|
|
||||||
You can find all the supported models from huggingface's model hub: [huggingface.co/nghuyong](https://huggingface.co/nghuyong), and model details from paddle's official
|
|
||||||
repo: [PaddleNLP](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers/ERNIE/contents.html)
|
|
||||||
and [ERNIE](https://github.com/PaddlePaddle/ERNIE/blob/repro).
|
|
||||||
|
|
||||||
## Resources
|
## Resources
|
||||||
|
|
||||||
- [Text classification task guide](../tasks/sequence_classification)
|
You can find all the supported models from huggingface's model hub: [huggingface.co/nghuyong](https://huggingface.co/nghuyong), and model details from paddle's official
|
||||||
- [Token classification task guide](../tasks/token_classification)
|
repo: [PaddleNLP](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers/ERNIE/contents.html)
|
||||||
- [Question answering task guide](../tasks/question_answering)
|
and [ERNIE's legacy branch](https://github.com/PaddlePaddle/ERNIE/tree/legacy/develop).
|
||||||
- [Causal language modeling task guide](../tasks/language_modeling)
|
|
||||||
- [Masked language modeling task guide](../tasks/masked_language_modeling)
|
|
||||||
- [Multiple choice task guide](../tasks/multiple_choice)
|
|
||||||
|
|
||||||
## ErnieConfig
|
## ErnieConfig
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user