Add RocBert (#20013)
* add roc_bert * update roc_bert readme * code style * change name and delete unuse file * udpate model file * delete unuse log file * delete tokenizer fast * reformat code and change model file path * add RocBertForPreTraining * update docs * delete wrong notes * fix copies * fix make repo-consistency error * fix files are not present in the table of contents error * change RocBert -> RoCBert * add doc, add detail test Co-authored-by: weiweishi <weiweishi@tencent.com>
This commit is contained in:
Weiwe Shi
@@ -339,6 +339,8 @@
|
||||
title: RetriBERT
|
||||
- local: model_doc/roberta
|
||||
title: RoBERTa
|
||||
- local: model_doc/roc_bert
|
||||
title: RoCBert
|
||||
- local: model_doc/roformer
|
||||
title: RoFormer
|
||||
- local: model_doc/splinter
|
||||
|
||||
@@ -154,6 +154,7 @@ The documentation is organized into five sections:
|
||||
1. **[RemBERT](model_doc/rembert)** (from Google Research) released with the paper [Rethinking embedding coupling in pre-trained language models](https://arxiv.org/abs/2010.12821) by Hyung Won Chung, Thibault Févry, Henry Tsai, M. Johnson, Sebastian Ruder.
|
||||
1. **[ResNet](model_doc/resnet)** (from Microsoft Research) released with the paper [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385) by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun.
|
||||
1. **[RoBERTa](model_doc/roberta)** (from Facebook), released together with the paper [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
|
||||
1. **[RoCBert](model_doc/roc_bert)** (from WeChatAI) released with the paper [RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining](https://aclanthology.org/2022.acl-long.65.pdf) by HuiSu, WeiweiShi, XiaoyuShen, XiaoZhou, TuoJi, JiaruiFang, JieZhou.
|
||||
1. **[RoFormer](model_doc/roformer)** (from ZhuiyiTechnology), released together with the paper [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864) by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.
|
||||
1. **[SegFormer](model_doc/segformer)** (from NVIDIA) released with the paper [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://arxiv.org/abs/2105.15203) by Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Ping Luo.
|
||||
1. **[SEW](model_doc/sew)** (from ASAPP) released with the paper [Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition](https://arxiv.org/abs/2109.06870) by Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi.
|
||||
@@ -305,6 +306,7 @@ Flax), PyTorch, and/or TensorFlow.
|
||||
| ResNet | ❌ | ❌ | ✅ | ✅ | ❌ |
|
||||
| RetriBERT | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| RoBERTa | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| RoCBert | ✅ | ❌ | ✅ | ❌ | ❌ |
|
||||
| RoFormer | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| SegFormer | ❌ | ❌ | ✅ | ✅ | ❌ |
|
||||
| SEW | ❌ | ❌ | ✅ | ❌ | ❌ |
|
||||
|
||||
93
docs/source/en/model_doc/roc_bert.mdx
Normal file
93
docs/source/en/model_doc/roc_bert.mdx
Normal file
@@ -0,0 +1,93 @@
|
||||
<!--Copyright 2022 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# RoCBert
|
||||
|
||||
## Overview
|
||||
|
||||
The RoCBert model was proposed in [RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining](https://aclanthology.org/2022.acl-long.65.pdf) by HuiSu, WeiweiShi, XiaoyuShen, XiaoZhou, TuoJi, JiaruiFang, JieZhou.
|
||||
It's a pretrained Chinese language model that is robust under various forms of adversarial attacks.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Large-scale pretrained language models have achieved SOTA results on NLP tasks. However, they have been shown
|
||||
vulnerable to adversarial attacks especially for logographic languages like Chinese. In this work, we propose
|
||||
ROCBERT: a pretrained Chinese Bert that is robust to various forms of adversarial attacks like word perturbation,
|
||||
synonyms, typos, etc. It is pretrained with the contrastive learning objective which maximizes the label consistency
|
||||
under different synthesized adversarial examples. The model takes as input multimodal information including the
|
||||
semantic, phonetic and visual features. We show all these features are important to the model robustness since the
|
||||
attack can be performed in all the three forms. Across 5 Chinese NLU tasks, ROCBERT outperforms strong baselines under
|
||||
three blackbox adversarial algorithms without sacrificing the performance on clean testset. It also performs the best
|
||||
in the toxic content detection task under human-made attacks.*
|
||||
|
||||
This model was contributed by [weiweishi](https://huggingface.co/weiweishi).
|
||||
|
||||
## RoCBertConfig
|
||||
|
||||
[[autodoc]] RoCBertConfig
|
||||
- all
|
||||
|
||||
|
||||
## RoCBertTokenizer
|
||||
|
||||
[[autodoc]] RoCBertTokenizer
|
||||
- build_inputs_with_special_tokens
|
||||
- get_special_tokens_mask
|
||||
- create_token_type_ids_from_sequences
|
||||
- save_vocabulary
|
||||
|
||||
|
||||
## RoCBertModel
|
||||
|
||||
[[autodoc]] RoCBertModel
|
||||
- forward
|
||||
|
||||
|
||||
## RoCBertForPreTraining
|
||||
|
||||
[[autodoc]] RoCBertForPreTraining
|
||||
- forward
|
||||
|
||||
|
||||
## RoCBertForCausalLM
|
||||
|
||||
[[autodoc]] RoCBertForCausalLM
|
||||
- forward
|
||||
|
||||
|
||||
## RoCBertForMaskedLM
|
||||
|
||||
[[autodoc]] RoCBertForMaskedLM
|
||||
- forward
|
||||
|
||||
|
||||
## RoCBertForSequenceClassification
|
||||
|
||||
[[autodoc]] transformers.RoCBertForSequenceClassification
|
||||
- forward
|
||||
|
||||
## RoCBertForMultipleChoice
|
||||
|
||||
[[autodoc]] transformers.RoCBertForMultipleChoice
|
||||
- forward
|
||||
|
||||
|
||||
## RoCBertForTokenClassification
|
||||
|
||||
[[autodoc]] transformers.RoCBertForTokenClassification
|
||||
- forward
|
||||
|
||||
|
||||
## RoCBertForQuestionAnswering
|
||||
|
||||
[[autodoc]] RoCBertForQuestionAnswering
|
||||
- forward
|
||||
Reference in New Issue
Block a user